Looking for a headless HDF reader

fiji
imagej
format
batch-processing
hdf5

#1

Hello everyone,

Questions

Would anyone know whether there exists a HDF5 reader for FIJI that operates in headless mode? Has anyone implemented batch analysis routines on data stored in HDF files in FIJI and can comment on how this was done?

What I would like to do

I am working with a custom database of images that are stored as separate datasets inside a single HDF5 file. The images are simply 2D arrays of ints (nothing fancy). I would like to batch process select images from the file using FIJI’s headless mode. The reason for wanting to work from the command line is that my full analysis pipeline utilizes different programs and languages which I glue together with bash scripts.

Possible solutions to my problem

I have tried the plugin from Uni. Freiburg lmb.informatik.uni-freiburg.de/resources/opensource/imagej_plugins/hdf5.html but it seems only to work with the GUI.

A rather inelegant solution would be to read the images from the HDF file and write them as separate TIFFs using a separate program, then open the TIFF’s in FIJI for processing.

Thanks everyone for your input!


#2

The latest version of this plugin is shipped via the HDF5 update site and includes the commands Scriptable load HDF5 and Scriptable save HDF5 that are recordable using the macro recoder and should also run in headless mode:

run("Scriptable load HDF5...", "load=C:\\path\\to\\your\\image.h5 datasetnames=/t0/channel0 nframes=1 nchannels=1");

#3

Thanks Jan! I overlooked this because was I only looking at File -> Import -> HDF5… and not in the Plugins menu.

Cheers,
Kyle


#4

Hello everyone,
Just a quick follow-up on this post that others might find useful:

As far as I can tell, there is only one scriptable load option currently available in the HDF5 Plugin that is available on the Fiji update site. This means, for example, that you will be unable to load images in HDF5 files with custom data layouts through a macro.

I wanted exactly the ability to load datasets with custom layouts without resorting to the GUI. I therefore slightly modified the HDF5 Plugin to expose the load functionality for custom data layouts to a scriptable interface. The modified code may be found here:

I hope this helps others who are looking for similar functionality.

Cheers,
Kyle


#5

Great!

Did you contact Olaf Ronneberger letting him know about this fork? It would be ideal to: A) get your improvements merged upstream; and B) get the HDF5 plugin source code onto GitHub in general, so that we can use PRs properly in the future.


#6

Hi Curtis,

I sent an e-mail to his Uni Freiburg account a couple of weeks ago prior to making the changes but didn’t get a response. (To be fair, it’s vacation time in Europe.) I’ll try his Gmail address and see if I get a response.

I’d like to help merge the code upstream. I’ll wait to see if I hear from him first and come back to you off-list with any questions.

Cheers,
Kyle


#7

Did you ever hear anything from Olaf Ronneberger? It seems there hasn’t been much development on the plugin since 2014.


#8

No, unfortunately I never received any word from Dr. Ronneberger. I also became a bit busy these past few months, which is why I myself have not followed up on trying push any changes upstream.

@ctrueden What would be the next steps in trying to make this HDF5 plugin code more accessible to Fiji users? Also, is there a recommended unit testing framework for Fiji plugins?


#9

The plugin itself is decent enough. But I think there is still plenty of room for improvements (they even have a wish list on the plugin web page), so it is too bad that development has apparently halted completely. Unfortunately, I do not have the necessary skills to continue the project, even if I had the time. I do hope, however, that someone will pick up the mantle, so to speak.


#10

Seconded. We could fork it into the Fiji project. It would be ideal, however, to make contact with the original authors before doing so. We need to confirm the project license (from my research, it looks like Apache 2, which is great, but not totally clear from the web site). We should also get the complete SCM history exported and pushed to GitHub, instead of trying to reconstruct it from source code inside JAR file releases.

Personally, I do not have time to resurrect and maintain this component. Someone should try really hard to contact everyone who might be involved in the original development via email etc.


#11

I am interested in this issue too. If only to get included a feature to handle larger files and/or select a sub-(hyper)-volume before trying to load the whole thing in memory.

Did anyone from Freiburg reply on the status of the original project ?


#12

@robert_a @BThorsted @kmdouglass If we fork this project into the Fiji space, would you be willing & able to maintain it? The maintenance levels can be seen here. I am willing to help maintain it from an infrastructure perspective, but would not have time nor expertise to fix bugs, add features, review PRs, or support the plugin on the ImageJ Forum. Someone also needs to make the effort to reach out to Dr. Ronneberger in all the known ways. He has several phone numbers listed on his bio page; it would be shame to give up on this project before trying everything possible to move things forward.


#13

Hi Robert,
No, unfortunately I never made any contact with the author on the original project. I have tried both e-mail addresses multiple times across a few month time-span but never received any sort of reply.


#14

I also emailed him just now using the Invite function here on the forum, asking if he would be willing to respond here, and/or export the source code history so that we can fork the project.


#15

Hi Curtis,

I’m afraid I may not have the time to maintain it just by myself. I am also not a FIJI developer so it would take me some time to become fully acquainted with FIJI’s plugin framework.

However, I also really want this plugin improved for many of my own analyses, so I would be happy to act as a co-maintainer of the plugin. If someone more experienced with FIJI development were also to agree to help maintain it, then I would gladly contribute my time to the effort.


#16

It sounds like you would be able to do it, then. I am volunteering myself to help with the “Fiji-ish” aspects, including setting up the POM structure, making sure CI builds work, and all that infrastructure stuff. You would not need to learn the SciJava plugin framework—you would only need to understand the code of this specific plugin and help others to do so as well when they have questions, or file PRs against it. Make sense?


#17

Makes sense to me. Since there seems to be a lot of interest, then I agree to help maintain this plugin if Dr. Ronneberger doesn’t respond. Just a fair warning though: I’m a bit busy with grant writing at the moment so it may take me some time to get familiar with the full code base.


#18

@kmdouglass Awesome. I emailed him, along with other developers of that project, one last time, and CCed you. I then created a new repository at https://github.com/fiji/HDF5_Vibez and made a commit for each of the available versions of the source code (3 total). It is coarse, but does provide a record of changes, as well as a working Maven-based build.

Take a look when you have time and let me know what you think. I guess the next step is to integrate the changes from your fork into the fiji/HDF5_Vibez repository. Once you are happy with it, I can cut a release, add it to Fiji and upload it to the Java-8 update site.


#19

@ctrueden Sounds great. I’ll have a look over the next week and start merging my changes.

BTW, do recommend any unit testing frameworks for Fiji plugins, or do you leave that up to the package maintainers to decide? Most recently I’ve been working in the Python world and need to reorient myself in Java a bit.


#20

Great! Please don’t hesitate to ask if you run into any difficulties.

It is ultimately your decision, but I urge you to use JUnit unless you have strong reasons not to do so. We use JUnit for all other SciJava projects, and in my experience it works very well. I also have experience with TestNG thanks to Bio-Formats, but it is actually very rare that one needs TestNG’s “data-driven testing” features. If you choose a test framework besides JUnit, then I will be less equipped to help you with any problems relating to that framework.