Loading and processing ImgLib2 images with the N5 API and lazy ImgLib2-cache, also some other useful links

Hey @axtimwalde,

a great initiative! I would love to add convenience functions to this repo but before I dig deeper, may I ask for some pointers/hints? I struggled earlier with imglib2 and that’s why I’m asking in advance :wink:

I would be interested in things like this:

  • Get dimensions of an image
  • Crop a region in space and time
  • Process a CellImg cell-by-cell
  • Combine multiple images (tiles) to a new image (potentially harddrive-cached, because it’ll be big)
  • Split channels
  • Read/write BDV-HDF5
  • Read/write N5
  • Applying a threshold method, e.g. Otsu 1979
  • Detect local maxima with a given tolerance
  • Flood fill
  • Voronoi diagrams
  • Binary invert
  • Binary fill holes
  • Connected components analysis

Are they possible with imglib2? Is this the right abstraction layer? Are there examples for these things you (or anyone else) know about?

I’m just trying to select the right challenges for myself :wink:

Thanks!

Cheers,
Robert

2 Likes

Hi @haesleinhuepf, for read/write N5 see this: https://www.ini.uzh.ch/~acardona/fiji-tutorial/#imglib2-n5

The examples are concise and straightforward thanks to this library https://github.com/saalfeldlab/n5-imglib2.git

3 Likes

Hey @albertcardona,

wow, awesome! That’s indeed very concise for opening N5:

img = N5Utils.open(N5FSReader(path, GsonBuilder()), dataset_name)  

But then it doesn’t belong in imglib2 functions because of additional dependencies, right? Where would be a good place for convenience functions like this? Thinking of an IJ2 gateway :slight_smile:

img = IJ2.openN5(filename);

Does something similar exist for BDV-HDF5? CC @tpietzsch

Thanks again, these links were very helpful!

Cheers,
Robert

Hi @haesleinhuepf,

This repo is not meant for convenience methods into ImgLib2 but for functions (as in function, the math thing) that we can use to test and visualize stuff.

For most of your questions, a tiny bit of Google, JavaDoc, and Imglib2-tutorials reading would probably get you much further away from knowing absolutely nothing ;). Generally, I think it is fair and helpful to assume that because I do not know X doesn’t mean that nobody knows X. Also, asking questions in a more goal oriented way is helpful for both the asker and the answerer. I had the ugly feeling that you just you just threw out some random standard image processing concepts without specifying what the expected output is and what the inputs are.

I agree that we need to do better in making things accessible. That does not mean, however, that everything is not accessible. In particular the ImageJ Ops and Knime folks, even if I disagree with them at times, have done a respectable shitload of work to implement standard algorithms for ImageJ on top of ImgLib2. All of that stuff is distributed with standard Fiji and searchable in it’s toolbar. You know that but your questions sound as if you hadn’t ever heard about it. This stuff is also very Googlable, write “imagej ops YOUR_QUESTION”, or “imglib2 YOUR_QUESTION”.

Sorry for the rant ;). So simple things first:

  • Get dimensions of an image

http://letmegooglethat.com/?q=imglib2+dimensions

dimensions(long[])

  • Crop a region in space and time

Views.interval(RandomAccessibleInterval, ...) there is no difference between space and time.

  • Process a CellImg cell-by-cell

Process with what? Pixelwise? Probably cursor() then, but what is the input and who consumes? If you want to keep the output and want to make little assumptions about the input, there is this cache method with a Consumer coming to probably imglib2-cache but eventually imglib2 that I am using daily to generate blocks of a cached image by doing X:

Here is a gradient generator for starters:

But that is not necessarily all you want. It is what Dask is doing though, so you may believe it is all you want ;).

  • Combine multiple images (tiles) to a new image (potentially harddrive-cached, because it’ll be big)

BigStitcher, N5. Overlap or no overlap? Too unspecific question.

  • Split channels

Views.hyperslice(...), you know that from prior discussions.

  • Read/write BDV-HDF5

http://letmegooglethat.com/?q=Read+write+bigdataviewer+HDF5

  • Read/write N5

n5-imglib2, n5-ij, UI in the works, please activate the N5 update site.

There is also this slightly outdated tutorial

This needs work but we’re on it.

  • Applying a threshold method, e.g. Otsu 1979

http://letmegooglethat.com/?q=imagej+ops+threshold+otsu

https://javadoc.scijava.org/ImageJ/net/imagej/ops/Ops.Threshold.Otsu.html

  • Detect local maxima with a given tolerance

http://letmegooglethat.com/?q=imglib2+detect+local+maxima

https://javadoc.scijava.org/ImgLib2/net/imglib2/algorithm/localextrema/package-summary.html

  • Flood fill

http://letmegooglethat.com/?q=imglib2+flood+fill

https://javadoc.scijava.org/ImgLib2/net/imglib2/algorithm/fill/FloodFill.html

  • Voronoi diagrams

Of what? Sets of points? In what output format? As a mask? No problem, check the NearestNeighborSearch examples in the tutorials. I stop with the Google links here because I feel like a jerk doing it, I guess I am a jerk.

  • Binary invert

Converters.convert(input, (a, b) -> b.set(!a.get()), new YourType), implement !(not).

  • Binary fill holes

https://forum.image.sc/t/fillholes-imagej-ops/2591/13

  • Connected components analysis

https://scijava.org/javadoc.scijava.org/ImgLib2/net/imglib2/algorithm/labeling/ConnectedComponentAnalysis.html

Sorry again if I come across rude, but I was thrown off by the questions because they did not seem targeted at solving concrete problems but rather at saying words. Hope you will take it ok.

Cheers,
Stephan

2 Likes

Ah, I see. My fault.

Hm, I was rather listing the things I struggled with earlier when trying to do them in imglib2. Again, I thought you were looking for building a Imglib2Functions gateway such as ImageJFunctions or BDVFunctions. Confusion on my side. Apologies.

Don’t worry :slight_smile: And thanks for the links!

I’m just trying to make the life of image analysts easier, you know.

And that really puzzles me. How can this work in practice? My images always have space and time - at least in ImageJ after opening them. But when I process them deep inside an imglib2 based library, I cannot know what’s space and what’s time (and channels). How do you work around these issues in practice? You cannot know if your image is stored as XYCTZ or XYZTC, can’t you?

My questions towards space, time, tiles and channels all head towards the same scenario: I have an image (let’s say 4 - 40 GB large) and cannot process it as a whole. I would like to split it into 4, 16, 32,… tiles, process them and then assemble them together. So I crop tiles (+ margin) and then would like to combine the results again in an Image. On a computer with 8 GB of RAM that’s already challenging. If you want to process the tiles on a GPU with 1 GB of memory even more. Hence, the links you provided appear really helpful! Thanks!

Yes, I’d like to do custom processing workflows (consisting of filters, thresholding, basic stuff) on tiles. That’s extremely challenging from a coding perspective at the moment, from the ImageJ GUI just impossible and I think imglib2&friends are the right base to make these things possible and accessible.

I was more hoping for something like

img = BDVTools.open(filename);

And I was hoping for a static function in fact because I deploy to environments where there is no ij.io() and ij.ops() gateway available.

I’m fine. Again, a misunderstanding. I thought you were heading towards convenience functions. And I would have loved to help putting them together from the links you provided. Some of them I knew, but again I’m a bit out of the imglib2 busines because I found simple things to complicated and thus, went down a different road. I just wanted to check if there were recent developments I may have missed. I left imglib2 behind in 2017, CCA arrived in 2018 :wink: You know it’s sometimes hard to find the right piece of code in github repositories of tutorials - especially for people who are not so deeply involved.

Thanks again for the links!

Cheers,
Robert

Excellent, but as we’re on making this thread about something else:

You should really check out this Lazy class, I think it will do exactly what you are after. In true ImgLib2 spirit, it does not ‘process’ anything but builds the graph of dependencies. No need to manually pad stuff or write complicated processing logic. Create the output of filter A, feed it into filter B, combine with C to create C, then show with BdvFunctions.show(...) or save with e.g. N5Utils.save(... ), use the variant with the ExecutorService to use multi-threading, lean back and enjoy. What you need to implement are the block-generators assuming infinite inputs. Check out this example:

You may like it.

This is covered by n5-imglib2. Use the N5HDF5Reader instead of the N5FSReader, BDV-HDF5 is just HDF5, you will have to select a dataset though, because ImageJ is not auto-multi-scale, right? You can use the list methods of the N5Reader to explore the HDF5 file just as you would with an N5 file or a Zarr file, or N5 data in the cloud. I could imaging e that you would like that, but you will have to spend a bit of time with it to appreciate it. It does not work as you expect, it is better! (Sorry for the self promotion, but it’s totally true ;).

Please check this script, it’s not so different from the tutorial, replace the N5FSReaders with N5HDF5Readers of your liking, of course writing also works.

1 Like

Awesome! A bouquet of helpful links! So good.

I see. And I get the concept. Maybe I’m abusing imglib2, but I was really thinking of shipping tiles to the GPU, doing the heavy lifting there and just using imglib2 for tiling and stitching. You know, that’s just not doable with ImagePlusses. The Lazy class appears to be exactly what I was looking for.

Again, thanks for the links - and for the google demo - still smiling over here :rofl:

2 Likes

No, this is not abuse, it’s exactly what this is for. You write a Consumer that fills the cell by doing some GPU processing on whatever combination of inputs. The result will be automatically memory cached, so as long as your memory is big enough, you do not do the same thing twice. I would love to have that for myself, so would be happy to create an example if you could help me with the CLIJ sauce.

I would expect that you will need some sort of queue for the GPU, so it doesn’t get bombarded with too many parallel requests, and you would have to make sure that you enqueue the GPU processing part only after you already have all the necessary inputs (because they may have to be generated by the GPU, i.e. enqueued before).

You will see that I wrote methods that can consume Ops, but the Ops framework is a bit too focused on finite inputs which we definitely do not want here, the Gradient examples are probably the best way to start.

That would be ultra-cool!

3 Likes

That sounds like a win-win!

Yes! I recently implemented a prototype which holds a couple of CLIJ instances distributed over multiple GPUs. Processing frame-by-frame works quite nicely:

But I would like to go a bit further. So here this is how to use Framor, a Frame-by-Frame-Processor. It takes any huge, potentially virtual ImagePlus and an Object implementing the FrameProcessor (Example implementation). Changing its interface from ImagePlus to RandomAccessibleInterval should be trivial, as CLIJ supports RAIs as well.

Oh and btw. if you’re looking for real challenge: Make this possible with imglib2 data structures in macro. :wink:

No hurry, this project will happen anyway at some point. I’ll take a look at Lazy in the meantime. :raised_hands:

Ok, I changed the topic of this thread so people do not get even more confused. No idea where to go with my functions now…

1 Like

The closest that you can get to this is

SpimDataMinimal spimData = new XmlIoSpimDataMinimal().load( xmlFilename );

This should open any bdv xml that has an associated known imageloader, e.g. N5, HDF5, TIFF, etc. The result is this SpimData object that is not really a single image, but rather can provide 3D stacks for each channel/timepoint/resolution-level combination.
You can just BdvFunctions.show(...) it, but probably you want to get to the image data directly. In the simplest case, assuming you have 1 timepoint, 1 channel, you would get the RandomAccessibleInterval for that like

spimData.getSequenceDescription().getImgLoader().getSetupImgLoader( 0 ).getImage( 0 );

If you just dig a bit through the objects in this sequence, it should be easy to find how to get the number of channels, timepoints, resolution levels, the data Types, image sizes, etc.

2 Likes

Hey Tobias @tpietzsch,

that looks great! Actually, having the other pyramid levels so close and easily accessible appears super useful for processing big data.

Thanks!

Cheers,
Robert

1 Like

Yes, that absolutely makes sense! If you are looking for big blobs it makes sense to look in the coarser resolutions…

2 Likes

We could split out your first two posts into a new topic, if you’re ok with that :smile:

2 Likes

I’d be ok with that, thanks! Oh, or can you move everything else into a new topic? Because I had an external link into this topic that is now pointless.

1 Like