Open CZI in Bio-Formats Large RAM requirements

Hi all

cc @dgault @sebi06

We are opening 68 czi files of 5 MB each (68 * 5 = 340 MB).
This occupies 2.5 GB of RAM, which is a bit much.

Debugging into this we found that there are 1.1 GB used in Strings being held by ZeissCZIReader instances.

We were wondering whether (a) this is a know issue and whether (b) there is a cure to this in terms of reducing the memory load.

In case this is a new phenomenon and there is no simple fix known, but someone happens to know someone that would have the resources (in terms of time and knowledge) to debug this I would try to share an example data set and create some example code, probably as a Fiji jython script.

Thank you for your help!

Christian and @NicoKiaru

Its not something I’ve looked into before but I wouldn’t be shocked if that reader was holding onto unnecessary metadata. Are you opening these 68 files via a jython script reusing the same reader?

We keep one reader per file and per channel and per series… So there are 68x3 instances of ZeissCZIReader.

This is how https://github.com/BIOP/bigdataviewer-bioformats is designed. The reason to keep the reader is for lazy loading of big files in bigdataviewer (but I guess the same situation happens with virtual stacks in ImageJ - the reader is maintained in memory for lazy loading).

I can try to reuse the same reader for the different channels (maybe we gain a bit of space), but we need at least one reader per file, because the creation of the reader is pretty slow, even with memoization.

@dgault do you think it’s possible to drop some metadata ? Is there by any chance a way to do this from the ImageReader interface ?

Sorry for the delayed response, I had meant to take a look at this further on Friday. There may well be some performance improvements we can make to the reader but it would like mean a minor release of BF, so not something that can easily be done via ImageReader I’m afraid. Let me try to do some profiling this week and see what we can do.

1 Like

Though as a possible easy solution, if the original metadata is not required then you can prevent it from being populated using reader.setOriginalMetadataPopulated(false); That could potentially reduce the size of metadata substantially.

1 Like