BioFormats high memory use for nd2 files

Hi,

I am trying to open a 15.9 GB nd2 file using Bio-Formats on a 32 GB or 64 GB RAM Windows PC. The file is 2 channels, 60,000 frames for each channel, 16-bit 256x256 pixel frames. With the latest, updated Fiji and the “Bio-Formats” update site checked (I’d say it’s BF 6.6.1, but the same error occurs without the update site checked where it’s 6.6.0), I get memory errors with an exception about Java heap space. It also happens if I check “open as virtual file” in the Bio-Formats dialog. Here’s the error on the 32 GB machine (it’s the same on the 64 GB machine, with 50 GB allocated to Fiji):

(Fiji Is Just) ImageJ 2.1.0/1.53g; Java 1.8.0_172 [64-bit]; Windows 10 10.0; 31891MB of 32000MB (99%)
 
java.lang.OutOfMemoryError: Java heap space
	at loci.formats.in.NativeND2Reader.initFile(NativeND2Reader.java:1454)
	at loci.formats.FormatReader.setId(FormatReader.java:1421)
	at loci.formats.DelegateReader.setId(DelegateReader.java:291)
	at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:499)
	at loci.plugins.in.ImportProcess.execute(ImportProcess.java:142)
	at loci.plugins.in.Importer.showDialogs(Importer.java:140)
	at loci.plugins.in.Importer.run(Importer.java:76)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:241)
	at ij.IJ.runPlugIn(IJ.java:204)
	at ij.IJ.runPlugIn(IJ.java:193)
	at HandleExtraFileTypes.openImage(HandleExtraFileTypes.java:503)
	at HandleExtraFileTypes.run(HandleExtraFileTypes.java:76)
	at ij.IJ.runUserPlugIn(IJ.java:241)
	at ij.IJ.runPlugIn(IJ.java:204)
	at ij.IJ.runPlugIn(IJ.java:193)
	at ij.io.Opener.openWithHandleExtraFileTypes(Opener.java:518)
	at ij.io.Opener.openUsingHandleExtraFileTypes(Opener.java:372)
	at ij.io.Opener.openImage(Opener.java:359)
	at ij.io.Opener.openImage(Opener.java:243)
	at ij.io.Opener.open(Opener.java:109)
	at ij.io.Opener.openAndAddToRecent(Opener.java:292)
	at ij.plugin.DragAndDrop.openFile(DragAndDrop.java:194)
	at ij.plugin.DragAndDrop.run(DragAndDrop.java:160)
	at java.lang.Thread.run(Thread.java:748)

I also tried Bio-Formats command line in anaconda, but the memory use (and java heap space exception) is the same. The issue is reminiscent of my previous struggles back in 2014 ([ome-users] Fwd: .ND2 reader in BioFormats).

Is such a large (> 3X on the 64 GB machine) memory overhead intended? Is there a way to open such a file with Bio-Formats using a reasonable amount of RAM? I just want to convert it to tif.

1 Like

Hi @christlet,

Weird issue and I don’t have enough knowledge about the .nd2 format to explain anything here…
For a conversion to a tif format, you might want to try AiviaCommunity (free version, no registration, just click on the download button):

We have a native .nd2 reader (hope it works for your version) and you can save the file as .aivia.tif, which is a tif format with some extra layers of info for all functionalities in Aivia.
You should be able to use it in any external software…
Hope this helps!
Patrice

PS: In the menu “Tools > Image Converter” you can do this as a batch…

1 Like

Hi @christlet, that really does seem unusual for the size of the file. Are you only seeing the issue with that 1 particular file or is it similar for other nd2 files?

Using the Use Virtual Stack option is a good idea, it might also be worth forcing the garbage collection first before the import, you can do this by clicking on the status bar or using Plugins > Utilities > Collect Garbage.

If you want me to take a more detailed look at the file you can send me a PM here with an email contact address and I will arrange ftp access for you to transfer the file.

2 Likes

As an update to this issue, having been able to profile the the issue with the sample file, it is indeed caused by the storage of the image offset values which is the same as a previous issue which we had attempted to fix in ND2: reset series count before allocating offset array by melissalinkert · Pull Request #1351 · ome/bioformats · GitHub. I have been able to do some initial testing on expanding this original PR to catch more cases such as this one, so far it looks positive but I will need to further test to ensure it will not cause any regressions for other filesets. Hopefully a PR should be open this week with a potential fix in place.

2 Likes

An initial PR to test a fix for this issue is now open at ND2: reset series count to limit offsets by dgault · Pull Request #3656 · ome/bioformats · GitHub

2 Likes

Hi David @dgault, has this been incorporated in BioFormats yet? How can I try it? I have the 6.6.1 snapshot from the BioFormat update site in Fiji but not sure it’s in it. Thanks!

It has not yet been merged but it is now passing our internal tests so I will try and get a final review of it to get it merged and into the snapshot builds next week.

2 Likes

thanks! Will test as soon as it’s out

Just as a follow up, the PR for this has now been merged and should be available in the latest snapshots. Apologies again for the delay in getting this integrated.

2 Likes