OMERO: large CZI import fails w/ Java heap space out of memory

Hi OME team,

I’m trying to import a very large CZI image (89GB, consisting of 152 scenes with 5-channel images per scene, stored as uncompressed) using the CLI importer into a current OMERO 5.6.2 on Ubuntu 18.04 (Java 11). The import fails with:

sudard@graylab:/data/share/dataxchange/QiTissue/BM-Her2N75-15$ omero import -d 4051 R2_PCNA.Her2.ER.CD45_BM-Her2N75-15_2017_08_11__2327.czi --debug WARN
Previous session expired for username on xxxxxx.ohsu.edu:4064
Server: [xxxxxx.ohsu.edu:4064]
Username: [username]
Password:
Created session for username@xxxxxxx.ohsu.edu:4064. Idle timeout: 10 min. Current group: GrayLab
2020-09-12 15:02:35,522 505        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.7
2020-09-12 15:02:35,552 535        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.5.1 revision: 6f50e4d52c9d96112635fd8b2dde737f31041cf0 date: 7 July 2020
2020-09-12 15:02:40,057 5040       [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /home/groups/graylab_share/OMERO.rdsStore/dataxchange/QiTissue/BM-Her2N75-15/R2_PCNA.Her2.ER.CD45_BM-Her2N75-15_2017_08_11__2327.czi
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
        at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:469) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148) ~[omero-blitz.jar:5.5.7]
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:997) ~[omero-blitz.jar:5.5.7]
Caused by: java.lang.OutOfMemoryError: Java heap space
        at loci.formats.in.ZeissCZIReader$Attachment.fillInData(ZeissCZIReader.java:4121) ~[formats-gpl.jar:6.5.1]
        at loci.formats.in.ZeissCZIReader.readAttachments(ZeissCZIReader.java:1655) ~[formats-gpl.jar:6.5.1]
        at loci.formats.in.ZeissCZIReader.initFile(ZeissCZIReader.java:1116) ~[formats-gpl.jar:6.5.1]
        at loci.formats.FormatReader.setId(FormatReader.java:1392) ~[formats-api.jar:6.5.1]
        at loci.formats.ImageReader.setId(ImageReader.java:849) ~[formats-api.jar:6.5.1]
        at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167) ~[omero-blitz.jar:5.5.7]
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) ~[formats-api.jar:6.5.1]
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:223) ~[formats-bsd.jar:6.5.1]
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) ~[formats-api.jar:6.5.1]
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293) ~[formats-bsd.jar:6.5.1]
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) ~[formats-api.jar:6.5.1]
        at loci.formats.Memoizer.setId(Memoizer.java:662) ~[formats-bsd.jar:6.5.1]
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650) ~[formats-api.jar:6.5.1]
        at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427) ~[omero-blitz.jar:5.5.7]
        ... 6 common frames omitted
2020-09-12 15:02:43,256 8239       [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
No imports due to errors!

There are no indications in the OMERO logs what exactly is causing the OOM and the fail happens very quickly so it appears to be the “front-end” importer itself that fails. Can I do something at the command line to give the Java import process more memory or to get a bit more logging to know what to do? The image itself loads fine into ZEN so appears to be not corrupted. It’s way too big to upload to the QA site. Any suggestions?

Thanks
Damir

Hi Damir,

You can increase the Java memory used client-side by the importer using JAVA_OPTS e.g.

$ export JAVA_OPTS=-Xmx4G
$ omero import -d 4051 R2_PCNA.Her2.ER.CD45_BM-Her2N75-15_2017_08_11__2327.czi --debug WARN

As long as the server is configured with at least as much memory which is usually the case, I expect this should fix the problem.

Sebastien

Thanks Sebastien, that indeed works fine. Just a quick follow-up:

  1. what is the default amount of memory the client-side gets if you don’t set the JAVA_OPTS?
  2. since I frequently have such large images, I’ll want to change that default to something higher for all users. Where can I do so?

Cheers,
Damir

Hi Damir,

the default memory settings for the client-side importer is defined in https://github.com/ome/omero-py/blob/fda6fc4955a7fde68bd5588ec0fa62da51e4b40a/src/omero/plugins/import.py#L510 as 1G currently.

Apart from setting JAVA_OPTS in your import environment or hardcoding a larger value in the library, I imagine a more flexible option might be to make this value configurable via an option passed to the command. Let us know what works best for you and/or feel free to open a contribution against omero-py.

Sebastien