OutOfMemory exceptions importing ome.tiff files over 9 or 10GB

I have several ome.tiff files that I have exported from our software that I am attempting to import into the Omero server using OMERO.insight.

Files around 9GB are throwing Java heap space out of memory exceptions. (And I am also seeing the same out of memory exceptions attempting to run bftools xmlvalid on the larger files.)

Is this addressable as a configuration issue? Can I allocate more memory to the software via Java or otherwise? Or is this an internal software code limit for Omero at this time?

Thanks.

I have no experience with Omero, but are your ome.tiff files formatted pyramidal? If not, could they be?

They are pyramidal, yes. And lzw compressed.

1 Like

@blasky my immediate suspicion is that the memory exception is caused by the size of the XML metadata.

You might likely need more memory allocated than the default in the various tools. Below are the various places to adjust these values in the different tools mentioned in the thread above:

  • for the Bio-Formats command-line utilities, you can set BF_MAX_MEM environment variable i.e. export BF_MAX_MEM=4g. This should give you a reasonable estimate of how much memory the OMERO client and server also need to process the metadata
  • for OMERO.insight, you can adjust the memory allocation as described here
  • for OMERO.server, you can adjust the memory allocated to the various services as described here. The blitz service is the one you will want to adjust if the value reported by omero admin jvmcfg is too low.
1 Like

Sounds right. The metadata includes mask data for the ROIs. So it is non-trivial. I will start with bftools as you suggest. Thanks.

  1. I modified the bf.bat file to up the BF_MAX_MEM. A setting of 2g was enough to get all of the bftools parsing and running just fine with my 12GB ome-tiff file.

  1. Then I modified the OMERO.insight files with an equivalent “-Xmx5g” setting. (Needed more memory there.)

  2. My Omero Server is running in a docker container. So my understanding is that I can still follow the server config recommendations linked above in the .yml file with something like:

    environment:
      CONFIG_omero_jvmcfg_strategy: manual
      CONFIG_omero_jvmcfg_heap_size: 5000

The Scanning phase in OMERO.insight succeeds. The Upload phase succeeds. (The file is on the drive in the container.)

However, I still get a failed import in the Processing phase with this (…and much larger) memory settings in the.yml file.

The “omero admin jvmcfg” command does not appear to be available to me in the docker container. Is my .yml syntax correct? Is there a way to validate and confirm the memory blitz is using in there? Thanks.

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
	at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:126)
	at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:112)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:647)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:310)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:298)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:286)
	at loci.common.RandomAccessInputStream.read(RandomAccessInputStream.java:681)
	at loci.formats.tiff.TiffParser.getIFDValue(TiffParser.java:566)
	at loci.formats.tiff.TiffParser.fillInIFD(TiffParser.java:526)
	at loci.formats.in.SEQReader.isThisType(SEQReader.java:89)
	at loci.formats.FormatReader.isThisType(FormatReader.java:612)
	at loci.formats.in.SEQReader.isThisType(SEQReader.java:79)
	at loci.formats.ImageReader.isThisType(ImageReader.java:864)
	at loci.formats.ImageReader.getReader(ImageReader.java:193)
	at loci.formats.ImageReader.setId(ImageReader.java:844)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:690)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.services.blitz.repo.CheckedPath.bfSetId(CheckedPath.java:412)
	at ome.services.blitz.repo.ManagedImportRequestI.open(ManagedImportRequestI.java:763)
	at ome.services.blitz.repo.ManagedImportRequestI.init(ManagedImportRequestI.java:245)
	at omero.cmd.HandleI.steps(HandleI.java:438)
	at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
	at omero.cmd.HandleI$2.doWork(HandleI.java:383)
	at omero.cmd.HandleI$2.doWork(HandleI.java:380)
, message=Java heap space}

	at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
	at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
	at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
	at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.lang.Thread.run(Thread.java:748)

	at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:615)
	at ome.formats.importer.ImportLibrary.notifyObservers(ImportLibrary.java:297)
	at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:813)
	at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
	at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
	at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
	at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:126)
	at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:112)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:647)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:310)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:298)
	at loci.common.NIOFileHandle.read(NIOFileHandle.java:286)
	at loci.common.RandomAccessInputStream.read(RandomAccessInputStream.java:681)
	at loci.formats.tiff.TiffParser.getIFDValue(TiffParser.java:566)
	at loci.formats.tiff.TiffParser.fillInIFD(TiffParser.java:526)
	at loci.formats.in.SEQReader.isThisType(SEQReader.java:89)
	at loci.formats.FormatReader.isThisType(FormatReader.java:612)
	at loci.formats.in.SEQReader.isThisType(SEQReader.java:79)
	at loci.formats.ImageReader.isThisType(ImageReader.java:864)
	at loci.formats.ImageReader.getReader(ImageReader.java:193)
	at loci.formats.ImageReader.setId(ImageReader.java:844)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:690)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.services.blitz.repo.CheckedPath.bfSetId(CheckedPath.java:412)
	at ome.services.blitz.repo.ManagedImportRequestI.open(ManagedImportRequestI.java:763)
	at ome.services.blitz.repo.ManagedImportRequestI.init(ManagedImportRequestI.java:245)
	at omero.cmd.HandleI.steps(HandleI.java:438)
	at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
	at omero.cmd.HandleI$2.doWork(HandleI.java:383)
	at omero.cmd.HandleI$2.doWork(HandleI.java:380)
, message=Java heap space}

	at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
	... 11 more

HI @blasky, assuming you are using or extending the openmicroscopy/omero-server image, the omero executable should be under /opt/omero/server/OMERO.server/bin:

[sbesson@idr0-slot3 docker-example-omero]$ sudo docker-compose up -d
Recreating docker-example-omero_omeroserver_1 ... 
Recreating docker-example-omero_omeroserver_1 ... done
Recreating docker-example-omero_omeroweb_1    ... done
[sbesson@idr0-slot3 docker-example-omero]$ sudo docker exec -it 0c70c8707333 bash
bash-4.2$ /opt/omero/server/OMERO.server/bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions