Import HDF5/H5 files to omero

Hello Omero team,

Out of curiosity, why is it that one can’t upload HDF5 or .h5 files to omero?
Is it because the feature is not developed yet or because there is something intrinsically different in the way data in stored in a .h5 versus say .tif ?

I like the fact that the size on disk of .h5 files seem to be significantly smaller than .tif
For example I have 1095 files .h5 for 1.77GB but the same files in .tif are 16.9 GB. Why is that ?

Thank you

Hi @LPUoO,

When it comes to HDF5, the answer is often “it depends”. Since TIFFs were initially created as an image format, just about any TIFF will be importable to OMERO somehow. It may not always do so as you expect, but it is fundamentally an image.

HDF5 files can be anything. They are designed for storing, most broadly, “hierarchical collections of multidimensional arrays with metadata”. One could argue OMERO should try to interpret every array as in image, but there’s just not been a driver for that. Instead there are a number of dialects of HDF5 that we support.

Where did the HDF5 and the TIFF come from? How were they created?

That depends again. One possibility is that a better compression setting was used when creating the HDF5.

~Josh

2 Likes

On that score https://github.com/ome/bioformats/pull/2473#issuecomment-676089759 may be of interest: generation settings greatly affect resulting TIFF size.

3 Likes

Seconding the answers mentioned above. For information, below is the list of HDF-based image file formats that should be detected and supported by the current version of Bio-Formats 6.5.1 and hence OMERO:

2 Likes

Hi,

I’m reviving this discussion because I experience troubles when trying to import BigDataViewer files (.h5 and .xml) to OMERO.
The upload worked correctly but the import failed during the processing phase (metadata importation)


I got the following error message:

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.NullPointerException
at ome.formats.model.ChannelProcessor.process(ChannelProcessor.java:477)
at ome.formats.OMEROMetadataStoreClient.postProcess(OMEROMetadataStoreClient.java:1849)
at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:647)
at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:523)
at omero.cmd.HandleI.steps(HandleI.java:448)
at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
at omero.cmd.HandleI$2.doWork(HandleI.java:383)
at omero.cmd.HandleI$2.doWork(HandleI.java:380)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy81.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:447)
at omero.cmd.HandleI.run(HandleI.java:379)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at ome.services.util.Executor$Impl$1.call(Executor.java:488)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
, message=}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
at IceInternal.Incoming.invoke(Incoming.java:221)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
at Ice.ConnectionI.message(ConnectionI.java:1056)
at IceInternal.ThreadPool.run(ThreadPool.java:395)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
at java.lang.Thread.run(Thread.java:748)

at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:615)
at ome.formats.importer.ImportLibrary.notifyObservers(ImportLibrary.java:297)
at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:813)
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
at IceInternal.Incoming.invoke(Incoming.java:221)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
at Ice.ConnectionI.message(ConnectionI.java:1056)
at IceInternal.ThreadPool.run(ThreadPool.java:395)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.NullPointerException
at ome.formats.model.ChannelProcessor.process(ChannelProcessor.java:477)
at ome.formats.OMEROMetadataStoreClient.postProcess(OMEROMetadataStoreClient.java:1849)
at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:647)
at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:523)
at omero.cmd.HandleI.steps(HandleI.java:448)
at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
at omero.cmd.HandleI$2.doWork(HandleI.java:383)
at omero.cmd.HandleI$2.doWork(HandleI.java:380)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy81.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:447)
at omero.cmd.HandleI.run(HandleI.java:379)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at ome.services.util.Executor$Impl$1.call(Executor.java:488)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
, message=}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
... 11 more

Consequently, I tried to import public sample images from the BioFormat website:
While the “drosophila” data import to our OMERO server worked seamlessly, the HisYFP-SPIM data failed during the import of metadata, in the same way as my own BigDataViewer data, and with the same “java NullPointerException” error message.

The two failing data can both be opened in FIJI with the Bioformat plugin.

Is there any explanation why some BigDataViewer files can be imported on our OMERO server while others can’t?

Many thanks,
Claire

Hi Claire,

thanks for sending us the complete stack trace and pointing us to a public sample BDV file allowing to reproduce. Using one of the production OMERO servers at the University of Dundee, I was unable to get a similar error (using the command-line import).

I might give the Desktop client a try. In the meantime, it might be useful to know the version of OMERO.server and OMERO.insight you are using to know if it corresponds to an obvious bug which could have been fixed in recent versions of the sofware.

Best,
Sebastien

1 Like

Hi Sebastien,

Thank you very much for your answer.
We have an up-to-date OMERO.server (version 5.6.3), and I did the imports with the OMERO.insight client version 5.5.12. I just installed OMERO.insight 5.5.14 and tried to import the same HisYFP-SPIM dataset, but I got exactly the same error as with version 5.5.12.

Best,
Claire

I tested the BDV files with Insight. I could import them without seeing an error message. However no thumbnails were generated. I created an issue on github: Thumbnail generation fails for BDV images · Issue #121 · ome/omero-server · GitHub

Regards,
Dominik

2 Likes