Andor Dragonfly IMS files with Z-stack failing to open with BioFormats 5.9.2

bio-formats
omero

#1

Somewhat related to this legacy thread, I just had a user try to push data from an Andor Draonfly (running Fusion 2.1) onto an OMERO 5.4.9 server and while single slice data uploads fine, I’m getting failures with anything that had ‘Z’ slices.

I uploaded an example file to OME QA this morning with the exception in a text file.

Exception follows:

java.io.IOException: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.iosp.hdf5.H5header$MessageDataspace.read(H5header.java:2826)
at ucar.nc2.iosp.hdf5.H5header$MessageAttribute.read(H5header.java:3683)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2364)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2332)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
… 32 more
at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
at ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
… 16 more
Caused by: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.iosp.hdf5.H5header$MessageDataspace.read(H5header.java:2826)
at ucar.nc2.iosp.hdf5.H5header$MessageAttribute.read(H5header.java:3683)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2364)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2332)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
… 32 more

I get the same type of issue opening them with an updated Fiji. Can confirm that Imaris (9.2.1) or Fusion (2.1) open the images without a problem.


#2

Hi Dave,

We also have a Dragonfly and Andor told us for files to open in FIJI we need to download the latest NetCDF java and put it in the jars folder https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp

This has always worked fine until a recent Fusion update we installed last month.

Currently, if NetCDFAll-4.6.10 OR 4.6.12 are installed, I can open the Dragonfly .ims files in FIJI (although it reads the LUTs incorrectly) but only on Windows machines (both 7 and 10). I have yet to get it to work on a Mac and I’ve tried a few different things. Not tried on anything Linux based.


#3

Thanks for the tip. I seem to remember there being such a suggestion. Any idea of a workaround for OMERO?

Are you running 2.1 ?


#4

Any idea of a workaround for OMERO

Sorry, not at all. Didn’t mean to get your hopes that I was being helpful, I meant to just add more information really. At the moment I’m focusing on try to get them to work on the Macs we have here.

Honestly, we’ve had the Dragonfly almost two years and having the files run in FIJI has always been non-trivial and I’ve never attempted it with OMERO.

Yes we are running Fusion 2.1.


#5

Not sure if that would work, but for Slidebook files we have a similar situation (a custom jar file needs to be in the jars folder for Fiji to open them), and the OMERO fix is simply adding that jar server-side to lib/server. Though the issues might be different, it’s probably still worth a try!


#6

Thanks for the suggestion @erickratamero. Might give that a shot.


#7

Thanks for reporting this, @dnmason. We are aware that a number of IMS files require a newer version of netcdf; we do plan to upgrade the version of that Bio-Formats uses as outlined here:

Unfortunately, previous attempts to upgrade caused issues in OMERO, as noted here:

I don’t know that we would recommend attempting to upgrade netcdf in OMERO by hand - the potential for something to go unexpectedly wrong is fairly high. We will update the Trello card linked above with any progress, though.


#8

Thanks @melissa I’ll keep an eye on the Trello board.

Ha! Much as I love a challenge, I don’t think this is something I will be attempting on the last day before break up for Christmas.


#9

Just a quick update on this issue (cc @lmurphy) in speaking with Andor, it turns out the Dragonfly IMS format is slightly different to the Imaris IMS format.

The workaround which has been 100% successful (albeit in limited trials), is to use the Imaris File Converter to batch convert the Dragonfly files into IMS files. These then open in Fiji and can be imported into OMERO with no software patching required.


#10

Oh great, I knew that the file formats differed and in the past have had the idea to convert but when I’ve tried it didn’t convert anything. I assumed that the .ims extension meant that the File Converter software wasn’t recognising it as needed converted.

If you’ve managed thought then maybe the newer versions of the file converter have started allowing conversion.

Thanks a lot for the update.


#11

I have only tried this with 9.2.1, you may be right about earlier versions not working.


#12

Great, I haven’t tried since maybe 8.4.x.

You may not have come across this problem yet but in case you (or interested readers) do - once Dragonfly .ims files become bigger than around 60GB they can’t be opened in FIJI. I thought maybe the above fix would also work for this but alas I just tested it and it doesn’t seem to.