Andor Dragonfly IMS files with Z-stack failing to open with BioFormats 5.9.2

Somewhat related to this legacy thread, I just had a user try to push data from an Andor Draonfly (running Fusion 2.1) onto an OMERO 5.4.9 server and while single slice data uploads fine, I’m getting failures with anything that had ‘Z’ slices.

I uploaded an example file to OME QA this morning with the exception in a text file.

Exception follows:

java.io.IOException: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.iosp.hdf5.H5header$MessageDataspace.read(H5header.java:2826)
at ucar.nc2.iosp.hdf5.H5header$MessageAttribute.read(H5header.java:3683)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2364)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2332)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
… 32 more
at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
at ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
… 16 more
Caused by: java.lang.IllegalStateException: MessageDataspace: unknown version= 0
at ucar.nc2.iosp.hdf5.H5header$MessageDataspace.read(H5header.java:2826)
at ucar.nc2.iosp.hdf5.H5header$MessageAttribute.read(H5header.java:3683)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2364)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2332)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
… 32 more

I get the same type of issue opening them with an updated Fiji. Can confirm that Imaris (9.2.1) or Fusion (2.1) open the images without a problem.

Hi Dave,

We also have a Dragonfly and Andor told us for files to open in FIJI we need to download the latest NetCDF java and put it in the jars folder https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp

This has always worked fine until a recent Fusion update we installed last month.

Currently, if NetCDFAll-4.6.10 OR 4.6.12 are installed, I can open the Dragonfly .ims files in FIJI (although it reads the LUTs incorrectly) but only on Windows machines (both 7 and 10). I have yet to get it to work on a Mac and I’ve tried a few different things. Not tried on anything Linux based.

1 Like

Thanks for the tip. I seem to remember there being such a suggestion. Any idea of a workaround for OMERO?

Are you running 2.1 ?

Any idea of a workaround for OMERO

Sorry, not at all. Didn’t mean to get your hopes that I was being helpful, I meant to just add more information really. At the moment I’m focusing on try to get them to work on the Macs we have here.

Honestly, we’ve had the Dragonfly almost two years and having the files run in FIJI has always been non-trivial and I’ve never attempted it with OMERO.

Yes we are running Fusion 2.1.

1 Like

Not sure if that would work, but for Slidebook files we have a similar situation (a custom jar file needs to be in the jars folder for Fiji to open them), and the OMERO fix is simply adding that jar server-side to lib/server. Though the issues might be different, it’s probably still worth a try!

2 Likes

Thanks for the suggestion @erickratamero. Might give that a shot.

Thanks for reporting this, @dnmason. We are aware that a number of IMS files require a newer version of netcdf; we do plan to upgrade the version of that Bio-Formats uses as outlined here:

Unfortunately, previous attempts to upgrade caused issues in OMERO, as noted here:

I don’t know that we would recommend attempting to upgrade netcdf in OMERO by hand - the potential for something to go unexpectedly wrong is fairly high. We will update the Trello card linked above with any progress, though.

Thanks @melissa I’ll keep an eye on the Trello board.

Ha! Much as I love a challenge, I don’t think this is something I will be attempting on the last day before break up for Christmas.

1 Like

Just a quick update on this issue (cc @lmurphy) in speaking with Andor, it turns out the Dragonfly IMS format is slightly different to the Imaris IMS format.

The workaround which has been 100% successful (albeit in limited trials), is to use the Imaris File Converter to batch convert the Dragonfly files into IMS files. These then open in Fiji and can be imported into OMERO with no software patching required.

Oh great, I knew that the file formats differed and in the past have had the idea to convert but when I’ve tried it didn’t convert anything. I assumed that the .ims extension meant that the File Converter software wasn’t recognising it as needed converted.

If you’ve managed thought then maybe the newer versions of the file converter have started allowing conversion.

Thanks a lot for the update.

1 Like

I have only tried this with 9.2.1, you may be right about earlier versions not working.

1 Like

Great, I haven’t tried since maybe 8.4.x.

You may not have come across this problem yet but in case you (or interested readers) do - once Dragonfly .ims files become bigger than around 60GB they can’t be opened in FIJI. I thought maybe the above fix would also work for this but alas I just tested it and it doesn’t seem to.

Dear All,

We have discovered the source of the IMS file opening problems in Fiji and it relates to the way meta data is stored in the HDF5 file. We were exceeding a size limit that exists in the NetCDF library, but is not actually defined in the HDF5 specification. We have implemented two changes to correct for this.
First we have relocated Protocol metadata to a different field type with greater capacity. Second, we have given the user an option to avoid time stamping every frame acquired, which can also exceed the metadata field capacity. The default for the time stamping is off, so that all files we have tested now open in Fiji and the virtual stack model works correctly. We release these fixes and some significant improvements to focus drift correction functionality (PFS, AFC and ZDC) in Fusion 2.1.0.80 which is now available for download on MyAndor (http://my.andor.com/user/).
For those who have existing data that needs to be imported to Fiji, we recommend one of the following:

  1. Load the file back into Fusion and Export to OME TIFF.
  2. Download the HDF5 Viewer (https://www.hdfgroup.org/downloads/hdfview/) and load the file and edit the headers by removing the time stamps and/or Protocol metadata.
    Please get in touch if you have further questions.
2 Likes