Bug in bio-format reader when grouping Cellsense (.vsi) files

Hi,

I am trying to load a series of .vsi images generated by Olympus Cellsense using the bio-format reader. Particularly, I need to use the “group files with similar names”. The reader loads only the first image although it detects the rest of the images (as seen by the <0-xxx> indication in the window name). There is an issue with reading a specific tag in the first .vsi/.ets file that I suspect creates this problem.

Help will be deeply appreciated!

Here is the stack:

out> dataFieldOffset = 5914846652683978913

out> flags = 858058961

out> Failed to read all tags
java.lang.IllegalArgumentException: null
at java.nio.Buffer.position(Buffer.java:244)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:650)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:330)
at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:203)
at loci.formats.in.CellSensReader.readTags(CellSensReader.java:1888)
at loci.formats.in.CellSensReader.initFile(CellSensReader.java:647)
at loci.formats.FormatReader.setId(FormatReader.java:1389)
at loci.formats.ImageReader.setId(ImageReader.java:843)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.DimensionSwapper.setId(DimensionSwapper.java:293)
at loci.formats.FileStitcher.initFile(FileStitcher.java:959)
at loci.formats.FileStitcher.setId(FileStitcher.java:925)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.DimensionSwapper.setId(DimensionSwapper.java:293)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.plugins.in.ImportProcess.setId(ImportProcess.java:701)
at loci.plugins.in.ImportProcess.initializeStack(ImportProcess.java:547)
at loci.plugins.in.ImportProcess.execute(ImportProcess.java:146)
at loci.plugins.in.Importer.showDialogs(Importer.java:140)
at loci.plugins.in.Importer.run(Importer.java:76)
at loci.plugins.LociImporter.run(LociImporter.java:78)
at ij.IJ.runUserPlugIn(IJ.java:230)
at ij.IJ.runPlugIn(IJ.java:193)
at ij.IJ.runPlugIn(IJ.java:182)
at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:189)

Thank you for your report, I’m not aware that we’ve seen this one before, perhaps there’s a new format variant. Could you provide us a sample image by uploading to http://qa.openmicroscopy.org.uk/qa/upload/ or any other means? That will help us to look into what’s going on with your data. (Which model of microscope is it from?) Please do let us know if we must keep your sample image(s) confidential or if it is okay to share publicly.

Thanks for answering to this issue. I have uploaded 4 .vsi files which point to a set of .ets larger files (whiting a specific file organization hierarchy). The OME uploaded didn’t allow me to upload the larger files while preserving the heirarchy so I provided a link to a Dropbox share. Please let me know if it does not work and I’ll upload separated files and provide details about the file hierarchy.
Best
Pablo

Thank you, the zip in the Dropbox share worked fine, we now have what you describe saved locally in our scratch/inbox/gh-3438/ for further investigation. At first glance I’m not having any trouble reading the images, which version of Bio-Formats are you using? The latest is v6.2.1.

Version 6.2.1

No problem reading single image. Did you manage to load with the following settings:

-Grouping by image name
-Split channels
-Virtual stack

Even when I do not select stitching. I am granted with the following:

Then only the first file of the series is loaded. Notice the <1-4>:

Hi @pbl007, the message displayed is not entirely clear (“File Stitching Options are not available for this format”), but this relates to the Group files by name option rather than tile stitching.

When the Group files by name option is selected Bio-Formats will try to use its File Stitcher to locate other related files. However some formats have their own specific groupings and in these cases the specific format reader handles the grouping rather than the File Stitcher. For these formats the file grouping option in FIJI will not be able to group the additional files by default.

Depending on what you are doing it may be possible to write a macro that opens each file first and then finds a way to combine them.

Hello @dgault, thanks for looking into this.

It looks like the grouping alg (whether the Bio-format File Stitcher or the vsi file reader) are grouping OK as the number of files (four in the case that I uploaded) are being detected (i.e. we can see the <1-4>). The problem seems to be an error reading some Tiff TAG based on the error stack I posted. Will be great to have this issue solved from first principles but a macro will be also a great workaround. The issue is that we need to preserve the metadata associated with each image as it is needed downstream the pipeline for image alignment and registration.

The <1-4> in the image title is a little misleading in this case. Internally the FileStitcher has attempted to use that file pattern as the ID which is why it is being displayed, however in reality it is never being used by the CellSensReader as this reader has its own logic for grouping filesets.

The Tag exception is unrelated but perhaps worth looking into further, I will try and debug it to see exactly what the cause is. The majority of metadata appears to be read successfully though.

In the short term the macro option might make the most sense depending on what you are doing. A good starting example of using the Bio-Formats library in jython is https://gist.github.com/ctrueden/6282856, Im happy to try to assist with using Bio-Formats in a macro.

Is it important to maintain the data as CellSens VSI files or would converting to TIFF for example be an option?

No problem to work with TIFF files. This issue will be to preserve the metadata associated with each vsi file.
Thanks again for looking into this!

As an update, I have started putting together a script to combine the series of files as a single OME-TIFF for easier reading: https://github.com/dgault/bio-formats-examples/blob/jython-cmbine-vsi/src/main/FIJI/combineVSI.py

In its current state it will combine the core metadata and pixel data, I still need to further work on ensuring all the original metadata from each file is included. I should hopefully have that finished tomorrow

Wow! Looks like a great workaround!

The sample script I linked above has been updated with the extra metadata and now includes a section at the bottom to read in the combined file and split the channels before displaying the images for each series.

Fantastic! Let us check and see if we manage to get the entire pipeline working. HUGE HELP!!