Can't open files acquired at MetaXpress

lguerard · April 10, 2019, 3:25pm

Hi all,

We’re trying to automate things at the MetaXpress microscope but I’m facing issues with opening the images using Bio-Formats.

It fails when opening the HTD files in Fiji (using the Bio-Formats plugin) but it works when using one of the two tif files. Do you have any ideas why ?

Here is the error I’m getting:

loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1)
	at loci.formats.FormatTools.checkPlaneNumber(FormatTools.java:1009)
	at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:997)
	at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:289)
	at loci.formats.in.MetamorphReader.openBytes(MetamorphReader.java:285)
	at loci.formats.in.CellWorxReader.openBytes(CellWorxReader.java:201)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:161)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:233)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
	at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
	at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
	at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
	at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
	at loci.plugins.in.Importer.readPixels(Importer.java:149)
	at loci.plugins.in.Importer.run(Importer.java:86)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:229)
	at ij.IJ.runPlugIn(IJ.java:193)
	at ij.Executer.runCommand(Executer.java:137)
	at ij.Executer.run(Executer.java:66)
	at ij.IJ.run(IJ.java:309)
	at ij.IJ.run(IJ.java:320)
	at ij.macro.Functions.doRun(Functions.java:623)
	at ij.macro.Functions.doFunction(Functions.java:97)
	at ij.macro.Interpreter.doStatement(Interpreter.java:270)
	at ij.macro.Interpreter.doStatements(Interpreter.java:256)
	at ij.macro.Interpreter.run(Interpreter.java:152)
	at ij.macro.Interpreter.run(Interpreter.java:91)
	at ij.macro.MacroRunner.run(MacroRunner.java:139)
	at java.lang.Thread.run(Thread.java:748)

Thanks a lot.

KaisMagicPlate_plt-1_ts-002583D7x03873418.zip (1.2 MB)

joshmoore · April 10, 2019, 3:38pm

Hi @lguerard

Can you explain more about what failure you are seeing? Optimal is if you can include the error output here.

Thanks,
~Josh

lguerard · April 10, 2019, 3:25pm

My bad ! I edited the post with the error from Fiji !

imagejan · April 10, 2019, 3:26pm

From the HTD file you posted:

"TimePoints", 2

"XWells", 3

"YWells", 2

… but you only provide 2 tiff files. It seems the dataset is incomplete, no?

lguerard · April 10, 2019, 3:48pm

Hi @imagejan,

Thanks for pointing it out, the software is stupid enough to actually write the number of wells in X and Y for the plate not caring about how many you used. I will need to find a way to change that.

However, I just tried changing it to 1 and 1 and the issue is still there.

dgault · April 11, 2019, 4:03pm

Hi @lguerard, using the sample fileset you attached and the latest Bio-Formats (6.0.1) I was able to open the HTD file without the exception. What version are you currently using?

lguerard · April 12, 2019, 6:37am

Hi !

I have the exact same version somehow.

Do you open directly the HTD file in Fiji ? Because opening the images work fine, just not the HTD somehow…

dgault · April 12, 2019, 8:39am

Yeah I was able to open using the HTD directly. The error you are seeing suggests that it has perhaps been unsuccessful in locating the second file for some reason. Is there anything different in the file structure compared to the zipped folder? Are you using Windows, Mac etc?

lguerard · April 12, 2019, 8:48am

Not at all, it’s the same files in the same folder with the same name. I even tried redownloading the ZIP above to check and it’s the same.

I tried on Linux (Mint) and on Windows 7 with the same issue…

dgault · April 12, 2019, 2:37pm

I retested with the Bio-Formats command line tools (6.0.1 as well) and am now reproducing the exact same error as you had. I will try and further debug the problem to see what the cause is.

lguerard · April 23, 2019, 7:31am

Hi @dgault,

“Great” that you could reproduce the error ! Let me know when you have an idea on how to fix the issue !

lguerard · May 6, 2019, 8:04am

Hi @dgault

Sorry for the bump, did you have time to look into it ? I can provide more test data if you need. If you want me to test things, I can also do it since we’re actually making the HTD file ourselves.

Thanks a lot for your help

dgault · May 6, 2019, 4:53pm

HI @lguerard, I have had some further time to debug the issue but have not yet got to the bottom of it. Are you creating the HTD files from scratch yourself or are they being generated by software?

lguerard · May 6, 2019, 4:57pm

Hi,

Thanks a lot for looking into it.

Files are generated by us through the software. We know what must be inside so we did a script to generate it.

lguerard · May 15, 2019, 2:23pm

Hi @dgault,

By checking again in the whole pipeline that we have to save the file and generate the HTD file I managed to fix the issue. I guess there were some problems between the info in the metadata and the info in the images themselves.

Thanks for looking into it !

lguerard · May 15, 2019, 2:31pm

Hi @dgault,

Sorry for the triple (!) post but I managed to make it work with multiple sites in a well. However, if I try with only one well then it fails again. With the regular HTD in this new dataset, you should get loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1) and with the 2nd HTD there, java.lang.ArrayIndexOutOfBoundsException: 1 and the 3rd one java.lang.NullPointerException

Is there any documentation about the HTD file format that Bio-Formats is expecting ? I couldn’t find anything…

Thanks for your help !

test-plate_plt-1_ts-002583FAx03680781.zip (4.9 MB)

dgault · May 17, 2019, 4:06pm

Thanks @lguerard for the updated datasets, I will do some further testing with these on Monday. Im afraid there is no public spec for the HTD file, the code for parsing the metadata from the HTD though is here: https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/in/CellWorxReader.java#L306

s.besson · May 20, 2019, 7:25pm

In addition to @dgault’s answer pointing at the source code, a few public representative samples illustrating a few variants of the CellWorx file format support by Bio-Formats are available here. These are all plates published in IDR. If you are interested in any of them, you should equally be able to use Aspera to download them locally.

lguerard · May 21, 2019, 9:38am

Thanks ! I’ll try to have a look at the code and find what could be the issue but everything seems ok except that it doesn’t work with single sites

Thanks a lot @s.besson ! I had a look and everything looks the same in our HTD files, the only difference is that one site acquisitions don’t seem to be working somehow. Once I understand the issue, I’ll try to fix multiple Z in our HTD.

lguerard · May 22, 2019, 11:35am

It feels like Bio-formats can read the metadata

<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd">
<Plate Columns="6" ID="Plate:0" Name="test-plate" Rows="4">
<Well Column="0" ID="Well:0:0" Row="0"/>
<Well Column="1" ID="Well:0:1" Row="0"/>
<Well Column="2" ID="Well:0:2" Row="0"/>
<Well Column="3" ID="Well:0:3" Row="0"/>
<Well Column="4" ID="Well:0:4" Row="0"/>
<Well Column="5" ID="Well:0:5" Row="0"/>
<Well Column="0" ID="Well:0:6" Row="1"/>
<Well Column="1" ID="Well:0:7" Row="1"/>
<Well Column="2" ID="Well:0:8" Row="1"/>
<Well Column="3" ID="Well:0:9" Row="1"/>
<Well Column="4" ID="Well:0:10" Row="1"/>
<Well Column="5" ID="Well:0:11" Row="1"/>
<Well Column="0" ID="Well:0:12" Row="2"/>
<Well Column="1" ID="Well:0:13" Row="2"/>
<Well Column="2" ID="Well:0:14" Row="2"/>
<Well Column="3" ID="Well:0:15" Row="2">
<WellSample ID="WellSample:0:15:0" Index="0">
<ImageRef ID="Image:0"/>
<Well Column="4" ID="Well:0:16" Row="2">
<WellSample ID="WellSample:0:16:0" Index="1">
<ImageRef ID="Image:1"/>
<Well Column="5" ID="Well:0:17" Row="2"/>
<Well Column="0" ID="Well:0:18" Row="3"/>
<Well Column="1" ID="Well:0:19" Row="3"/>
<Well Column="2" ID="Well:0:20" Row="3"/>
<Well Column="3" ID="Well:0:21" Row="3"/>
<Well Column="4" ID="Well:0:22" Row="3"/>
<Well Column="5" ID="Well:0:23" Row="3"/>
<PlateAcquisition ID="PlateAcquisition:0:0" MaximumFieldCount="1">
<WellSampleRef ID="WellSample:0:15:0"/>
<WellSampleRef ID="WellSample:0:16:0"/>
<Instrument ID="Instrument:0">
<Detector ID="Detector:0:0" Type="Other"/>
<Objective ID="Objective:0:0" LensNA="0.45"/>
<Image ID="Image:0" Name="Well C04 Field #1">
<AcquisitionDate>2019-05-15T15:52:30.617</AcquisitionDate>
<Description/>
<InstrumentRef ID="Instrument:0"/>
<ObjectiveSettings ID="Objective:0:0" RefractiveIndex="1.0"/>
<ImagingEnvironment Temperature="-29.95" TemperatureUnit="°C"/>
<Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:0" Interleaved="false" PhysicalSizeX="0.325" PhysicalSizeXUnit="µm" PhysicalSizeY="0.325" PhysicalSizeYUnit="µm" SignificantBits="16" SizeC="2" SizeT="2" SizeX="1392" SizeY="1040" SizeZ="1" Type="uint16">
<Channel ID="Channel:0:0" Name="RFP" SamplesPerPixel="1">
<DetectorSettings Binning="1x1" ID="Detector:0:0"/>
<LightPath/>
<Channel ID="Channel:0:1" Name="GFP" SamplesPerPixel="1">
<LightPath/>
<MetadataOnly/>
<Plane DeltaT="0.0" DeltaTUnit="s" ExposureTime="0.01" ExposureTimeUnit="s" PositionX="41380.0" PositionXUnit="reference frame" PositionY="29240.0" PositionYUnit="reference frame" PositionZ="0.0" PositionZUnit="reference frame" TheC="0" TheT="0" TheZ="0"/>
<Image ID="Image:1" Name="Well C05 Field #1">
<AcquisitionDate>2019-05-15T15:52:30.617</AcquisitionDate>
<Description/>
<InstrumentRef ID="Instrument:0"/>
<ObjectiveSettings ID="Objective:0:0" RefractiveIndex="1.0"/>
<ImagingEnvironment Temperature="-29.95" TemperatureUnit="°C"/>
<Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:1" Interleaved="false" PhysicalSizeX="0.325" PhysicalSizeXUnit="µm" PhysicalSizeY="0.325" PhysicalSizeYUnit="µm" SignificantBits="16" SizeC="2" SizeT="2" SizeX="1392" SizeY="1040" SizeZ="1" Type="uint16">
<Channel ID="Channel:1:0" Name="RFP" SamplesPerPixel="1">
<DetectorSettings Binning="1x1" ID="Detector:0:0"/>
<LightPath/>
<Channel ID="Channel:1:1" Name="GFP" SamplesPerPixel="1">
<LightPath/>
<MetadataOnly/>
<Plane DeltaT="0.0" DeltaTUnit="s" ExposureTime="0.01" ExposureTimeUnit="s" PositionX="41380.0" PositionXUnit="reference frame" PositionY="29240.0" PositionYUnit="reference frame" PositionZ="0.0" PositionZUnit="reference frame" TheC="0" TheT="0" TheZ="0"/>

but there seems to be something wrong with the series, even the thumbnails for showing the different series don’t work somehow…

Here is the complete error I’m getting with Bio-formats 6.1.1, hope that helps.

Warning: Error loading thumbnail for series #1
Warning: Error loading thumbnail for series #2
loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1)
	at loci.formats.FormatTools.checkPlaneNumber(FormatTools.java:1012)
	at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:1000)
	at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:289)
	at loci.formats.in.MetamorphReader.openBytes(MetamorphReader.java:285)
	at loci.formats.in.CellWorxReader.openBytes(CellWorxReader.java:224)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:161)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:233)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
	at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
	at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
	at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
	at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
	at loci.plugins.in.Importer.readPixels(Importer.java:149)
	at loci.plugins.in.Importer.run(Importer.java:86)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:229)
	at ij.IJ.runPlugIn(IJ.java:193)
	at ij.Executer.runCommand(Executer.java:137)
	at ij.Executer.run(Executer.java:66)
	at ij.IJ.run(IJ.java:309)
	at ij.IJ.run(IJ.java:320)
	at ij.macro.Functions.doRun(Functions.java:623)
	at ij.macro.Functions.doFunction(Functions.java:97)
	at ij.macro.Interpreter.doStatement(Interpreter.java:270)
	at ij.macro.Interpreter.doStatements(Interpreter.java:256)
	at ij.macro.Interpreter.run(Interpreter.java:152)
	at ij.macro.Interpreter.run(Interpreter.java:91)
	at ij.macro.MacroRunner.run(MacroRunner.java:139)
	at java.lang.Thread.run(Thread.java:748)