Can't open files acquired at MetaXpress

Hi all,

We’re trying to automate things at the MetaXpress microscope but I’m facing issues with opening the images using Bio-Formats.

It fails when opening the HTD files in Fiji (using the Bio-Formats plugin) but it works when using one of the two tif files. Do you have any ideas why ?

Here is the error I’m getting:

loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1)
	at loci.formats.FormatTools.checkPlaneNumber(FormatTools.java:1009)
	at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:997)
	at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:289)
	at loci.formats.in.MetamorphReader.openBytes(MetamorphReader.java:285)
	at loci.formats.in.CellWorxReader.openBytes(CellWorxReader.java:201)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:161)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:233)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
	at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
	at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
	at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
	at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
	at loci.plugins.in.Importer.readPixels(Importer.java:149)
	at loci.plugins.in.Importer.run(Importer.java:86)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:229)
	at ij.IJ.runPlugIn(IJ.java:193)
	at ij.Executer.runCommand(Executer.java:137)
	at ij.Executer.run(Executer.java:66)
	at ij.IJ.run(IJ.java:309)
	at ij.IJ.run(IJ.java:320)
	at ij.macro.Functions.doRun(Functions.java:623)
	at ij.macro.Functions.doFunction(Functions.java:97)
	at ij.macro.Interpreter.doStatement(Interpreter.java:270)
	at ij.macro.Interpreter.doStatements(Interpreter.java:256)
	at ij.macro.Interpreter.run(Interpreter.java:152)
	at ij.macro.Interpreter.run(Interpreter.java:91)
	at ij.macro.MacroRunner.run(MacroRunner.java:139)
	at java.lang.Thread.run(Thread.java:748)

Thanks a lot.

KaisMagicPlate_plt-1_ts-002583D7x03873418.zip (1.2 MB)

Hi @lguerard

Can you explain more about what failure you are seeing? Optimal is if you can include the error output here.

Thanks,
~Josh

My bad ! I edited the post with the error from Fiji !

From the HTD file you posted:

"TimePoints", 2

"XWells", 3

"YWells", 2

… but you only provide 2 tiff files. It seems the dataset is incomplete, no?

Hi @imagejan,

Thanks for pointing it out, the software is stupid enough to actually write the number of wells in X and Y for the plate not caring about how many you used. I will need to find a way to change that.

However, I just tried changing it to 1 and 1 and the issue is still there.

Hi @lguerard, using the sample fileset you attached and the latest Bio-Formats (6.0.1) I was able to open the HTD file without the exception. What version are you currently using?

Hi !

I have the exact same version somehow.

image

Do you open directly the HTD file in Fiji ? Because opening the images work fine, just not the HTD somehow… :confused:

Yeah I was able to open using the HTD directly. The error you are seeing suggests that it has perhaps been unsuccessful in locating the second file for some reason. Is there anything different in the file structure compared to the zipped folder? Are you using Windows, Mac etc?

Not at all, it’s the same files in the same folder with the same name. I even tried redownloading the ZIP above to check and it’s the same.

I tried on Linux (Mint) and on Windows 7 with the same issue…

I retested with the Bio-Formats command line tools (6.0.1 as well) and am now reproducing the exact same error as you had. I will try and further debug the problem to see what the cause is.

2 Likes

Hi @dgault,

“Great” that you could reproduce the error ! Let me know when you have an idea on how to fix the issue ! :slight_smile:

Hi @dgault

Sorry for the bump, did you have time to look into it ? I can provide more test data if you need. If you want me to test things, I can also do it since we’re actually making the HTD file ourselves.

Thanks a lot for your help :slight_smile:

HI @lguerard, I have had some further time to debug the issue but have not yet got to the bottom of it. Are you creating the HTD files from scratch yourself or are they being generated by software?

Hi,

Thanks a lot for looking into it.

Files are generated by us through the software. We know what must be inside so we did a script to generate it.

Hi @dgault,

By checking again in the whole pipeline that we have to save the file and generate the HTD file I managed to fix the issue. I guess there were some problems between the info in the metadata and the info in the images themselves.

Thanks for looking into it !

Hi @dgault,

Sorry for the triple (!) post but I managed to make it work with multiple sites in a well. However, if I try with only one well then it fails again. With the regular HTD in this new dataset, you should get loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1) and with the 2nd HTD there, java.lang.ArrayIndexOutOfBoundsException: 1 and the 3rd one java.lang.NullPointerException

Is there any documentation about the HTD file format that Bio-Formats is expecting ? I couldn’t find anything…

Thanks for your help !

test-plate_plt-1_ts-002583FAx03680781.zip (4.9 MB)

Thanks @lguerard for the updated datasets, I will do some further testing with these on Monday. Im afraid there is no public spec for the HTD file, the code for parsing the metadata from the HTD though is here: https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/in/CellWorxReader.java#L306

In addition to @dgault’s answer pointing at the source code, a few public representative samples illustrating a few variants of the CellWorx file format support by Bio-Formats are available here. These are all plates published in IDR. If you are interested in any of them, you should equally be able to use Aspera to download them locally.

Thanks ! I’ll try to have a look at the code and find what could be the issue but everything seems ok except that it doesn’t work with single sites :confused:

Thanks a lot @s.besson ! I had a look and everything looks the same in our HTD files, the only difference is that one site acquisitions don’t seem to be working somehow. Once I understand the issue, I’ll try to fix multiple Z in our HTD.

It feels like Bio-formats can read the metadata

<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd">
<Plate Columns="6" ID="Plate:0" Name="test-plate" Rows="4">
<Well Column="0" ID="Well:0:0" Row="0"/>
<Well Column="1" ID="Well:0:1" Row="0"/>
<Well Column="2" ID="Well:0:2" Row="0"/>
<Well Column="3" ID="Well:0:3" Row="0"/>
<Well Column="4" ID="Well:0:4" Row="0"/>
<Well Column="5" ID="Well:0:5" Row="0"/>
<Well Column="0" ID="Well:0:6" Row="1"/>
<Well Column="1" ID="Well:0:7" Row="1"/>
<Well Column="2" ID="Well:0:8" Row="1"/>
<Well Column="3" ID="Well:0:9" Row="1"/>
<Well Column="4" ID="Well:0:10" Row="1"/>
<Well Column="5" ID="Well:0:11" Row="1"/>
<Well Column="0" ID="Well:0:12" Row="2"/>
<Well Column="1" ID="Well:0:13" Row="2"/>
<Well Column="2" ID="Well:0:14" Row="2"/>
<Well Column="3" ID="Well:0:15" Row="2">
<WellSample ID="WellSample:0:15:0" Index="0">
<ImageRef ID="Image:0"/>
<Well Column="4" ID="Well:0:16" Row="2">
<WellSample ID="WellSample:0:16:0" Index="1">
<ImageRef ID="Image:1"/>
<Well Column="5" ID="Well:0:17" Row="2"/>
<Well Column="0" ID="Well:0:18" Row="3"/>
<Well Column="1" ID="Well:0:19" Row="3"/>
<Well Column="2" ID="Well:0:20" Row="3"/>
<Well Column="3" ID="Well:0:21" Row="3"/>
<Well Column="4" ID="Well:0:22" Row="3"/>
<Well Column="5" ID="Well:0:23" Row="3"/>
<PlateAcquisition ID="PlateAcquisition:0:0" MaximumFieldCount="1">
<WellSampleRef ID="WellSample:0:15:0"/>
<WellSampleRef ID="WellSample:0:16:0"/>
<Instrument ID="Instrument:0">
<Detector ID="Detector:0:0" Type="Other"/>
<Objective ID="Objective:0:0" LensNA="0.45"/>
<Image ID="Image:0" Name="Well C04 Field #1">
<AcquisitionDate>2019-05-15T15:52:30.617</AcquisitionDate>
<Description/>
<InstrumentRef ID="Instrument:0"/>
<ObjectiveSettings ID="Objective:0:0" RefractiveIndex="1.0"/>
<ImagingEnvironment Temperature="-29.95" TemperatureUnit="°C"/>
<Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:0" Interleaved="false" PhysicalSizeX="0.325" PhysicalSizeXUnit="µm" PhysicalSizeY="0.325" PhysicalSizeYUnit="µm" SignificantBits="16" SizeC="2" SizeT="2" SizeX="1392" SizeY="1040" SizeZ="1" Type="uint16">
<Channel ID="Channel:0:0" Name="RFP" SamplesPerPixel="1">
<DetectorSettings Binning="1x1" ID="Detector:0:0"/>
<LightPath/>
<Channel ID="Channel:0:1" Name="GFP" SamplesPerPixel="1">
<LightPath/>
<MetadataOnly/>
<Plane DeltaT="0.0" DeltaTUnit="s" ExposureTime="0.01" ExposureTimeUnit="s" PositionX="41380.0" PositionXUnit="reference frame" PositionY="29240.0" PositionYUnit="reference frame" PositionZ="0.0" PositionZUnit="reference frame" TheC="0" TheT="0" TheZ="0"/>
<Image ID="Image:1" Name="Well C05 Field #1">
<AcquisitionDate>2019-05-15T15:52:30.617</AcquisitionDate>
<Description/>
<InstrumentRef ID="Instrument:0"/>
<ObjectiveSettings ID="Objective:0:0" RefractiveIndex="1.0"/>
<ImagingEnvironment Temperature="-29.95" TemperatureUnit="°C"/>
<Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:1" Interleaved="false" PhysicalSizeX="0.325" PhysicalSizeXUnit="µm" PhysicalSizeY="0.325" PhysicalSizeYUnit="µm" SignificantBits="16" SizeC="2" SizeT="2" SizeX="1392" SizeY="1040" SizeZ="1" Type="uint16">
<Channel ID="Channel:1:0" Name="RFP" SamplesPerPixel="1">
<DetectorSettings Binning="1x1" ID="Detector:0:0"/>
<LightPath/>
<Channel ID="Channel:1:1" Name="GFP" SamplesPerPixel="1">
<LightPath/>
<MetadataOnly/>
<Plane DeltaT="0.0" DeltaTUnit="s" ExposureTime="0.01" ExposureTimeUnit="s" PositionX="41380.0" PositionXUnit="reference frame" PositionY="29240.0" PositionYUnit="reference frame" PositionZ="0.0" PositionZUnit="reference frame" TheC="0" TheT="0" TheZ="0"/>

but there seems to be something wrong with the series, even the thumbnails for showing the different series don’t work somehow…

Here is the complete error I’m getting with Bio-formats 6.1.1, hope that helps.

Warning: Error loading thumbnail for series #1
Warning: Error loading thumbnail for series #2
loci.formats.FormatException: Invalid image number: 1 (series=0, imageCount=1)
	at loci.formats.FormatTools.checkPlaneNumber(FormatTools.java:1012)
	at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:1000)
	at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:289)
	at loci.formats.in.MetamorphReader.openBytes(MetamorphReader.java:285)
	at loci.formats.in.CellWorxReader.openBytes(CellWorxReader.java:224)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:161)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:233)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
	at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
	at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
	at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
	at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
	at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
	at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
	at loci.plugins.in.Importer.readPixels(Importer.java:149)
	at loci.plugins.in.Importer.run(Importer.java:86)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:229)
	at ij.IJ.runPlugIn(IJ.java:193)
	at ij.Executer.runCommand(Executer.java:137)
	at ij.Executer.run(Executer.java:66)
	at ij.IJ.run(IJ.java:309)
	at ij.IJ.run(IJ.java:320)
	at ij.macro.Functions.doRun(Functions.java:623)
	at ij.macro.Functions.doFunction(Functions.java:97)
	at ij.macro.Interpreter.doStatement(Interpreter.java:270)
	at ij.macro.Interpreter.doStatements(Interpreter.java:256)
	at ij.macro.Interpreter.run(Interpreter.java:152)
	at ij.macro.Interpreter.run(Interpreter.java:91)
	at ij.macro.MacroRunner.run(MacroRunner.java:139)
	at java.lang.Thread.run(Thread.java:748)