Group files by pattern or reader

#1

Dear Ome-Team!

Was is the best way to group files in a way, that I have one handle object in OMERO after import for a multi files experiment?

In detail:
I have the following sequence of files:
_ch0_stack0000_000000msec_0114904622msecAbs.tif
_ch0_stack0001_002400msec_0114907022msecAbs.tif
…
_ch1_stack0000_000000msec_0114904622msecAbs.tif
_ch1_stack0001_002400msec_0114907022msecAbs.tif
…

where “ch<>” presents the channels and “stack<>” presents the timepoints. Each file contains a Z-stack.
I thought about 2 possible solutions:

  1. Import a pattern file:
    I know that there is the possibility to create a pattern file. But as far I understand the description under https://docs.openmicroscopy.org/bio-formats/6.1.1/formats/pattern-file.html I need “right character” for interpretion. So “stack” will not be interpreted as timepoints and I want to avoid renaming the files.
  2. Grouping files by a Bio-Format reader.
    I still have a special reader for this kind of files, but I don’t understand how can I group that inside the reader.
    Perhaps you know which existing reader has the same functionality, that would be very helpful.

Best,
Susanne

#2

Hi Susanne,

if all the files constituting your fileset are TIFF-based, an option might be to create an OME-TIFF fileset. In that case, OMERO would effectively detect and read the data using the OMETiffReader.

Since Bio-Formats 6.0.0, it is possible to have companion OME-XML files that map individual TIFF files without the requirement that these TIFF files should link back to the metadata file. We have not promoted and documented this feature externally but it has been extensively tested in the context of IDR to create rich representations of submissions - see here or here for published examples.

This files are obviously more verbose than pattern files but they should allow you to construct multi-dimensional datasets from arbitrary sets of TIFF files. A secondary advantage is the ability to include OME metadata like channels, instruments, planes and the support for multiple images.

If that was a solution fit for your purpose, happy to give additional pointers as needed,
Sebastien

#3

Hi Sebastien,
This is extremely useful. Is this the facility described here: https://docs.openmicroscopy.org/ome-model/6.0.0/ome-tiff/specification.html#partial-ome-xml-metadata and that file should have the .companion.ome extension? And if the tiff files themselves are actually ome.tif files, can some of the metadata actually come from those ome.tiffs? If so, what are the precedence rules in case of conflicting entries?
I’m looking at this facility (probably similar to what @sukunis would like to do) as a way to collect individual channel data one by one where each has its own channel-specific metadata and then use the .companion.ome file as a “wrapper” to bundle them and have that provide just the common metadata. As @sukunis we currently encode the per-channel metadata in the filename but that really doesn’t scale well, is difficult to parse properly, and is too subject to accidental changes.
Cheers,
Damir

#4

Hi Sebastian,

thank you for your answer.
I created and imported a companion.ome file to map my TIF-files. I’m having a little trouble with the right arrangement of the planes, but I am confident that I can solve that.
The default reader for the mapped TIF files seems to be the MinimalTiffReader, although when importing a single TIF, the customized reader is used. Is there a way to specify the reader to used for the mapped files? Do you create the companion file manually before importing it or it is also possible to create such a companion file in a custom Bio-Format reader? And call then the OMETIFReader for the created companion file?

Best,
Susanne

#5

Hi Susanne,

Is there a way to specify the reader to used for the mapped files? Is there a way to specify the reader to used for the mapped files?

At the moment, the underlying OME-TIFF reader used for reading the individual TIFF files is MinimalTiffReader indeed. What is your use case for using a customized TIFF reader? And where would you see how to specify the underlying reader to use?

Do you create the companion file manually before importing it or it is also possible to create such a companion file in a custom Bio-Format reader? And call then the OMETIFReader for the created companion file?

In the example above, the companion file is created manually to represent the multi-dimensional aspect of the data (which is usually missing from the primary data). Pointing the OMETiffReader e.g. through the OMERO importer to the companion file will detect and group all the files.
In the example above, the companion file were not created from custom Bio-Formats readers primarily because we needed to specify metadata that was not present in the original data. However, in the case a custom TIFF-based reader contains all the required metadata, it should be doable to create a serialized representation as an interoperable companion file which can then be loaded/re-used without the need of a customized reader

#6

Hi Damir,

Is this the facility described here: https://docs.openmicroscopy.org/ome-model/6.0.0/ome-tiff/specification.html#partial-ome-xml-metadata and that file should have the .companion.ome extension?

That’s right. We haven’t updated this section to promote the fact the TIFF files do not need to be OME-TIFF. However, the metadata file still must have a .companion.ome extension as per the current specification.

And if the tiff files themselves are actually ome.tif files, can some of the metadata actually come from those ome.tiffs? If so, what are the precedence rules in case of conflicting entries?

Not at the moment. If the TIFF files are themselves OME-TIFF, the expectation is still that they would link back to the OME metadata file using the BinaryOnly element.
Distributing the metadata in a hierarchical way as you suggest reads like a very attracting layout. However it would require proper scoping to ensure the specification and the reference libraries resolve such metadata unambiguously.

#7

Hi Sebastian,

I have implemented the following solution:

  • the metadata of one of the selected *.tif will read out with my customize reader (LatticeReader)
  • the LatticeReader generate a companion.ome file with this metadata + metadata of a setting file
    but I don’t understand how I can call the Reader for the companion.ome file. A call of OMETiffReader.initFile(my.companion.ome) inside the LatticeReader.initFile(…) method has not an affect, that means: run of showinf in tools open only a single tif and not the companion.ome file volume.
    Do you have any idea, how I can solve this?

Best,
Susanne

#8

Hi Susanne,

I assume the first step i.e. the reading of the TIFF-based PFF is working as expected. For the second part, would it be possible to share a minimal companion OME sample file generated by your reader?

Sebastien

#9

Hi Sebastian,

when I open the generated companion.ome file directly with showinf, the volume will visualize correctly.
I have to append a xml extension to the file because of the upload filter of the forum…03_190815_MDL3B1_H488_S549.companion.ome.xml (1.4 KB)

Best,
Susanne

#10

Hi Susanne,

I didn’t check in detail but the content of the companion file makes sense. Additionally the fact that the command-line showinf utility displays the volume is a good sign although this really tests the workflow where OMETiffReader is initialized with your generation companion file.

If you want to keep LatticeReader as your primary reader, you might need to have OMETiffReader acting as an internal helper reader. A couple of thoughts:

  • you might need to call helperReader.setId rather than initFile to also complete the metadata initialization
  • once initialized, you might need to reconcile/overwrite the core metadata of your LatticeReader with the initialized metadata from the OMETiffReader
  • looking at existing examples, HitachiReader might be a good starting point for creating a helper reader.