Bio-Formats returns all metadata *except* OME meta data?

Hi!

I am exploring the Java reader, but it seems that I am not able to load the OME metadata.

DynamicMetadataOptions vOptions = new DynamicMetadataOptions();
vOptions.setMetadataLevel(MetadataLevel.ALL);
vOptions.setValidate(false);

this.mImageFileName = aImageFileName;
this.mImageReader = new ImageReader();
this.mImageReader.setMetadataFiltered(false);
// Specifies whether or not to force grouping in multi-file formats.
// We have this disabled by default
this.mImageReader.setGroupFiles(false);

ServiceFactory vFactory = new ServiceFactory();
OMEXMLService vService = vFactory.getInstance(OMEXMLService.class);
IMetadata vOMEMeta = vService.createOMEXMLMetadata();

this.mImageReader.setMetadataStore(vOMEMeta);
this.mImageReader.setMetadataOptions(vOptions);
// specify whether or not to save proprietary metadata in the MetadataStore
this.mImageReader.setOriginalMetadataPopulated(true);

this.mImageReader.setId(mImageFileName);
[...]

HashMap<String, String> vTransformedMetadata = new HashMap<String, String>();
// transforming object to string representation
for (Map.Entry<String, Object> item : this.mImageReader.getGlobalMetadata().entrySet()) {
    vTransformedMetadata.put(item.getKey(), item.getValue().toString());
}

I expect that vTransformedMetadata to contain all metadata, including OME metadata.

I have also tried to follow this example bioformats/ImageInfo.java at 304290bcf9d8edf906793d9e7074d8db47b71b3a · ome/bioformats · GitHub :

MetadataStore ms = this.mImageReader.getMetadataStore();
IFormatReader baseReader;

// TODO: why is this needed?
if (this.mImageReader instanceof ImageReader) {
    baseReader = ((ImageReader) this.mImageReader).getReader();
}

OMEXMLService service;
try {
    ServiceFactory factory = new ServiceFactory();
    service = factory.getInstance(OMEXMLService.class);
}
catch (DependencyException de) {
    throw new ImageReaderException(OMEXMLServiceImpl.NO_OME_XML_MSG + "\n" + de.toString() + "\n" + ExceptionHelper.getStackTrace(de));
}
String version = service.getOMEXMLVersion(ms);
String xml = service.getOMEXML((MetadataRetrieve) ms);

I expected to find the original OME XML inside the xml variable, but it seems like I am only generating an OME XML from the metadata that BioFormats loads.

Am I doing something wrong? Is there any way to load also the original OME meta data?

Thanks!

Dear @IAL32 ,
so this is from a TIFF file with embedded OME meta data? And you can not get that OME meta data back?
All the best, Mario

Hi @emmenlau , that is exactly the problem that I am having

The file in question is https://s3-us-west-2.amazonaws.com/viqi-test-images/cells.ome.tif

Looking at the TRACE output of showinf, it seems as if there should be many entries for the original meta data, but those seem to get lost in the conversion Running UPDATE_200809 stylesheet. Is that possible?

How are you creating the OME-XML in the file? The additional metadata should be stored as part of the structured annotations (Structured Annotations — OME Data Model and File Formats 6.2.2 documentation).

If you are using the Bio-Formats API to save the file then you can use the below for adding additional metadata from a HashTable

ServiceFactory factory = new ServiceFactory();
OMEXMLService service = factory.getInstance(OMEXMLService.class);
OMEXMLMetadata metadata = service.createOMEXMLMetadata();
IFormatWriter writer = new ImageWriter();
Hashtable<String, Object> originalMetadata = null;
//set metadata
service.populateOriginalMetadata(metadata, originalMetadata);
writer.setMetadataRetrieve(metadata);

Alternatively if its easier you can include all the additional fields as a single XMLAnnotation using the MetadataStore interface:

annotationID = "Annotation:" + annotationCount;
      store.setXMLAnnotationID(annotationID, annotationXMLCount);
      store.setXMLAnnotationNamespace("annotationNamespace", annotationXMLCount);
      store.setXMLAnnotationValue(myXML, annotationXMLCount);
      store.setImageAnnotationRef(annotationID, imageIndex, annotationRefCount);

This can then be retrieved again using the MetadataRetrieve interface:

  int xmlAnnotationID = retrieve.getXMLAnnotationCount();
      for (int i = 0; i < xmlAnnotationID; i ++) {
         String annotationID = retrieve.getXMLAnnotationID(i);
         String annotationNamespace = retrieve.getXMLAnnotationNamespace(i);
         String annotationValue = retrieve.getXMLAnnotationValue(i);
      }
2 Likes

Thanks a lot @dgault , this is quite helpful! I’m in touch with @IAL32 to look into this issue. We will try your code and use that in the future to store and retrieve the meta data. This actually answers our most important question!

But I also have a related question: This file (linked above) is from a test set that we received a longer time ago from an open source project. We do not know who created the file. But it seems that the file has a larger set of meta data stored in OME-XML (we can see that in the TRACE output of showinf but the meta data can not be retrieved via Bio-Formats (showinf does not show that data in the final results).

Can you help us understand if the file is corrupt, or if there is an issue with Bio-Formats? We are a bit concerned now that Bio-Formats does not give us any way to understand whats going on, no error or other means to debug this problem. Or am I overlooking something?

The issue seems to be that this particular file has been created with an extremely early version of the schema (2003-FC - Open Microscopy Environment OME Schemas). Normally when an older schema version is read it passes through a series of transforms (Index of /Schemas/Transforms) to upgrade the metadata to the newest model. With this original file however it seems that the custom tags have not been able to be handled as part of the transform step.

Is this file typical of the data you will be dealing with or was it simply a test file from a very old dataset?

1 Like

Is this file typical of the data you will be dealing with or was it simply a test file from a very old dataset?

Actually, I don’t know. This file was part of a test dataset that we received. I do not know how many real datasets are from that time in the users repository.

But the problem matters to me because it makes us question how we can ensure data integrity in the light of this finding. I would have (naiively) assumed that Bio-Formats would never silently “loose” meta data. However here we encountered two problems:

  • The meta data of this file could not be fully recovered, and
  • There is no indication of the loss of data in Bio-Formats

Each of these problems is severe by itself. But in my personal opinion, the second issue rates higher than the first one. I would actually expect that under all circumstances, we must discover loss of data.

It would have been assumption that the data never gets lost either, that is the reason why we implement the transforms from the original schema through to the latest as we always support backwards compatible files. For this particular file the transform steps should have upgraded the custom attributes to newer xml annotations, currently i’m trying to understand exactly why that has not occurred in this instance. There are a number of validation errors with the XML in that file but I am not sure if they are directly related. I will update this thread once I found the root cause of this issue.

2 Likes

Thanks a lot @dgault ! The issue has raised some concerns and it would be great to better understand how robust Bio-Formats is with respect to such issues.