Prepping and including ROI masks

First time post. Thanks. : - )

I’m in the process of translating and exporting our meta info and imagery into OME-TIFF format. Much of that work is done. I’m using the Omero server and viewer as my test environment to validate my exported files.

My ROIs are coming out as valid, well formed ROIs with labels and shapes and etc. My (…limited) understanding from the OME schema is that we will also be able to include our masks for each ROI.

Our mask images are stored as 8 bit PNG imagery with white foreground and black background data. So I’ve been programmatically converting the mask images to TIFF and further converting that to a Base64 string to include as BinData in the Mask element. (See below.)

My OME-XML file passes schema validity (…using xmlvalid) and the mask element is present in Omero’s ROI info dialog as part of the ROI. But the mask imagery itself is not visible on screen in any way. And indeed “showing” and “hiding” the mask element alters nothing on screen.

Am I misunderstanding how to prepare the image data? Or is my understanding of the schema off? Are there any sample OME-TIFFs anywhere that include realistic mask data for comparison? (I’ve seen the ROI.ome.xml sample on openmicroscopy.org.)

  <Image ID="Image:0">
    [ etc...]
    <ROIRef ID="ROI:0" />
  </Image>

  <ROI ID="ROI:0">
    <Union>
      <Rectangle ID="Shape:0" X="4047" Y="7390" Width="324" Height="260" />
      <Mask ID="Shape:1" X="4047" Y="7390" Width="324" Height="260">
        <BinData BigEndian="false" Length="[ data length ]">[ data ]</BinData>
      </Mask>
    </Union>
  </ROI>
  [ etc...]

Thanks.

Hi Barry

Congratulations on your first post.

If I get you right, you are trying to view masks in OMERO.iviewer on OME-TIFF images which you have created. You have validated locally their OME-XML headers but you are not seeing the masks after you imported such an OME-TIFF image into OMERO.

There could be several possible reasons:

  1. The masks do not have any binary data in your OME-TIFF.
  2. The masks got imported, but you are not able to view it in OMERO.iviewer because
    a. the masks could be outside the image
    b. the masks’ borders are too thin

I do not know which of the above reasons is the true one.

The Ad. 1 above is most probable, as I am able to repeat a similar workflow to yours.

  • First, I have a mask in OMERO on an image
  • Then I export it as OME-TIFF
  • Then re import the exported OME-TIFF into OMERO.
  • On the reimported image in OMERO, there is a mask listed in the ROI table of OMERO.iviewer, but the mask is not visible on the image. This is probably because during the export, the binary data of the mask do not seem to have been written into the OME-XML header of the OME-TIFF. The mask is declared in the OME-XML, but the binary data is missing. I imagine you are using a similar workflow when writing the OME-TIFF out from your software and thus getting similar problems .

The Ad. 2a above is possible, please check the position of your mask with respect to the image sizes. I have managed to get a completely invisible mask in OMERO.iviewer when preparing some example OME-XML containing masks for you (see below more about that). The parameters to adjust are the X and Y in the header of the masks xml bit.

The Ad. 2b above is possible as well, and can be verified or refuted easily, by clicking onto the line with the ROI mask shape in the right-hand side table. In such a case, the mask might still not be visible, but the encompassing rectangular border will show in the image as blue. See the screenshot attached to this post.

You further ask for an example of OME-TIFF with a mask. I do not have the OME-TIFF for you. Nevertheless, I am attaching a valid and tested OME-XML file to this post with a mask element in it. This file can be imported into OMERO (see below how) onto any image of your choice which is larger than the size of the mask (if the image is smaller than the mask XY offset position, then the problem 2a above will occur. You can simply adjust the X and Y parameters of the mask inside that OME-XML, setting them to zero, and import the OME-XML again into OMERO. )

The way to import the the OME-XML into OMERO is highlighted in the QuPath walkthrough of the omero-guides, see step 36 of Analyze OMERO data using QuPath — OMERO guide 0.2.0 documentation please. In the same walkthrough, you will see in Resources what version of ome-omero-roitool plugin to use for your import and how to get this plugin from GitHub. This OME-XML was created in QuPath using the walkthrough I am pointing you to. Feel free to go through it, but if you want to spare time, you might just use the OME-XML and step 36 to explore the behaviour of the mask in your OMERO.iviewer and get inspiration from the OME-XML layout.

Further, you might be interested in how the OME-XML was written. The code for that is inside the following groovy script for QuPath. There is a short description of this export script in README which will give you an idea about the ROI objects writing and matching in the QuPath case. It can also give you a hint regarding the problem 2b above (line width), although, in your case, you cannot overtake this width hint directly for your workflow.

Finally, if you think it is helpful, then you can go through the QuPath walkthrough yourself in full and create any OME-XML files with masks from images which you choose to open in QuPath. You just have to draw a “magic wand tool” ROI on the image in QuPath and use the export script. But please do not draw too big ROIs, as the export script might not scale well with the size of the exported masks, and you may hit performance issues. For the OME-XML I am pointing you to, I have used the image which is publicly available on IDR - you can open it in QuPath without downloading it, just following the instructions in the QuPath walkthrough as indicated above and putting in this url when asked for the url to the image. The screenshot I am attaching to this post is showing a mask which I created as OME-XML and re-imported into another OMERO server on that very image (I could not import it into IDR, as this is a read-only server, so if you want to go exactly down this route, you would have to download the image locally. Let me know if you think you need/want to have this precise image, I will give you instructions about how to get it.

Lastly, let me give you a link to an example of a mask in OMERO from IDR This mask is very different from the one in the OME-XML I am giving you above and was created in a different manner, but you will definitely be able to view it, just to get an idea about how masks are displayed in OMERO.iviewer.

Best wishes
Petr
OME Team

[1]
OME-XML
for-forum-post-exported-from-QuPath.ome.xml (22.1 KB)
[2]
Screenshot

Lots to chew through, but it sounds like I am generally on the right track and I’m sure my answer is in there. Thank you. I’ll report back.

Looks like I will need to do some further processing on my mask images. They are PNG files to start, as I mentioned, but they are 1 band bit masks. So what are the requirements for a proper OME-TIFF mask image?

  • I am assuming they must be converted from PNG to TIFF data?

  • I believe I read elsewhere that 1 band masks are not supported? Must they be 3 band RGB? Or 4 band SRGB or RGBA? Other?

  • Is there a bit depth requirement or other band format requirements?

  • From the spec, I am assuming I must convert to a Base64 string if I wish to include the mask image as BinData in a Mask element?

Thanks.

Hi @blasky

  • I am assuming they must be converted from PNG to TIFF data?

If your goal is to embed the mask within the OME-TIFF directly, there is no need to convert the mask data into an intermediate TIFF. Instead the mask binary data must be stored in the BinData element of the OME-XML.

  • I believe I read elsewhere that 1 band masks are not supported? Must they be 3 band RGB? Or 4 band SRGB or RGBA? Other?
  • Is there a bit depth requirement or other band format requirements?

The BinData associated to a Mask is a single bit stream.

  • From the spec, I am assuming I must convert to a Base64 string if I wish to include the mask image as BinData in a Mask element?

Yes the mask binary data must be base64 encoded, optionally compressed, and stored in the BinData element. See below for an example of Java code performing this operation:

If your end goal is to manage the data in OMERO, an alternative to encoding the mask within the OME-TIFF would be to create mask ROIs and attach them to the images as a post-import process. This is an approach we use regularly for IDR submissions where the segmentation data is typically provided as separated TIFF/PNG files. If that is of interest to you, you might be interested in the utilities from GitHub - ome/omero-rois: OMERO python ROI utils which allow to read from PNG/TIFF and create masks. The main drawback compared to your approach is that the link between the image and the mask will only be expressed in the OMERO database rather than the file itself.

Thanks very much for the additional info and links.

…create mask ROIs and attach them to the images as a post-import process.

For the moment I believe we will continue to try to get the mask info into the OME-Xml as a single file package. Read only is ok. And we are looking to see if we can get all or most of our info into a single OME-Tiff file for easy interoperability. Export from our system as an OME-Tiff >> Import into Omero.

Yes the mask binary data must be base64 encoded…

Not a problem, of course.

The BinData associated to a Mask is a single bit stream.

So our 8 bit Png mask has the info that is needed. It sounds like I just need to focus on getting that into the right format / data structure for inclusion. Pardon the explicit walk through / questions, I just want to make sure I understand the requirements for the data as I will need to do the translation in C# in our Export code.

In the QuPath script above the mask data is…

  1. A byte array
bits.toByteArray())
  1. Consisting solely of 1 bit on/off data for each mask pixel
bits.write(b == 0 ? 0 : 1, 1))
  1. With the pixels arranged in what order? Starting at 0,0 upper left? And proceeding through each row? 0,0 0,1 0,2…1,0 1,1 1,2…?

  2. And presumably the Width and Height set in the Mask element XML will be used to parse the byte array back into the proper rows and columns?

Thanks.

And just to be extra, extra clear, I am currently importing the OME-TIFF into my Omero server using the Omero.insight client v5.5.14. This client’s import functionality will import and process the Mask BinData from the OME-XML when properly formatted and encoded?

I went back to the original sample XML I had been looking at when I began the work of translating our ROI info into OME-Xml metadata.

https://downloads.openmicroscopy.org/images/OME-XML/2016-06/ROI.ome.xml

This sample has one Mask element. (It’s part of where I looked when structuring my XML.) I don’t know exactly what this mask is supposed to look like as all I can see in the XML is the Base64 BinData. But it’s there.

I imported this sample XML into my Omero server using Omero.insight. And then I am viewing it in Omero.iviewer. Below you’ll see that this mask in the sample XML doesn’t seem to appear either. It’s similar to the experience I am having with my masks.

You can see the mask element in the ROI list. And you can see its dimensions and info and etc. But no actual mask ever appears on screen. The first screen shot is the image with the mask turned off and “hidden”. The second is the image with the mask turned on and “shown”. No difference? (…aside from the fact that the missing mask element is “highlighted”)

image

image

image

Apologies for the delayed response. We are trying a few things to see if we can find a workflow that might work for you. We’ll let you know ASAP…
Will

Hi,

So we investigated and found that the <BinData> in the ROI.ome.xml example file was incorrect. I’ve opened a PR to fix it at Fix BinData in Mask in ROI.ome.xml by will-moore · Pull Request #140 · ome/ome-model · GitHub
and there’s some sample python code there too, which was used to create the BinData.
Hope that helps.
NB: the display of this 2 x 2 Mask in iviewer doesn’t appear correctly (offset by 2 pixels), which I assume is due to it being a very small mask. As in the iviewer screenshot above, I think it will work for a “real” mask.

Hope that helps,
Will.

Thanks. Appreciate it. I will make these changes locally in my version of ROI.ome.xml and re-import it to start, so i can see the import and masking working properly. Then I will return to my mask BinData to see if it is in line as well.

The ROI Union that contains that mask BinData in Roi.ome.xml also has a Transform. I believe that is where the offset is coming from.

And looking at the Python code you linked, it looks like the missing piece for my mask data is that I need to write/run a C# equivalent to NumPy’s np.packbits()…

Thanks, I’d missed the Transform there.
Let us know if there are any other issues etc.
Regards,
Will

Looks good. Thanks again for the responses and pointers.

Although it would be nice if the data structure requirements for the Mask BinData was clearly documented somewhere. (Or highlighted better if it is fully documented and I missed it.)

  1. 1 bit image pixel masking data
  2. Packed into a uint8 byte array
  3. Base64 string encoded.
  4. etc…

Here’s an import from my new OME-Tiff export → Omero import code.

1 Like

Looks good.
You might want to follow the discussion at Fix BinData in Mask in ROI.ome.xml by will-moore · Pull Request #140 · ome/ome-model · GitHub about the handling of incomplete bytes, since that PR may not be merged.

Will