Blind annotation in OMERO

Hi everyone!

We have a project at work where a researcher wants to annotate images from certain OMERO datasets/projects blindly, that is, a user shouldn’t be able to see filenames, metadata, anything but the actual image. We came up with a few ideas and would like to hear what people think! In order of (decreasing) complexity:

  1. omero-web-blind: fork omero-web to create a separate web client that doesn’t show the sensitive data to users. It would still look exactly like omero-web and talk to the same omero-server, so annotations would be linked to the correct images by default. It involves writing/deploying a separate web client, though.

  2. Standalone Python client: write a small Python application that pulls and displays images from the desired project/dataset. User can write annotations that are linked back to the original image on the server. Problems: large images become an issue, annotations will probably be limited to free text or controlled vocabulary (trickier to have users draw ROIs, etc).

  3. “annotation” dataset: crawl through original images, check which ones haven’t been annotated and import those to a separate annotation dataset where they will be stripped of all identifying information (no metadata on key-value pairs, randomized name - no duplication of data because we in-place import everything anyway). Store the new image name as a key-value pair on the original. In addition to that, crawl through the “annotation” dataset, check which images have already been annotated, link the annotations back to the original and delete them. Not “truly” blind because someone with ill intent can always find the original from the annotation copy, but probably good enough and fairly simple to put together.

Thoughts? Are we reinventing the wheel and something like that already exists? At the moment we’re tending towards 3 because it’s fairly generic and easy to implement, and it’s a fairly small project for the moment. We’d love to hear what you have to say!

Hi Eric,

What’s the starting point? How does the researcher find the Datasets/Projects they are meant to annotate? Do you have to build that UI, or can you provide the researcher with e.g. Dataset IDs?

When you say “wants to annotate images”, do you mean add Tags, Comments, Map Annotations, Ratings, ROIs?

If you were starting with Dataset IDs, you could use the iviewer/?dataset=1 and add ROIs to all the images in that Dataset. You’d only need to hide the Image name, either via your 3) routine above, or by making a slightly modified iviewer?

Will

1 Like

My understanding is that they will need to find the dataset numbers themselves in OMERO (for the original images). “Annotation” here is pretty broad, but mainly comments and/or map annotations, plus ROIs. I suppose using iviewer/?dataset=1 would solve the ROI issue with a bit of modification, but we still have the text annotations to worry about, unfortunately.

One of the simplest solutions I could imagine is a workflow similar to @evenhuis scripts at https://code.research.uts.edu.au/MIF/OMERO-instructions/wikis/organising_data/csv/Managing-Key-values-with-csv-files.
The user could view the images in iviewer and compile their comments and Key/Value pairs into a CSV file, using only the Image ID instead of Name.
Then you could use a Python script to upload the CSV and apply annotations to the Images. This could be run via the CLI, so the user doesn’t get to see the images in the webclient (to avoid seeing other metadata etc).

How will they find the Datasets in OMERO? By name, or by some annotation? Tag etc?
Again, you could write a small Python script that could list the Images or Datasets that need annotating and then create the empty .CSV file with Image IDs and column names only.
This could even contain links to open the images/datasets in iviewer.

With OMERO 5.6 it’s a lot easier for users to install the client python libs, so they can use the omero CLI (to run scripts on the server) or to run a local Python script to read/write annotations to OMERO.
https://docs.openmicroscopy.org/omero/5.6.0/developers/Python.html

conda create -n myenv -c ome python=3.6 zeroc-ice36-python omero-py
conda activate myenv

You could even package your code as a pip-installable omero plugin “omero-blind”, so users could do:

pip install omero-blind
# e.g. list datasets that need annotating
omero blind datasets
# create a blank 'blind.csv' to fill out for a chosen Dataset
omero blind create_csv Dataset:1
# after filling it out, upload to create annotations in OMERO
omero blind upload blind.csv

Does this sound feasible? The advantage is you can create your own client without worrying about developing a UI. Let us know if you want any pointers with this (or any other) approach?

Will

1 Like

Hi Will,

thanks for the suggestions again! The idea of omero-blind and a small tweak on iviewer sound feasible. We’ll need to sit down with the researchers to understand exactly what the use case will look like and which option will probably work better given that. We appreciate the input!

As a side note, this sounds slightly similar to some of the teaching scenarios that have arisen over the years. One example is https://github.com/crs4/moodle.omero-qtypes. @lucalianas has mentioned that he doesn’t regularly come to the forums, so if you’re interested, you might also post a GitHub issue there. ~Josh

1 Like

Lets say we choose something in the direction of step 2.
I use a python script to generate annotations and link it to the specific image ID.
Currently, I am guessing that it is possible to link the annotations only considering the image ID.
Is it possible to link annotations considering both image and dataset ID?

Consider the following example:
I have two datasets: originalDataset (id: 1), testDataset (id: 2).
Both datasets contain the same images (or images with the same ID, lets say 11, 12, 13, 14, 15)
If I link annotations to originalDataset, I can see those annotations also in testDataset. Is it possible to make sure that the annotations are specific to both imageID and datasetID?

Best Regards
Vignesh

@vigi04

I am guessing that it is possible to link the annotations only considering the image ID.

Do you mean that it is possible to link annotations to specific images, which are identified by their IDs ? Yes, this is possible.

Is it possible to link annotations considering both image and dataset ID?

Maybe you are asking whether it is possible to link annotations both to images and datasets ? Yes, this is possible, you can link annoations to only datasets, or only to images in a specific dataset, or to both dataset and images. As long as you either know or query the IDs of all the objects you want to annotate (i.e. IDs of the datasets and images you want to annotate), you can annotate these using your python script.

If I link annotations to originalDataset, I can see those annotations also in testDataset

No, not true as formulated. Maybe you mean “If I link annotations to images 11, 12, 13… then I will see those annotations on those images whether they are displayed as linked to originalDataset or to testDataset” - this is true, because your images 11, 12… are the same (being the same and having the same ID is the same thing :slight_smile: ), the OMERO clients such as OMERO.web will then display such images twice, once per each dataset which “contains” then (better said “to which they are linked to”, but this is just a way of displaying. See https://omero-guides.readthedocs.io/en/latest/introduction/docs/data-management.html#manage-images-in-datasets-projects for that.

Is it possible to make sure that the annotations are specific to both imageID and datasetID?

Are we still inside your example with originalDataset and testDataset ? I suppose yes, and in that case, you probably mean “Is it possible that the images (11, 12, 13…) which are displayed in originalDataset are displaying annotations on them, whereas the same images (11, 12, 13…) which are displayed in testDataset do not show any annotations ?” No, unfortunately, this is not possible. I think it is clear why if you read how I re-formulated your question - the datasets in OMERO are actually not “containing” the images, instead, the images are doubly linked to the two datasets in youir example and are the same entities in OMERO.
Hope this anwers your questions.

Thank you

All the best

Petr

@vigi04

Maybe just one more suggestion towards your example with originalDataset and testDataset. The setup you wish for would be possible if you would have 2 different set of images linked to the two Datasets. Thsi can be achieved by duplicating the images. Then you would have indeed two different sets of Images, one of which you would annotate (say, in originalDataset) and one set of images which you would not annotate (say, in testDataset). These two sets of images would have different IDs of course. You can read about the omero-duplicate in https://omero-guides.readthedocs.io/en/latest/introduction/docs/data-management.html#duplicatecli

Best

Petr

Hi Petr
Thanks for your detailed answer.
So, as far as I ve understood, i guess the only way to achieve the use-case is by copying the dataset and treating them as different images. Hmm… understood.

Thanks
Vignesh