OMERO.mapr Questions

[mdug (Mark Duggan)]

Omero Team,

We’ve encountered a couple of issues with mapr that have me scratching my head and I once again have to ask for your help.

Firstly, we’ve added some links across the top to allow us to browse the data based on image attributes as is the stated aim of mapr. For example, we add attributes to images to reflect the experiment id in our electronic notebook

config append -- omero.web.mapr.config '{"menu": "elnid", "config":{"default":["eln_page"], "all":["eln_page"], "ns"
:["openmicroscopy.org/omero/client/mapAnnotation"], "wildcard": {"enabled": true}, "label":"ELN ID"}}'

config set -- omero.web.ui.top_links '[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}],["History", "history", {"title": "History"}], ["Operator", {"viewname": "maprindex_operator"}, {"title": "Find Operator"}], ["ELN ID", {"viewname": "maprindex_elnid"}, {"title": "Find ELN ID"}], ["Cell Line", {"viewname": "maprindex_cellline"}, {"title": "Find Cell Line"}], ["Program", {"viewname": "maprindex_program"}, {"title": "Find Program"}], ["Model System", {"viewname": "maprindex_modelsystem"}, {"title": "Find Model System"}], ["Help", "https://help.openmicroscopy.org/", {"title":"Open OMERO user guide in a new tab", "target":"new"}]]'

Right now, two experiments show up in the list but the rest do not, however if we perform a search for an experiment the list of experiments show in the search dialogue shows many more, however if we click one of the experiments not in the original visible list it doesn’t display any of the images associated with that experiment. I’m not sure I’m doing a great job of explaining this but hopefully the screen shots attached will help.

Sorry for the blurring, not quite sure of our level of comfort sharing potentially identifying information, but suffice to say what I’ve blurred is a common prefix on all experiments

But I hope it’s clear that it seems that it is being detetected that there are 20 datasets/screens with images with the eln_page attribute set, but only 2 are displayed, and I cannot readily identify what’s different between them

The second issue seems to have cropped up since upgrading OMERO.web to 5.9. If we click into one of the experiments shown and drill down to the images, the attributes section on the right hand side won’t expand and when I look at developer tools in Chrome it shows an Uncaught ReferenceError, with the reference being to the Experiment ID prefix.

Note, I’ve used the experiment ID example as I felt it was easiest to explain, but this behaviour is the same across any of the attributes we’re trying to use, be it Operator, Cell Line, or Program.

I hope this makes sense, but if any clarification is needed please let me know

For reference, we’re running a docker based config with OMERO.server 5.6.3 and OMERO.web 5.9. If we revert to OMERO.web 5.7 the second problem appears to go away, but the first problem exists across either version

Thanks, as always

Mark

[will-moore]

Hi Mark,
The first issue (where you see 20 items in the top of the Tree, and auto-complete, but only 2 are shown when you expand the tree) I managed to reproduce with Datasets that are not contained within Projects.
The tree ONLY shows Project > Dataset > Image, so if any Datasets are “orphans” (not in a Project) then they won’t be shown in the Tree.
Does this explain what you’re seeing?
I realise that this assumption is not well documented. Will try to improve that…

I can’t reproduce the second issue, or be sure what’s creating it. Looking at the changes between OMERO.web 5.7 and 5.9 doesn’t suggest anything obvious.
It looks like it might be because you’re getting different data when loading the map annotations. I wonder if you could pick an Image ID (e.g. 36642 from your screenshot) and go to `/webclient/api/annotations/?type=map&image=36642). That will return JSON data, which is easier to inspect if you have a JSON viewer Chrome add on:

Specifically, I wonder if the ‘link’ data is missing from each item in the ‘annotations’ in the list on OMERO.web 5.9?

Another thing you could try is to uninstall omero-mapr:

$ omero config remove omero.web.apps '"omero_mapr"'

and then we’d know if the error is due to mapr or not.

Sorry I can’t be more helpful,
Regards,
Will.

[mdug (Mark Duggan)]

Thanks Will

The datasets are all part of projects. That’s been part of our usage of Omero is to use the ‘Project’ to organise the data from each instrument, and then have individual experiments represented by the Datasets. We do move some data from the Projects/Dataset hierarchy to the Screen/Plate/Well hierarchy where appropriate, but from looking at IDR I think the Screen/Plate/Well is supported with mapr too.

I also tried moving a Dataset to a project that has working datasets and that appeared to not change the behaviour, so the apparent issue is not limited to Screens etc.

On the second issue, it’s only when viewed through a mapr ‘filter’ that we see the issue with the annotations not being visible on the right. When viewing the same image through the standard “Data” view the annotations appear fine, so I’m not sure removing mapr will tell us much more, but let me know if you still want me to try that, I can do so

Mark

[mdug (Mark Duggan)]

Answering my own question a little bit here.

The Screen vs Project thing seems to be what is tripping me up here a bit. We imported the images under Project/Dataset originally and then converted to Screen/Plate, but didn’t remove the Project/Dataset and I think that’s part of the issue. The experiment ids are now appearing in the tree although the Screen isn’t populating under that. But I think we’re moving in the right direction

[will-moore]

I managed to reproduce the “second issue” and opened a fix for omero-web at treat manager.obj_id as a String in Django template by will-moore · Pull Request #282 · ome/omero-web · GitHub
There were actually 2 errors. The other one is a long-standing issue that didn’t break your workflow - I created an issue for that at right panel map annotations for a Value · Issue #67 · ome/omero-mapr · GitHub

Going back to the first issue. You are definitely right that messing with the hierarchies of Project/Dataset/Image vv Screen/Plate/Well is causing you issues.
When I created a Plate with the Dataset_to_Plate script (and put the Plate in a Screen) and didn’t remove the Images from the Dataset, then they don’t show up in the mapr tree (in my example the images are annotated with Primary Antibody: plate_and_dataset):

Strangely, when I remove some of those images from the Dataset, that allows the Project and Dataset to show up in the Tree (even though ALL the images that remain in the Dataset are still in the Plate) - see the browsing of hierarchy in the right panel:

When I remove ALL the Images from the Dataset, I still see the plate_and_dataset in the Tree, but it is empty - the Screen/Plate is still not shown.

OK - I figured it out. Mapr assumes that Map Annotations are on Wells AND their child Images when in a Plate. I think this is a historical issue due to the fact that the webclient didn’t always show annotations on Wells in the right panel (so annotations were added to the child Images) but when that changed, we added annotations to Wells too.
So, if I have a Plate that contains my 6 annotated Images from above, and I add the same Key-Value pair to 4 Wells, I see this in the tree:

The counts in the Tree (4) are referring to the number of Wells that match my query, but when I get down to the children, it’s showing me the Images that match my query (the 6 Images are not even in the 4 Wells that I annotated!).
If I ONLY annotate the Wells (and not the Images) then the Value will not even show up in the Tree.

Apologies this is kinda messy and not predictable.
You will likely need a script to go through all the Map Annotations on Images in plates and apply the same Map Annotations to their parent Wells.
If you need any help with this, let us know.

Regards,
Will

[mdug (Mark Duggan)]

Will

That’s fantastic. Confirmed that I added the annotations at the well level and the Screen is now showing up in mapr. We’ll be able to script the linking of the annotations to the well, no problem, but thanks for the offer.

As an aside, I really like the different namespaces for the Annotations in IDR, I must figure out how to do that in our implementation

Really, thanks so much for digging into it!

Mark

[mdug (Mark Duggan)]

Will,

One more question now as I attempt to clean this up. I noticed some images had multiple Well Sample parents, due to how the data was imported. I tried to remove the redundant Well Samples by calling

conn.deleteObject(wellSampleId)

This removed the redundant parent from the image, but it has left two null objects in the in the list of children of the Well and I can’t figure out how to remove them.

list(well.listChildren())

[<_WellSampleWrapper id=123994>,
 <omero.gateway._WellSampleWrapper object at 0x7efd247ebdd0>,
 <omero.gateway._WellSampleWrapper object at 0x7efd247eba50>,
 <_WellSampleWrapper id=123997>,
 <_WellSampleWrapper id=123998>,
 <_WellSampleWrapper id=123999>,

This causes Omero.Web to throw an exception as it tries to get the image information from a now null object.

Sorry, making a mess for myself, and by extension, you.

Mark

[mdug (Mark Duggan)]

Disregard

Figured it out by digging into the code a bit and understanding the _wellSamplesSeq list on the WellI object and removing the null entries and pushing the updated object to the updateService. Not sure if that was the ‘right’ way to do it but it seems to be fixed now…

Mark