Are Attributes in an Image's Table Searchable?

I’ve been playing around with the 5.5.1 populate-metadata script lately which uses a csv to populate a table for each image referenced in the csv. However, I’ve noticed that I cannot search on those fields. When I add the same fields as key-value pairs, however, they become searchable. Are the values in the table supposed to be searchable? If not, is there a way to put the values in the table into the key-value pairs?

  • Philip

Hi Philip,

Yes it is possible to convert tables into key/value pairs. Below are a few hindsights from the way IDR annotates its reference image datasets.

Metadata files submitted by the authors and curated by the IDR team are first turned into OMERO.tables using the omero-metadata plugin and the omero metadata populate CLI command. These tables are most of the times attached at the screen and project level, occasionally at the datasets or plate level.

These tables are then converted into key/value pairs using different namespaces for top-level categories (Organism, Gene, Cell Line…). This process is achieved using another functionality of the omero-metadata plugin and the omero metadata populate --context bulkmap CLI command. This command can be run against any object annotated with a table using a configuration file which contains the mapping between the table columns and the keys to be created. Published IDR configuration files can be found on the curated metadata repository - see idr0044-experimentA-bulkmap-config.yml for an example.

Best,
Sebastien

Hello Sebastien,

Good to know! Is there a reason that the context option is not listed on the readme in https://github.com/ome/omero-metadata? I’m using the updated metadata plugin which means creating the tables can be done for a dataset from the web via the script. Does the bulkmap operation still require a yaml file? Also, can it be done from the web?

Thanks for the help,
Philip

Hi Philip,

Is there a reason that the context option is not listed on the readme in https://github.com/ome/omero-metadata?

I don’t think there is any obvious reason other than time and capacity although this seems like it would be a timely addition given the growing interest across various posts in this forum.

Does the bulkmap operation still require a yaml file?

Yes the conversion between tables and key/value pairs require a configuration which can be serialized expressed either as JSON or YAML. The CLI command bin/omero metadata populate --context bulkmap can read such a configuration either stored on a local file on disk or as a file attachment on the server.

Also, can it be done from the web?

As of OMERO 5.5.1, the core OME scripts can only create tables but do not have any option to convert tables into key/value pairs. It might be possible others have implemented such a functionality.

1 Like

Thanks for the explanations.
The current populate-metadata script is usable but it’s really disappointing that the resulting table is not searchable. That’s basically the reason why we wanted metadata in the first place. If we need to have an admin create a new yaml file and execute a specific CLI command every time we create a new set of images, then the web script is basically useless to us because there’s no reason not to make the tables in the CLI as well. It’s also a very tedious process that will take a lot of time for everyone involved.

I have two suggestions/requests:

  1. Make the yaml file optional, with the default behavior mapping all attributes as if they were string values (I suspect this will be the case 100% of the time for our use cases).
  2. Add a script to do the mapping on the web, possibly as an option in the current populate metadata script. Since the yaml file is optional, this will be made a bit easier. If you really, really need the yaml file, then it could be included as an attachment.
1 Like

Hi Philip,

Thank you for your suggestions.

  1. Make the yaml file optional, with the default behavior mapping all attributes as if they were string values (I suspect this will be the case 100% of the time for our use cases).

Thanks, I have captured the idea of a default configuration allowing to convert tables into Key-Value pairs (MapAnnotations) in the omero-metadata library - see https://github.com/ome/omero-metadata/issues/33 . Note that “bulk annotations” mentioned in that context are OMERO.tables.

Please note that the Key-Value pairs (MapAnnotations), which are searchable, at the moment will only support strings (numerical values can be entered, but they will be stored as strings). This might be the reason why using OMERO.tables might be sensible - see OMERO.parade https://pypi.org/project/omero-parade/ - a data mining tool which we are developing. OMERO.parade can filter images and create graphs according to (preferably numerical) values stored in OMERO.tables.

  1. Add a script to do the mapping on the web, possibly as an option in the current populate metadata script. Since the yaml file is optional, this will be made a bit easier. If you really, really need the yaml file, then it could be included as an attachment.

We have captured your suggestion - see https://github.com/ome/scripts/issues/151 and will have a look at implementations. In the meantime, please note that the community already came up with scripts (server-side scripts, just like the Populate metadata script you are exploring.) These are to be found in https://github.com/evenhuis/omero-user-scripts, more precisely https://code.research.uts.edu.au/MIF/OMERO-instructions/wikis/organising_data/csv/Managing-Key-values-with-csv-files#4-add-key-values-pair-from-the-csv , and https://github.com/saeri-ims/omero-docker-compose/tree/master/custom_scripts/for_tags_adminONLY .

You might find it worthwhile to study, adjust and overtake those scripts for your use case.

All the best

Seb and Petr

1 Like