Image upload with Metadata

First of all let me set the scene as to what we’re trying to do.

We have a project where we’re collecting data from a number of students at different institutions. Students are collecting this data in the form of images and associated metadata (e.g. experiment type, experiment duration, results)

They basically end up with images with the file names in the format:
Institution, experiment number, replicate number, image number - e.g.:
uod-exp001-rep001-image001.tiff
uod-exp001-rep002-image002.tiff
uod-exp001-rep003-image003.tiff
uod-exp002-rep001-image004.tiff
uod-exp002-rep002-image005.tiff
uod-exp002-rep003-image006.tif
etc etc

and a CSV of meta data for each experiment:
exp001.csv
imageName,expType,weight,time,count
uod-exp001-rep001-image001.tiff, transfer, 0.5kg, 30s,20
uod-exp001-rep002-image002.tiff, transfer, 0.5kg, 30s,22
uod-exp001-rep003-image003.tiff, transfer, 0.5kg, 30s,21
exp002.csv
imageName, expType, weight, time, count
uod-exp002-rep001-image004.tiff, transfer, 1.0kg, 30s,30
uod-exp002-rep002-image005.tiff, transfer, 1.0kg, 30s,29
uod-exp002-rep003-image006.tiff, transfer, 1.0kg, 30s,33

We’d like to collect this data and make it available/searchable in OMERO.

In an ideal world we’d like to have them submit all this information via a web portal straight into OMERO but we know this isn’t really possible just yet. So what is the best way to do this?

It’s likely for the data collection side we’re simply going to create our own web front end that allows images to be uploaded with the metadata. This will store all the data in my MySQL database… but from where what’s our best steps? None of the OMERO docs make it particularly clear how we import image with associated metadata using OMERO.insight and in an ideal world we’d like to automate this import process as well.

Suggestions welcome as to how we could best achieve this! :slight_smile:

Thanks in advance,
Roy

Hi Roy,

we developed a couple of scripts for OMERO that might help. There repository is on GitHub and there are some instructions here.

The scripts can be run by the users from either OMERO.insight or OMERO.web. The ones that are most useful for you would be:

  • Key_value_from_csv.py: attaches key-value metadata from an uploaded csv to images
  • Key_Val_to_csv.py: export filenames and metadata to a csv file
  • Key_Val_from_Description.py: splits filenames based on regex-like syntax to create key value metadata

We usually get users to

  1. upload images with a minimal metadata encoded in the filenames.
  2. split filenames to add key-value metadata to the images.
  3. export an initial csv from OMERO which is then cleaned and merged with more detailed experimental metadata.
  4. upload an updated csv to OMERO and attached key-values to each image.

I don’t think that it is possible to upload the csv as part of the image upload process. Maybe this could be a feature request at some point.

Hope it helps,

Chris

2 Likes

Thanks Chris - it’s typical that I found your very solution just before heading home yesterday!

I’d been searching around for “EXIF” and “Metadata” solutions but the KVP does seem like the way to go to make sure that we can query the data.

However, we’ve got two problems with this at the moment. 1) We don’t have the ability to install scripts on our hosted OMERO instance at this moment in time (something we’ll need to resolve if we want to use the CLI tool anyway) 2) Without being able to script it as part of the upload process I’m not sure of it’s long term viability for us. Because it’s going to be a distributed data gathering exercise we need to make sure it’s a sustainable solution to cope with both adhoc additions and surges.

Please don’t tell me I have to spend time writing my own plugin :sweat_smile:

Only as a text annotation at the moment.

:+1:

I understand that likely an admin will need to install the scripts for you. But you should be able to make use of the CLI remotely to the server with no special privileges.

Either via a bash script running the CLI or a dedicated python script (or Java for that matter), you can definitely script the upload plus the attachment of a file and even the launching of @evenhuis’ scripts. Happy to help.

~Josh

Hi Roy,

As @joshmoore said, installing scripts isn’t a big deal. If you’re happy working from the CLI this post from @johnmc outlines a way to do a similar thing.

I’m not sure what your needs are exactly, is it something like a 1-step process where:

  • The user imports a directory containing a image and a metadata csv
  • OMERO imports the images as attaches metadata for each file

What I suggested before is user-friendly 3-step process (upload images, attach csv, run script). If you’re after a user friendly 1-step process the options get tougher.

The java route

The OMERO.insight already does things like this. On import it detects log files (plan text) that are associated with images and uploads the image+log file as a Fileset. The metadata is extracted is extracted from the log file and added to image data in OMERO.

So it should be possible to customise the OMERO.insight client to look for csv file (possibly specially named) and attach the metadata. This would involve getting into the guts of the OMERO.insight java code, then compiling and distributing your own client. A pretty big undertaking.

The python route

The other way would be to use the Python API to upload the images and attach the metadata from the csv. If the operations are kept simple (upload single directories of restricted file types) the complexity won’t be too bad. Distributing to users will be trickier though as as the environment needs to be right (python 2.7 + specific ice libraries). Conda is good for this but your users will have pretty command line savvy to do this.

Eventually, you’d implementing much functionality of the OMERO.insight client. If you’ve got experience in python or are reluctant to use java this may still be viable option.

No worries, you need just write your own upload client :smile:.

Cheers,

Chris

1 Like