Setting up OMERO in a new lab - suggestions, stories and advice welcome

My lab does a lot of correlative imaging, so we have a lot of images from the same samples with fluorescence & electron microscopy from multiple instruments. There’s a lot of variety & we don’t have a good way to keep track of which images are part of the same sample sub-region. We think we need some kind of database to help us manage this.

  1. Is OMERO a good choice here?
  2. What’s the experience of other groups, and do you have advice for setting a system up from scratch?
  3. Any suggestions for how best to structure the database?
  4. How can we automate or enforce consistency in this structure as much as possible?

I appreciate your thoughts

3 Likes

There were a couple of nice blog posts by @erickratamero about this topic.
Maybe he can give you more pointers :slight_smile:

https://blogs.warwick.ac.uk/camdu/entry/implementing_omero_from/
https://blogs.warwick.ac.uk/camdu/entry/omero_deployment_the/

3 Likes

That’s really handy to know, thanks for sharing

Hi Genevieve,

we’ve had quite a lot of success with OMERO at Warwick Medical School. Setting it up took some work, but anyone who is comfortable around a Linux terminal should be able to do it. You could structure it as one dataset per sub-region; definitely the trickiest part is enforcing consistent standards for that structure. The one thing that OMERO doesn’t do is folder structures, which on one hand limits the complexity of the possible data organisation schemes, but on the other hands keeps things fairly simple and easy to get. I’d probably have a browse at IDR before anything else. If you like what you see there, then OMERO is probably a good choice for you. I’d be more than happy to give some pointers in addition to those blog posts!

4 Likes

OMERO does allow images to be placed into datasets (multiple at once) which can be placed into projects but that’s only two levels of hierarchy. It also allows other structured metadata to be added, for instance at https://help.openmicroscopy.org/managing-data.html#annotating, which in the server’s database is stored more like https://docs.openmicroscopy.org/ome-model/5.6.4/developers/structured-annotations.html. You’ll see this kind of thing on the right as you explore images in the IDR.

If you are willing to push beyond what the graphical clients easily support, e.g., by writing some scripts, then you can make extensive use of the OME model to manage your data. For example, with ROI folders at https://blog.openmicroscopy.org/data-model/future-plans/2016/05/23/folders-upcoming/ the regions of interest on different images that are actually different views of the same region of the sample could share the same folder. Or, the aforementioned structured annotations can be used to represent metadata as both images and ROIs may be annotated. However, as an example of where the graphical clients currently fall short: I don’t think they would show annotations on ROIs.

Someday we’d love to get to substantially improve OMERO’s support for correlative imaging. If you could tell us more about your use case and what you’re trying to do in your workflows then we might be able to help you along and that could guide us in proposing future work.

Cheers,
Mark (OME)

2 Likes

how is the management of different permissions to access/sahre the images and data?

OMERO follows a UNIX-like permissions approach of allowing owners, members and others various levels of read and write access, with “annotate” being a level meaning that they can’t write data but they can annotate it. Every image is owned by some user and is in a group; users may be in many groups and images may be moved among groups. The permissions can be set from wholly private to the image owner all the way up to read-write by anybody; intermediate levels allow labs to collaborate usefully. An introduction is at https://help.openmicroscopy.org/sharing-data.html and there is more information at https://docs.openmicroscopy.org/latest/omero5.4/sysadmins/server-permissions.html.

Here in Dundee we have an OMERO production server that is used by many research groups with various levels of collaborative and private access but it also hosts published data in other (OMERO-level) public groups where the DOI links in their papers bring readers back to data on that same server.

Cheers,
Mark

3 Likes

Thanks Erick! I’ll have a chat with out IT team, and might end up taking you up on that offer down the track.

I read your follow up post at https://blogs.warwick.ac.uk/camdu/entry/omero_deployment_the/ - now it’s a year later is there anything else you’d add to those comments?

Thanks for those detailed comments, Mark

Everyone I’ve talked to has mentioned the limited hierarchy available, seems like that’s a pretty common annoyance. We will definitely need to see what we can do to compensate with user added metadata & tags. I expect most of our analysis will be scripted.

@mtbc

Someday we’d love to get to substantially improve OMERO’s support for correlative imaging. If you could tell us more about your use case and what you’re trying to do in your workflows then we might be able to help you along and that could guide us in proposing future work.

Happy to discuss this more with you, maybe email is a better choice for an extended discussion? Let me know what you prefer.

Hi @GenevieveBuckley,

I’d say keeping everything on this thread at the moment could make for good reading for someone else trying to do the same thing. I’ve added the #correlative tag so hopefully others can find it. (“correlative-microscopy” is too long. Other suggestions?)

At the moment, OMERO doesn’t provide anything out of the box, so someone else will probably have to say if there’s a better choice out there. That being said, we’d certainly look forward to both capturing what it is that you’d like to see OMERO look like with your data, as well as find workarounds that would bridge the gap for you and others until there is a complete solution.

~Josh

2 Likes

I think since then the biggest thing for us was finding a way to get data automatically from the microscope computers - it lowered the entry barrier and solved our biggest problem at the time (adoption). Now we have a core of heavy users who love it and a large amount of people who use it because they don’t need to worry about data transfer, external storage or anything else.

1 Like

that is very interesting!

What were the challenges there?
How would a workflow then be for a user, from acquisition to analysis?

Cheers,
Christopher

There were a few different challenges - we’re running multiple operating systems on our microscopes, with computer that range from brand-new to, well, relatively old - there’s everything from Windows XP and CentOS 5 to up-to-date Windows 10. We’ve looked at adapting existing tools like AutoTx or the existing CLI importer, but we had nothing that would work in every single system. We settled for having an extra Ubuntu 18.04 machine that mounts data directories on the microscope computers as SMB shares inside a local network and runs the CLI importer for individual users. It’s extremely improvised and it breaks more often than I’d like, but 90% of the time it works every time.

For the user, they just need to save their data onto their folder inside a specified “Data” directory in the microscope computer - overnight, the data is imported to their user account in OMERO and (upon a successful import) moved to an “imported” directory on the microscope computer (in case they want a local copy). From there, they just need to access the OMERO server (using their university credentials) in any way they prefer.

4 Likes

Thank you!

Performing an automated import is for sure the best way to ensure proper usage.

I just wonder how useful it is to import everything directly from the acquisition.
I mean one needs to have some metadata from the experiment in order to make use of the data. Also not all images are of useful quality… So I would be afraid of drowning in useless stuff, either by missing information or by useless quality.

Do you have any feedback from the groups? How actively are they using the OMERO server or is this just a data graveyard?

I don’t think the “data graveyard” issue is any worse than it was before OMERO - people who aren’t doing much with their data now are the people who would just have transferred all their acquired data to a storage server somewhere and not do much with it anyway. We’ve found that, even for the “data graveyard” users, they’re still coming back to OMERO to prepare figures for talks/publications. In addition to that, we have a good core of heavy users who actually care about their data management, and for them life is now much easier than it was. Sharing, annotating and publishing take a fraction of the time it used to. Of course, there’s no way to force anyone to use anything, so we still have users who will just acquire data, save it outside the specified Data folder and do whatever they used to anyway.

In general, I don’t think life is worse than it used to be for any kind of user, and it’s definitely much better for a certain subset. I’ll consider that a success!

2 Likes

Cool! Thanks a lot for the very nice discussion :slight_smile: I would love to push such an image database at some point at my place too. Have to prepare the ground first though.

1 Like

Thanks much for the information! How do you handle populating metadata? I am thinking about more study-specific data and not that which is auto-populated by the imaging software.

Many good comments, thanks for sharing everyone! We run Omero on Centos7 in AWS using the dual server and webserver configuration. I struggled to install it, but it was only mildly difficult for the capable sysadmin person I have access to. It’s been super stable and big timesaver. Our images are often a few Gb in size and viewing the images is fast and powerful. I wish I had tried Omero sooner. The Figure tool alone is worth the price of admission. Promote whoever designed that immediately.

Arborvitae’s question about study metadata was a big one for us. For plate based work there are built-in annotation tools, but bulk uploading annotations for a “Dataset” thru the webclient is not currently possible. It is easy to annotate manually using the UI, but that is not sustainable if there are many images and/or many annotations. I tried using tables, but it turns out that the Parade tool for searching annotations can’t handle a mix of numbers and strings (numbers only, again future versions may include this function) so I switched to key-value pairs and figured out how to use the command line interface (CLI). The CLI is bundled with the server and is very useful.

This is an example of how to make that work.

Prerequisites:

  • Omero and the metadata module. The metadata module is not the same as the Populate Metadata function and which can import metadata for a plate or a screen (many plates), but not a Dataset at this time.

  • Images that have been uploaded to Omero. Many ways to do this; we use the InSight client.

  • a .csv with the annotations. If using excel save as a “Comma Separated Values (.csv)”. Other formats might not work and always use a text editor that plays nice with the unix kids in the sandbox. Image Name and Dataset Name are required. Use the image name given by omero and the dataset created during image upload. This is a screenshot of what that might look like in excel:

50%20PM

  • a .yml file to tell the metadata module what to do with the annotations. Shown is a screenshot in TextEdit that I adapted from a dataset in the IDR (website mentioned earlier worth investigating). Add as many columns as necessary, but make sure the name matches the column name (exactly) in the .csv file. Everything after the last “include: true” is required, but not doing anything in this example:

Both the .csv and .yml files need to be in a place that the CLI can access. It’s easy to upload files through the webclient or InSight tool. I struggled to link them to CLI commands. I use sftp to move the files to the omero server from my laptop and I make sure they are owned by the omero user.

Then, finally, these commands can be run on the server using the CLI (as the omero user with admin privileges):

[omero@ip home]$ ~/OMERO.server/bin/omero metadata populate Dataset:382 --cfg omero_test-bulkmap-config.yml --file omero_test_annotations.csv

[omero@ip home]$ ~/OMERO.server/bin/omero metadata populate --context bulkmap --cfg omero_test-bulkmap-config.yml Dataset:382

and then, if all goes well, the images are annotated with the key value pairs from the .csv:

now the annotations can be used to search for images with the Parade tool (another module that needs to be installed). Here the image matching two key value filters is shown in the middle panel:

if the annotations need to be updated this command will delete them and one can start over with the above commands:

~/OMERO.server/bin/omero delete Dataset/MapAnnotation:382

7 Likes

At the moment we don’t - we leave that up to the user. Now seeing the work @johnmc has done we might actually give it a shot!

1 Like