Restoring single images in the Omero database

Hi all,

the official Omero documentation tells well how to restore the Omero full database via pg dump, but how about specific entries?

The standard use case would be: a user deletes important images by accident. We can restore them on the storage side from the backup system, but we still would need to restore the Omero DB entry for the annotations, attachment, hierarchy, etc. So, how would the full procedure for this use case look on the Omero server/DB side?

Sorry if this is a trivial question, but any explanations are much appreciated.

Thanks in advance
Anna

It’s so far from trivial that it’s essentially unsupported. (-: Far better might be to fire up another OMERO instance in a throwaway VM sitting atop a binary repository and DB restored from a pre-deletion backup, import the files afresh into your production server then write a script that uses the Blitz API to copy any post-import metadata (analysis results, manual annotations, etc.) between servers (probably the query service on the VM to the update service on production).

Once you delete an image you will see a number of EventLog model objects get created noting each related thing that was deleted. The query service will provide these; they live in the DB’s eventlog table. This records what was deleted but not much of what the previous database rows contained. However, you could probably use the object types and IDs to figure which rows to restore from a DB backup. Additionally, omero delete --dry-run --report Image:63,64,65 run on your restored VM will report exactly what is deleted from the database for the images in question, with IDs 63 thru 65 in this case. I am not aware of anybody who has used this to restore deleted data and I am curious as to the outcome of attempts! I do think fixing up a fresh import is probably the best way to go though, also repeating any attachments / tables from the backup storage instead of trying to “fake” them post hoc.

Hi Mark,

thanks a lot for your detailed answer. I’m really surprised that this is not a standard procedure. Especially when thinking about moving also the analysis of images into Omero and saving ROIs, results, graphs, etc. in Omero such a restore procedure would be a fundamental part to assure the users trust into the system. If we want them to use Omero as central data and analysis storage, we need to assure a reliable backup and restore. Until now, our users are not using the annotations much, therefore after restoring the images they basically re-upload them into the system. But this will not be so simple then, when for example attached measurement results are lost and that is most likely the direction for the future.

Maybe this is something you could think of for future releases? A function to export and import single images (or datasets?) with all their metadata? Isn’t that also something interesting, when people want to copy their local Omero data for a publication into the IDR? Then all the original information could be imported into the IDR without further manual effort.

Cheers,
Anna

Hi Anna,
@mtbc will be able to respond in great detail but for a quick preview: yes, there is a utility to export individual images and collections of images (datasets, projects, screens) incl. most (or all?) of the associated metadata and annotations. It’s OMERO.downloader at: https://github.com/ome/omero-downloader
Cheers,
Damir

OMERO.downloader does cover some functionality but only in a very limited way. OMERO has its own object model that is closely aligned with the OME Data Model but not the same; code in the server translates between the two and has very incomplete coverage either way. For example, sure, you can ask the downloader to fetch a project but it will just give you a set of images with no metadata about their containers. Indeed, if you encoded the OME-XML for those containers into an OME-TIFF or whatever and imported it into OMERO, the outcome would disappoint you. The downloader can include much metadata on the image but not all. For example, it currently omits attachments and masks. In short, it can be helpful but it is rather incomplete. For plates you will really notice what it lacks: default OMERO.server configuration disables download of original files for plates and even in an OME-TIFF export the downloader will not include that container structure, just a set of fields identified by image ID.

Fuller coverage by OMERO.downloader and fuller facility in general for transferring data out from and into OMERO.server is indeed both possible and desirable, it is simply a matter of what we can get funded. We are happy to advise others who wish to do the work and if anybody wishes to fund us to do it then we would love to get on with it. There is a long list of such work: we add or enhance features as current funding directs and try to find a way to someday get the rest done anyway!