Are there any fundamental limits to OMERO? For example, FIB-SEM can create +100GB-sized images. And 3D CLE microscopy can produce images in the TB range. What are the limits in terms of individual size and volume/throughput on images like these?
On a per-experiment basis, raw data size is mostly limited by the available storage since OMERO largely just manages that storage. The Image Data Resource (IDR) which is based on OMERO has individual files which are over 400 GB and experiments consisting of 500 plates totaling 25 TB.
There are of course data types that cause more problems than others:
- Filesets with large XY planes (10s to 100s of thousands pixels wide and/or high) that do not come with their own pyramid of subresolutions require OMERO to generate pyramids in the background which can become a bottleneck. (Here is is often beneficial to generate the sub-resolutions beforehand.)
- Another bottleneck arises from very large high-content screening plates with millions of database rows of metadata. Importing such a plate can take many hours. (At the moment, there’s no substantial workaround beyond optimizing your database.)
The question of throughput depends on several factors including upload vs. download, file format, concurrent users, concurrent filesets (i.e. is everyone looking at the same image, etc.). There are a number of things that can be tuned depending on how you want to grow.
Happy to help work through specifics as likely others on image.sc are as well!
This is a great reply! Thank you. Can you comment about OMERO’s capacity to display big 3D FIB-SEM datasets? Something that would come off of a new Titan Krios with a top-of-the-line camera system? You mentioned large XY planes; is there a better tool for generating those pyramids as a pre-processing step? Or is it better to let OMERO build those and maybe speeding it up by attaching a HPC or something like that?
From Neuroglancer vs OMERO.FPBioimage I see that you’re already finding what’s out there on 3D viewers, but I was going to say, none come with OMERO out of the box. Nonetheless importing of images should be feasible but I don’t know where limits lie. @paulkorir or someone else from the EMPIAR crew may be able to say more.
We’d strongly suggest using the new pipeline of tools from Glencoe: Converting Whole Slide Images to OME-TIFF: A New Workflow (which works on more than just WSI!) It’s built to perform parallel processing, with compression etc. being configurable. There are several posts on the forums under “bioformats2raw”, but let us know if you need any more info.
All the best,