Bioformats Image Reader : reading from file ? a URL ? S3?

Dear @OMETeam,

I’m using in this repo the method IFormatReader::setId to access data from a file. I noticed that it also works if I enter a URL to open a file located on the web. But how good is the URL support ?

It works for small files (I tested a random png found on the web). However it looks like that if the file is big (like a ome.tiff of a few GB), then it takes a long time… before the setId method actually failed. I had hoped that maybe it would work smoothly like in Avivator (http://avivator.gehlenborglab.org/) by @ilan-gold. I tried as an example the URL of this demo image from Avivator : https://viv-demo.storage.googleapis.com/LuCa-7color_3x3component_data.ome.tif . But it doesn’t work at all. I get this error (and I have 4Go RAM for a file of 680Mo):

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.StringBuffer.toString(StringBuffer.java:669)
	at loci.common.Location.list(Location.java:600)
	at loci.common.Location.list(Location.java:1033)
	at loci.common.Location.isDirectory(Location.java:943)
	at loci.formats.in.BDReader.isThisType(BDReader.java:119)
	at loci.formats.ImageReader.isThisType(ImageReader.java:864)
	at loci.formats.ImageReader.getReader(ImageReader.java:193)
	at loci.formats.ImageReader.setId(ImageReader.java:844)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:662)
	at ch.epfl.biop.bdv.bioformats.bioformatssource.BioFormatsBdvOpener.auto(BioFormatsBdvOpener.java:271)
	at explore.OpenAndDisplayImageFromWeb.main(OpenAndDisplayImageFromWeb.java:31)

Process finished with exit code 1

Questions:

  • Is there a chance giving a URL refering to a big file like the one mentioned above may work smoothly? (lazy loading…)
  • What can be given to the setId method apart from a filePath ?
    • a link to a S3 bucket (it looks like there are S3 related stuff in the source code) ?
    • can we enter somehow an omero address ?

Best,

Nico

1 Like

Bio-Formats accepts URLs such as https:// and s3:// to image files. For example:

showinf s3://s3.example.org/BUCKET/KEY/image.jpg
showinf https://s3.example.org/BUCKET/KEY/image.jpg

OMERO isn’t supported as a connection type, since the OMERO API itself provides an interface to images.

Bio-Formats doesn’t have special handling to take account of the increased latency of remote requests compared to typical filesystems. Most image readers assume they can seek backwards and forwards to any location in a file, and often make very small reads of a few bytes before seeking again. Changing this requires modifying the low-level internals of the image readers for every format supported by Bio-Formats.

The bio-imaging community is instead working towards a new open cloud-optimised image format which solves several of the problems of dealing with large images, both local and remote. If you’re interested in finding out more please follow the #ome-ngff tag on this forum.

3 Likes

Thanks for your answer!

Is there somewhere a demo s3 server ? compatible with bio-formats ?

How different is the OMERO API compared to Bio-formats ? Totally unrelated ? Is multiresolution supported ? I’d like to see how difficult it is to import image file from omero into BigDataviewer - and using mutiresolution support.

@petebankhead I believe there’s an Omero importer into QuPath, by any chance could you point me to some relevant code that you made for opening (maybe there’s an OmeroImageServer?)

I look forward to the arrival of #ome-ngff !

@NicoKiaru the imagej-omero repository might be of interest for you:

and in particular @awalter17’s presentation slides here:

1 Like

Hi @NicoKiaru, see https://github.com/qupath/qupath/tree/main/qupath-extension-omero/src/main/java/qupath/lib/images/servers/omero
Entirely based upon the web API (with the limitations that implies…).

@melvingelbard is working on more OMERO-related changes – he’s the authority on QuPath’s OMERO support from now on :slight_smile:

2 Likes