OMERO extension in QuPath: working with raw pixel data/fluorescent images?

Dear all,

At our institution, we are continuously encouraging our users to work with QuPath for their analysis on both fluorescent and RGB images. And for quite some time now, we have also adopted OMERO and are trying to convince users to use it for their data management.
But we noticed that the OMERO extension in QuPath is currently handling only rendered images, meaning that RGB images stored in OMERO can be opened with QuPath, while fluorescent images are not supported.
We are wondering if the OMERO extension in QuPath has been developped from the OME team or on the QuPath side?
Is it planned to access OMERO’s raw pixel data from QuPath via the extension?
Or is it something that will be handled directly from OMERO, i.e when raw pixel data will be provided via the web in the form of zarr data?

All the best,
Claire

3 Likes

Sounds similar to:

1 Like

QuPath, very earliest version by me, much much improved version by @melvingelbard :slight_smile:

Ideally yes – but as mentioned on the other thread, as far as I know it isn’t currently possible via the web API. Therefore in the near future it will remain restricted to RGB images (even with the new features in v0.3).

I hope so! We definitely want to be able to access raw pixel values in QuPath, if that can be done without introducing too much extra complexity / more dependencies.

For now, the QuPath-OMERO extension is really for image viewing and (in v0.3 at least) annotation. It may also be ok for some kinds of analysis, particularly on brightfield histology slides, if the extra JPEG compression artifacts don’t matter much.

4 Likes

Me too!

Happy to discuss how to make this happen now. There are two pieces. On the server-side, omero-ms-zarr will allow sharing public images now. So for a full solution, it will need to learn authentication. Having a client-side to drive that would be helpful. :wink:

The client-side need not wait on omero-ms-zarr, though. There are sample OME-Zarr data available now which QuPath could test against (sans login). The trick is going to be deciding on which library stack to use: bioformats2raw uses https://jzarr.readthedocs.io/ but it needs a Java FileSystem implementation for remote access. I’ve successfully tested S3-based FileSystems (S3 with Jzarr by joshmoore · Pull Request #89 · glencoesoftware/bioformats2raw · GitHub) but haven’t done anything with pure HTTP implementations. That being said (and knowing @petebankhead) it is all just GET calls, so :man_shrugging:

Looking forward to comments/questions.
~Josh

5 Likes

Dear all,

Since we are heavily relying on both OMERO and QuPath, we have been inspired by @phaub’s QuPath scriptings and are now considering the idea to develop a transformed image server in QuPath to access data from OMERO using the RawPixelsStore(s) from the OMERO API, like we did in this early-stage BDV-OMERO connector.
Using the RawPixelsStores would allow us to access raw pixel data from OMERO in QuPath, thing which is currently not possible with the OMERO extension in QuPath since it relies on the OMERO web API.

Do you think it is reasonable and could be useful to start developing this, while awaiting the future stream of raw pixel data from the web API via omero-ms-zarr?

We’re happy to get your feedback on this !

Cheers,

Claire
@NicoKiaru @oburri @romainGuiet

4 Likes

This sounds like something we should discuss next week at #ome2021! (Perhaps during your demo slot?)

In general, big thumbs up to enabling what QuPath needs. It may be more a question for @petebankhead, @melvingelbard, et al. but a heads up that if the desire is to use a web API rather than Ice, I would strongly urge that that web API just be Zarr.

~Josh

3 Likes

A very interesting topic for me as well: other software I’m involved with would also like to pull full raw pixel images from OMERO and avoid the ZeroC-Ice complications (since that adds a huge and messy dependency to the build). I’ve had my eyes on the omero-ms-pixel-buffer (GitHub - glencoesoftware/omero-ms-pixel-buffer: OMERO pixel buffer Vert.x asynchronous microservice server) and/or omero-ms-image-region (GitHub - glencoesoftware/omero-ms-image-region: OMERO image region Vert.x asynchronous microservice server) microservices. I haven’t had the courage yet to try set those up on my open-source OMERO server but they look to be very solid on our institutional Glencoe OMERO server.

Am I off-base or would that be a way to achieve what @stoffelc (and I) wants to do but without needing the whole Ice-based OMERO API on the client side? Of course, we would need to get those microservices installed on our OMERO’s first … but I just noticed there’s now a Docker container with some of that included: GitHub - ome/docker-example-omero-microservices: Example of running OMERO.server microservices , at least a way to test it.

@joshmoore Any wise words on that concept?

I’ll try to join Claire’s demo slot next week but being on the US west coast, that’s a tall order.

2 Likes

Hi @stoffelc and everyone,

It took a while to respond because we’re in the process of making some rather big structural changes that involved rewriting all our build scripts – and separating out our OMERO web extension into its own repository, in a move to become more extension-friendly overall. I was waiting to see how these would look before replying, in case the plan fell apart. But it seems to be looking ok.

Sounds good! I did look into raw pixel axis a long time ago (2018?), and I faintly recall having something moderately working. However, as @dsudar mentions, it was the dependencies that stopped me. QuPath doesn’t have a nice jar management system, and it seemed like too much of a maintenance headache for us to take on – and since our institute hasn’t quite adopted OMERO yet, there was no real incentive to prioritise it.

When I saw the web API addressed a large percentage of our needs without introducing new dependencies, and learning Zarr is on the horizon, that became our focus.

One way or another, we definitely need to be able to get raw pixel values into QuPath. It may be more of a @joshmoore / OME team question what the likely value / lifespan of a RawPixelsStore-based extension might be.

In terms of QuPath distribution, it would have to be an optional extension and that’s certainly easier if it’s limited to just one or two files. I’m not sure if a fat/shadow jar is an option, or if there’s a limit to quite how many dependencies can be squeezed in.

I think ‘transformed’ shouldn’t be necessary – that’s just if you want to do something to an existing ImageServer.

You can implement an entirely new ImageServer, with its companion ImageServerBuilder, to add any kind of other image reader to QuPath. The easiest pattern to follow is for OpenSlide, since the extension doesn’t do anything else: qupath/qupath-extension-openslide at main · qupath/qupath · GitHub

As long as you have the magical services file it should be auto-discovered when added to QuPath’s extensions directory.

The tricky bit (which I’ve discussed with @phaub a few times) is that any new servers should be JSON-serializable in a maintainable way. QuPath achieves that for its own servers with the help of the (badly-named…) ServerBuilder interface. In simple cases, you can use a default implementation that takes the name of the implementing class and corresponding URI(s), like we use with OMERO web here.

Whenever transformations are involved, the serialization and maintainability is a lot harder to achieve. Server transformations are fun, but can be difficult to support and that’s why QuPath doesn’t use them as much as it could. If you do use them, I’d suggest relying as little as possible on other QuPath classes – because if an API-breaking change occurs in a later release, it will make the entire image unreadable.

The new QuPath OMERO extension adds support for browsing OMERO servers to select images (as demo’ed by @melvingelbard in his lightning talk), and sending annotations in both directions. Requesting pixels is really not so much (i.e. just ImageServer and its associates).

It would require some changes to the current extension, but perhaps we could broaden it out to request pixels using any of the following

  • Web API (RGB-only)
  • Zarr
  • RawPixelsStore

The last two would require more code to be written, perhaps separately, but if these focus only on pixel reading and not on selecting images & adding to projects then it is probably not such a huge amount.

Do you mean for requesting pixels, or more broadly? It doesn’t feel good to be requesting rendered (and JPEG-compressed) images as we do right now, and I’d very happily get away from that. But I presume we’d still need the current web API for selecting images, sending ROIs – right?

In any case, we’d want to follow whatever you and the OME team strongly urge :slight_smile:

6 Likes

Of course, we can always arrange a follow-up call. :slight_smile:

You’re right, and thanks for bringing it up. pixel-buffer will get non-rendered image data via a web API, and as with RawPixelStore, no reason not to use it right now.

In terms of the long-livedness of these APIs, our hope would be that the dependency on Ice will go first, then custom OMERO APIs next, leaving a generic, community approved NGFF API, which at the moment is Zarr.

For requesting array data in general.

Yes.

~J.

2 Likes

Is this post helpful here in any way ?