Open an image from OMERO in ImageJ.JS

stoffelc · January 19, 2021, 2:58pm

ImageJ-JS looks really amazing!
I was wondering if there is a way to open images from an OMERO server in ImageJ.JS?

I added @jerome.mutterer’s Open in ImageJ.JS chrome extension and noticed that the option “open in imageJ.JS” appeared when right clicking on image thumbnails from OMERO.web.

So I tried, but the corresponding image did not open in imageJ.JS.

Then, I also tried to open OMERO images directly from imageJ.JS with File > Import > URL.. using the link to an image from our OMERO server, but to no avail.

I know that the OMERO plugin is required to open images from OMERO in FIJI/ImageJ, but was still wondering if ImageJ.JS could also provide that!

All the best,
Claire

will-moore · January 20, 2021, 11:52am

Hi,

Ideally, you’d want ImageJ-JS to work with the raw pixel values from the original image, and not the “rendered” jpeg that OMERO provides. OMERO doesn’t provide a direct way to access raw pixel data via it’s web API, except via the Zarr microservice (GitHub - ome/omero-ms-zarr: Experimental micro-service for implementing the Zarr spec over HTTP).
It would be great to put the pieces in place to allow full analysis of OMERO images in ImageJ-JS, but this will be a bit of work…

In the meantime, would it be useful to open the rendered images (e.g. for manual analysis or segmentation)?
I tried a bunch of images like a public image from IDR: http://idr.openmicroscopy.org/webgateway/render_image/9621401/0/0/
This is a jpeg but gave me scrambled text image when I opened in ImageJ.JS.
Maybe because the URL doesn’t end with .jpg?

I tried opening a .png from the OMERO login screen (https://omero.lifesci.dundee.ac.uk/static/webclient/image/logo_login_web.png)
But that image in ImageJ.JS
gives me CORS errors in the console (not seen that before with images).

Hope we can get something working…

Will

oeway · January 28, 2021, 7:03pm

Hi @stoffelc , sorry for the delayed reply. Would be happy to help with supporting this, in fact I have been discussing with OME team for the possibility of supporting ImageJ.JS and other ImJoy plugins in OMERO.

As explained by @will-moore it doesn’t allow raw pixel access from the web client for now, and I am also wondering whether there is a plan to support it? Or the plan is to support Zarr in future version of OMERO? Maybe Will can clarify a bit.

For now, we can support the rendered images, and I just submitted a PR to support open OMERO image url without file extension. Now we can do: ImageJ.JS (You may need to refresh twice to update to the latest version of ImageJ.JS).

To make it even easier, I submitted an request to the open-in-imagej.js repo: Support open images in omero web client · Issue #10 · mutterer/open-in-imagej-js-extension · GitHub and let’s see if @jerome.mutterer can help with the button.

jerome · January 27, 2021, 8:32pm

Hi @oeway, that’s very nice!
Your link didn’t work as is in Chrome or Firefox, I had to change:
open=http://idr.openmicroscopy.org/
to https

I will try to update the Chrome extension to support this.
Jerome.

oeway · January 28, 2021, 7:03pm

@jerome Thanks, just fixed the link.

will-moore · January 31, 2021, 11:26pm

@oeway That’s great, thanks.

We have started supporting raw pixel data via the web in zarr format using a microservice (see GitHub - ome/omero-ms-zarr: Experimental micro-service for implementing the Zarr spec over HTTP) and this works well for publicly available data, but it doesn’t support authentication so is not ready for production servers.

In fact, even with the current support of opening a rendered image from OMERO in ImageJ-JS, this seems to fail for non-public images. I think this is because the session credentials are not being included in the AJAX request.

So this may need a different approach.
Regards,

Will.

oeway · February 2, 2021, 6:57pm

Hi @will-moore , it’s great that the team is working on this.

You are right, we need a different approach. Currently we have the idea of injecting the ImJoy core with ImageJ.JS into the current page through the chrome extension, and another idea would be make an ImJoy extension for OMERO. Wondering if anyone in your team would be interested in that. More specifically, the idea would be injecting the ImJoy core to the omero frontend, such that any ImJoy plugin can be loaded to the page. We can register a zarr store service to the core, such that plugins such as vizarr, ImageJ.JS and ITK/VTK viewer can discover that zarr store service and use that.

will-moore · February 3, 2021, 4:00pm

Hi @oeway,

I am certainly interested in making an ImJoy plugin for OMERO. I could imagine offering “Open with > ImJoy” or “Open with > ImageJ.JS” options. I think we could start with the opening of an 8-bit rendered image, and progress to using the raw pixel data in due course.

To work around the failure of ImageJ.JS to load non-public images from OMERO, I wonder if I could open https://ij.imjoy.io/ in a window and then to call some API to pass in a base64 encoded image?

Another approach I thought of trying was to create a page that loaded the ImJoy core and the rendered image from OMERO, and then loaded ImageJ.JS plugin and passed it the image.
I started with a locally-hosted copy of imjoy-core/core-example.html at 493c39922082a6ffcf038f881e01592d7a48855e · imjoy-team/imjoy-core · GitHub, trying to load ImageJ.JS with /core-example.html?plugin=https://ij.imjoy.io but this failed to load the plugin with the error: failed to load the plugin, error: Timeout error: failed to connect to the plugin.
For my limited home bandwidth, it takes about 3 minutes for https://ij.imjoy.io/ to fully load.
Is there any way to allow imjoy.pm.reloadPluginRecursively({ uri: p }) to take longer?

Thanks,
Will.

oeway · February 3, 2021, 11:34pm

Hi, @will-moore, that’s great!

To work around the failure of ImageJ.JS to load non-public images from OMERO, I wonder if I could open https://ij.imjoy.io/ in a window and then to call some API to pass in a base64 encoded image?

Yes, that is a good idea. Or later on if we have the zarr data ready, we can even pass a zarr store object to imagej.js.

So assuming we loaded the ImJoy core to the webpage, and in a “Open with ImageJ.JS” button, we can do something like:

const image = ...
const ij = await api.createWindow(src="https://ij.imjoy.io")
await ij.viewImage(image)

This is the viewImage api.

Similarily, we can add buttons for “Open with ITK/VTK viewer”:

const viewer = await api.createWindow({src: "https://kitware.github.io/itk-vtk-viewer/app/"})
await viewer.setImage(image)

Or “Open with Vizarr”:

const viewer = await api.createWindow({src: "https://hms-dbmi.github.io/vizarr"})
await viewer.add_image(image)

For the timeout issue you have, I think the best way forward is allow to expose the plugin interface before the loading of imagej.js completed. Then you can just call imjoy.pm.reloadPluginRecursively and it will not hit the connection timeout issue.

I will change that in the ImageJ.JS and keep you updated.

will-moore · February 4, 2021, 2:39pm

Hi @oeway,

So I’ve made some progress, but not there yet…

Looking to do

const ij = await api.createWindow(src="https://ij.imjoy.io")
await ij.viewImage(image)

I couldn’t work out how to get hold of the api within the core-example.html page itself, so I instead loaded a plugin from that page. The plugin I created is at imjoy-starter/omero_image_ij.imjoy.html at 1e8473479d4002b4a89005afe3429665b4e76e9e · will-moore/imjoy-starter · GitHub

I added the image-loading code to the core-example.html using the query string ?open=image_url and this loads the image into a canvas, grabs the image buffer from the canvas and passes it to this function of the plugin:

async openImageJ(buffer){

        console.log('opening https://ij.imjoy.io....')
        const ij = await api.createWindow({src:"https://ij.imjoy.io"});

        console.log("VIEW IMAGE...", buffer);
        ij.viewImage(buffer);
    }

However, the last 2 lines never run (don’t see the console.log).

My core-example.html code is named index.html in omero-imjoy/omero_imjoy/templates/omero_imjoy at main · will-moore/omero-imjoy · GitHub and it can be run without OMERO for now.
I included a test.jpg in there so you can go:

$ cd omero_imjoy/templates/omero_imjoy/
$ python -m http.server

Then go to http://localhost:8000/index.html?open=http://localhost:8000/test.jpg and you should see:

Am I taking the right approach here? Any ideas what’s not working for me?

Many thanks,

Will.

oeway · February 11, 2021, 2:29am

Hi @will-moore Sorry for the delay, and I missed your last message until now.

Yes, I think you are taking the right approach, except you can change your plugin type to web-worker because you don’t really need a window for the omero_image_ij plugin.

It’s great that you have this already in a repo, I will try it and get back to you later when I tried it.

oeway · February 11, 2021, 2:35pm

Hi again @will-moore I just fixed the demo you made and sent a PR: Load imagej.js directly from the core by oeway · Pull Request #1 · will-moore/omero-imjoy · GitHub

The way you did can also work but it will need another plugin, in the PR I just call the imjoy api directly in the page, that remove the need of another plugin.

For the ij.viewImage api, it is designed to accepting numpy array which we don’t have in js, so we need to represent the image as an object, you can find a full table here: GitHub - imjoy-team/imjoy-rpc: The RPC library used in ImJoy.

Regarding the omero-imjoy plugin, can we inject the core directly to the omero page, so when the user right click on an image, we can open it in a panel in omero? Or it has to be an independent page?

will-moore · February 11, 2021, 11:34pm

Hi @oeway,

Thanks for the PR. Merged and making progress now…

I believe it will be possible to load Imjoy into the centre panel of the webclient. That could allow you to browse images in OMERO while opening multiple images in ImageJ.JS.

I’ll try and get the basics working as a standalone page first. And I think I will leave that as an option for users, since there may be times you want.a full page for ImJoy.

I’ll let you know when I have more questions, but the next step I’ll try is to handle multi-plane images (Z or T stack) by loading multiple planes, stitching them into a multi-dimensional ‘numpy’ array.
At some point, it would be nice to lazy load planes (virtual stack) but I don’t know if that’s possible yet, so I’ll load them up front for now…

Thanks for the help,
Will.

oeway · February 12, 2021, 5:49am

Hi @will-moore Sounds great! I am glad that you are working on this, and please don’t hesitate to ping me if you need anything from me.

Regarding the lazy-load planes, this can be done via by sending an object with functions via imjoy-rpc, and this how we support sending zarr store to vizarr. So we can easily support that for other viewers, but for ImageJ.JS, further work will required to expose the virtual stack API. I added an issue here: Expose virtual stack API · Issue #53 · imjoy-team/imagej.js · GitHub

will-moore · February 12, 2021, 8:42am

Great, thanks for that.
I should say that “working on this” is still very much a personal side project and not (yet) supported by the OME team. So it won’t be “production ready” until time and resources allow.
Cheers,
Will.

oeway · May 12, 2021, 7:25am

Hi @will-moore I took out your implementation for loading NGFF and wrapped as an independent ImJoy plugin: NGFF-Loader.imjoy.html · GitHub

Try the Load NGFF plugin menu here: ImageJ.JS

I started to play with the image url from your blog post: Public OME-Zarr data (Nov. 2020) · The OME Blog

will-moore · May 13, 2021, 2:50pm

Hi @oeway, that looks very nice.
It works well on smaller images, but I’m afraid that a lot of the example images we have are quite large. Loading all the data up front and stitching it all into one array can be prohibitive.

We are working towards version 0.2 of Next-generation file formats (NGFF) which will use “nested chunks” (where the chunk is nested into directories, e.g. 0/0/0/0/0). See Hierarchical chunk storage · Issue #29 · ome/ngff · GitHub

Trevor helped with upgrading vizarr to support this format (see Use Proxy store for nested chunks by will-moore · Pull Request #85 · hms-dbmi/vizarr · GitHub) which uses zarr.js. I don’t know if it’s possible to do the same with Trevor’s zarr-lite that we’re using here, or if we should switch to zarr.js instead? @manzt

Cheers,
Will.

manzt · May 13, 2021, 8:42pm

@will-moore. Do you know if the nested format in in v0.2 will add the dimension_separator field to the .zarray metadata? I should be able to cut a release of zarr-lite that respects that field.

Otherwise, the nested proxy from vizarr can be reused.

will-moore · May 13, 2021, 10:03pm

Hi @manzt,
Yes, the dimension_separator will be in v0.2. cc @joshmoore
That would be great to have that supported in zarr-lite.
Many thanks,
Will

joshmoore · May 14, 2021, 7:14am

Morning all,

I’ll add: nested storage is in 0.2: http://ngff.openmicroscopy.org/0.2

but of course, dimension_separator is in zarr-python 2.8 so you’ll need to update.

~Josh