Omero-parade: accepting StringColumns from omero-metadata and loading multiple plates

Hi OME team,

I’m checking if OMERO.parade is the right solution for a new thing we are trying to do and ran into a couple issues that had been raised before.

  1. when loading a table with the omero.metadata CLI plugin, any columns with numeric data are imported as StringColumns and thus do not allow visualization in OMERO.parade’s graphing tool. @will-moore pointed me at the --columns argument in populate_metadata.py (see: https://github.com/ome/omero-parade/issues/47) and while that works, it would be better to have that available from the CLI.
  2. in the current omero-parade version 0.1.3, multiple plates (or multiple screens) functionality does appear to be there yet. @chris-allan pointed at PR #39 (https://github.com/ome/omero-parade/pull/39) but that doesn’t appear to be merged yet. Any idea when this will be part of OMERO.parade?

Thanks,
Damir

1 Like

Hi Damir,

I don’t know about PR #39 I’m afraid. We’ll have to discus and let you know.

But for the --columns support in CLI, I recently added documentation to the metadata plugin to show how you can add a # header to the CSV itself to specify the column types: See https://github.com/ome/omero-metadata/#populate
Let us know how that works for you.

Regards,
Will.

Hi Will,

Working late, I see. Thanks for the quick reply.

I hadn’t noticed that latest omero-metadata improvement but that’s indeed great. So that works in version 0.4.0? I’ll try it right away. Great to have that “# header” format line integral to the data file itself.

Re: having Parade work for multiple plates in a screen, I’m not missing something else, right? I’m looking forward to hear what you find out re PR #39.

Thanks,
Damir

Hi Damir

The PR #39 you indicated has been excluded from the build system. It works fine for a small number of plates/screens but it highlighted some major issues when working on a larger scale (more realistic).
That initial work should be considered as experimental.

As it stands, we won’t be making it available as part of an official release since we know
most users will eventually want to use it at scale.
To make that possible, we are thinking of new designs. When ready, we’ll make them available for discussion. That’s probably months away due to other priorities.

Sorry for not being more positive

Thanks
Jmarie

1 Like

Hi Jmarie,
Thanks for the response and too bad it isn’t the answer I was hoping for :slightly_frowning_face:
Being able to use Parade on a collection of plates is exactly where it will shine.

I have a project with 18 plates where each plate has 21 wells occupied with 9 images each. So total is 3402 images. Would that still be within the magnitude where a version with PR#39 merged would work or will it likely fall over? I’m wondering whether to try and build a Parade so I’d appreciate your gut feeling on that.

Thanks,
Damir

Hi Damir,
I’ve just tried playing with that branch, merging in the latest master branch and it is working fine for me. See https://github.com/ome/omero-parade/pull/60
I think you should be fine with the number of plates, especially since you’ll never view all those thumbnails at once (only a single field at a time). The default view doesn’t even load thumbnails unless you choose to view them.

If you checkout my branch, have npm installed, then you can build the JS and make sure the omero-parade directory is on your PYTHONPATH.

cd omero-parade/
npm install
npm run build

export $PYTHONPATH=PYTHONPATH:/path/to/omero-parade/

This should then work with omero settings omero.web.apps etc same as for the pip-installed app.
Hope it works OK for you.
Cheers,

Will.

Testing with plates of various sizes:

Hi Will,

I really appreciate that you tested this out and that it appears to be workable. I’m trying to follow your instructions (on Ubuntu 18.04 with npm and stuff from the regular Ubuntu apt repo) and am running into an error:
sudard@lincs:~/src/omero-parade$ npm run build

> omero_parade@0.1.3 build /home/users/sudar/src/omero-parade
> webpack --progress --config webpack.prod.js

/home/users/sudar/src/omero-parade/node_modules/webpack/bin/webpack.js:90
let notify =
^^^

SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions…js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Function.Module.runMain (module.js:442:10)
at startup (node.js:136:18)
at node.js:966:3

npm ERR! Linux 4.4.0-53-generic
npm ERR! argv “/usr/bin/nodejs” “/usr/bin/npm” “run” “build”
npm ERR! node v4.2.6
npm ERR! npm v3.5.2
npm ERR! code ELIFECYCLE
npm ERR! omero_parade@0.1.3 build: webpack --progress --config webpack.prod.js
npm ERR! Exit status 1
etc…

In the “npm install” step I noticed some warnings about versions of node.js components. Am I working with a version of npm/node.js that’s too old maybe?

Thanks
Damir

@dsudar,

Will is away, but I ran into issues recently when trying to build omero-parade. Looks like something may have recently changed in the upstream packaging. Ultimately, https://github.com/ome/omero-parade/pull/59/commits/28e3507f8c1c494fd1090d8a1716e061163292ff seemed to work if you want to give that a try.

~Josh

Hi @joshmoore,
Thanks for jumping in. I tried the changes in the PR you referenced and a few more changes to versions of nodejs/npm packages but didn’t get any further. I’ll wait until @will-moore is back and hope he can find a bit of time to look at it. Javascript development/deployment tools are well outside my area of expertise.
Thanks both,
Damir

Hi Damir,
I’m not sure what’s causing your error. I’m using

$ npm -v
6.9.0

One thing you can try is to use

$ git reset --hard HEAD  # make sure package-lock.json is unchanged
$ npm ci      # use exact versions from package-lock.json

instead of $ npm install (see https://docs.npmjs.com/cli/ci.html).

Otherwise, looking at Josh’s link, I wonder if using Docker to build would be an option?

Will. (still away)

Hi @will-moore,

Sorry to bug you while you’re on vacation.
I deleted my previous attempts and started again with a freshly git-ted setup and with npm 6.9.0 and node.js v10.16.0 and then it built just fine as you showed earlier. Should have tried that before … and not wasted your time.

So now I can indeed select the whole Screen worth of plates and Parade shows the whole list. Unfortunately, most everything else doesn’t work. My bulk_annotations table is attached at the Screen level but nothing shows in the “Add data…” drop-down except ROI_count and sizeT but even those don’t do anything. Also, I have created K-V pairs at the Well level (and also replicated at the Image level) (using omero metadata populate --context bulkmap …) but they do not show up in the Pick Key list in Add Filter. And I do have ROI Counts at the Image level for each image.

Thanks for any ideas how to proceed.

Cheers,
Damir

Hi Damir
Will did some minimal work on that branch so it could be re-included in the build.
We are not indenting to push that work further.
It will be preferable to move the discussion to GitHub so your workflow is captured and easily accessible. Could you open an issue (https://github.com/ome/omero-parade/issues) describing how your data are stored and at which levels elements are linked to e.g. wells, screens etc.? So we can take that information into account for newer versions of omero-parade.

Thanks in advance
Jmarie

Hi Jmarie,
Thanks and yes, I’ll do that.
Cheers,
Damir

Hi @dsudar. I used my script at https://gist.github.com/will-moore/31c25fdbfc4dafebe245 to create some sample values for my testing above.

Do you see the “Add Data” options listed under /parade/dataproviders/?screen=[screen_id]?
Any errors on the developer tools console?

ROI_count uses URL /parade/data/Uk9JX2NvdW50/?screen=[screen_id] to load ROI count for the first image in each Well. Does that URL work for you?

Will.

Hi Will,

Hope you had a great time off. And thanks for helping with this - just a quick explanation: we have this one data set of 18 plates that is featured in a forthcoming paper. We would like to give public access to the image data but also a good tool to visualize what is going on in that data set. Parade pretty much offers exactly what’s needed. So great that you are able to give the pointers I need.

With those pointers I figured out that while I was getting the correct .js static files, my PYTHONPATH settings was not honored (because of the web’s virtualenv) so I was getting the vanilla parade 0.1.3 python files instead. Now fixed and so far things are working well.

One quick question: how do you tend to deal with multiple fields per well? I have 9 fields in each well and a rich set of measurements per image. As it currently works, Parade only uses the measurements from the first field.

Thanks much,
Damir

PS. I’ll submit further feedback for feature requests and other suggestions directly via the Github repo as @j.burel suggested.

Hi Damir,
Glad to hear it’s working.
Currently we don’t handle multiple fields per Well, mostly because we don’t have a good idea of what the UI would look like (at least in the Plate layout filtering and heatmap). If you have any ideas, it would be great to hear.
I guess for the table (list) view we could imagine listing Images instead of Wells, and the same for the scatter plot, but we’d have to think about how you’d toggle from this viewing Images vv Wells and switching to the Plate layout view.
Regards,
Will.