Using OMERO.parade on dataset-level and image-level OMERO.tables

[stoffelc (Claire Stoffel)]

Dear OME team,

Users at our facility are discovering the power of OMERO.tables for data management and of OMERO.parade for data mining. However, they noticed that they are only able to use the parade tools when the OMERO.tables are generated at the project level, and not at the dataset nor the image levels.

Would it be possible to make OMERO.parade read dataset- and image-based OMERO.table data?

For some experiments, it is really useful to create separate tables at different hierarchy levels. Therefore, being able to pick a level-specific table in the add filter... dropdown menu of OMERO.parade could be a really nice functionality for example!

All the best,
Claire

[will-moore]

Hi Claire,

Thanks for the feedback. Great to hear that OMERO.parade is being useful.

OMERO.parade can partially work with Datasets, although I realise that loading Filters from an OMERO.table on a Dataset doesn’t work.
That’s a simple fix: I just opened a PR to fix it at Support filtering by OMERO.tables on Dataset by will-moore · Pull Request #86 · ome/omero-parade · GitHub
It’s not tested yet, and it might be a while before this is released. But feel free to grab those changes and try it locally if you wish.

In the meantime, I noticed that if the Table is in the form of a CSV file on the Dataset, this can be used for filtering.

When you say image-based OMERO.table data, do you have rows in the table corresponding to ROIs on the Image? Or are they other measurements?

At the moment, OMERO.parade only filters or plots Images, so every row of a table has to represent an Image (or a Well).
I have been thinking about removing this restriction, which allows a lot of other functionality. You can see some of this thinking in a prototype app at GitHub - will-moore/parade-crossfilter: Prototype OMERO.parade implementation using crossfilter.js
There is a demo video that includes a Table (csv) on a single Image.
A more recent video (showing an improved UI and some newer features) is at Feedback fixes by will-moore · Pull Request #4 · will-moore/parade-crossfilter · GitHub

This is a side project that is only moving slowly at present. But any feedback on those videos would be interesting to hear.

Many thanks,

Will.

[stoffelc (Claire Stoffel)]

Hi Will,

Thank you very much for your comprehensive answer.

@will-moore

OMERO.parade can partially work with Datasets, although I realise that loading Filters from an OMERO.table on a Dataset doesn’t work.
That’s a simple fix: I just opened a PR to fix it at Support filtering by OMERO.tables on Dataset by will-moore · Pull Request #86 · ome/omero-parade · GitHub

That’s great! It’s exactly what we need.

@will-moore

In the meantime, I noticed that if the Table is in the form of a CSV file on the Dataset, this can be used for filtering.

I am not experimenting the same thing here, I can only use the filtering on tables in the form of CSV files when they are attached to projects. CSV files attached to dataset do not show up in the filtering dropdown menu of OMERO.parade.

@will-moore

When you say image-based OMERO.table data, do you have rows in the table corresponding to ROIs on the Image? Or are they other measurements?

Exactly, I have rows corresponding to ROIs. Therefore, the parade-crossfilter prototype sounds really promising! The new functionalities it includes are very interesting as well.

All the best,
Claire

[will-moore]

Hi Claire,

I think there must be some other issue you’re having, since it should work with a CSV on a Dataset.
I just tried a very simple example, creating a demo-test.csv file with Image IDs like this, using a text editor (I tried with Excell first but it adds hidden characters that can break things):

Image,Treatment,Duration,Phenotype
15326,Drug1,10,OK
15328,Drug2,15,Alive
15327,Drug3,20,Dead

Having attached this to a Dataset containing those 3 images, I can choose it as a filter in Parade:

Although it is only the Number columns that allow me to filter:

It might be that Parade is not handling the csv name correctly (avoid spaces, slashes etc. - is OK).
And try a small, simple example like my test above and see if you can get that working.
Also you could try looking in the browser dev-tools Console and Network tabs to check for any errors.

Hopefully we can identify the issue,
Cheers,

Will.

[stoffelc (Claire Stoffel)]

Dear Will,

Thank you very much for your answer, it helped me to identify more precisely what’s going wrong here:
The point is, Parade DO indeed recognize .csv attached on datasets when the dataset is not included in a project, like shown here, and as in your example:

However, when the dataset is located inside a project, the .csv attached at the dataset level do no show up in Parade, even when there is no other .csv attached to the project nor to an other dataset from the same project:

All the best,
Claire

[will-moore]

Hi Claire,
You’re right. I just reproduced what you’re seeing. But after linking the csv (that was on the Dataset) to the Project itself, things started working again.
The same behaviour will apply to the filtering by OMERO.tables on Dataset fix at Support filtering by OMERO.tables on Dataset by will-moore · Pull Request #86 · ome/omero-parade · GitHub
If the Dataset is in a Project then it’s only the Project that is queried for the available filters etc.
Changing that assumption would require a bit of thought, since the list of all the tables/csvs on the different Datasets within a Project could get quite long and each one would only have data for a subset of all the images you’re filtering in the Project.
So you could end up with quite a heterogenous mix of partial filters - e.g. Filter by Cell_count from Dataset:1 table and Cell_count from Dataset:2 table would currently end up as 2 filters (loaded from different sources) so you’d probably want to combine them? Or would you only load filters from 1 Dataset at a time?

Regards,
Will

[stoffelc (Claire Stoffel)]

Hi Will, thank you for your answer!

@will-moore

So you could end up with quite a heterogenous mix of partial filters - e.g. Filter by Cell_count from Dataset:1 table and Cell_count from Dataset:2 table would currently end up as 2 filters (loaded from different sources) so you’d probably want to combine them? Or would you only load filters from 1 Dataset at a time?

If I had for example the Cell_count entries in both Dataset:1 and Dataset:2, I think that I would create a table at the project level rather than at the dataset level. I would create tables at the dataset level only if the table entries are not shared between the different dataset from a same project. This also means that I would more likely load filters from one Dataset at a time.

Would it be possible to only load the filters from the datasets that are opened in the left-hand tree in OMERO.parade for example?

All the best,
Claire

[will-moore]

Hi Claire,

OK, I think I can understand how this could work. I’ve created an issue at Load Tables from Datasets within a Project · Issue #88 · ome/omero-parade · GitHub
But this could be quite a bit of work, so I don’t know when it might be tackled.

I don’t think it would be easy to only look for filter option on the expanded Datasets since the list of filter options is loaded when the Project is opened in parade and doesn’t dynamically update after that.

Thanks for the feedback,
Will.