Will `TileExporter` have a `TileImporter` sibling in QuPath 0.2.0?

Dear @petebankhead,

Congratulations on the latest QuPath release!

Looking at the changelog something really cool caught my eye

** New ‘Import objects’ option when adding images to a project
Supports importing ROIs/overlays from ImageJ TIFF images and OMERO**

Is this going to be linked to the TileImporter/Exporter I remember you mentioning somewhere? (I tried googling it and looking on the forum and readthedocs, but I seem to be missing the right keywords to find it).

Will the ability to export tiles, run some processing on them outside of QuPath, and reimporting the results (ROIs, binary masks) be something for this release or is it farther down the line?

Thanks for the info and congrats again.

Best

Oli

1 Like

Probably further down the line. Although if you can add your masks to an ImageJ TIFF you can get them in that way.

I remember the conversation you’re referring to, and it is still something I have in mind, but I expect it will be challenging to solve in the same kind of generic way as the TileExporter (what should happen with overlaps…?).

At the time of the first conversation, I thought the importer would be needed with StarDist, but in the end it could all be solved via directly linking to TensorFlow and working with geometries rather than rasters.

So right now I am tentatively thinking v0.3.0… although moderately tempted to think about it again if a use case is sufficiently clear, since it would be fairly self-contained and not impact the GUI.

How would you plan to use it? With Fiji macros, Python scripts… or something else?

2 Likes

I was thinking it would behave along the lines of AbstractTileableDetectionPlugin and make use of the logic that is implemented there (or somewhere near there).

We had created a StarDist-Python connection in Groovy by exporting tiles, running the native StarDist on them and re-importing RoiSet.zip files. We had to deal with the overlaps ourselves which worked pretty well.

Then when cellpose came along, we figured we could use it in the same way, especially considering how it needs GPU to be fast enough even for inference, is not tensorflow based and needs MKL as a dependency.

So we got this groovy script that exports the tiles, calls a conda or virtualenv environment to process the image folder, and then re-imports whatever RoiSets were churned out by the script.
Right now they are not generic, and we do not make use of TileExporter, but it would make a lot of sense to have the TileImporter as well…

That way our script is reduced to a ProcessBuilder command and some logic to handle normalization for the whole image rather than the individual tiles.

The idea of running python virtualenvs was that at least this way we will not have to reimplement deep learning pipelines in Java (For Fiji, or QuPath) each time we want to use them. DeepImageJ and CARE are great, but as you noticed when re-implementing StarDist, they do not contain the post-processing steps that make the algorithm so powerful (NMS suppression and Star-Convex shape estimation for StarDist).

This way we can use any pipeline from Python and it costs us one python script to handle the command line arguments that are needed.

Anyway that’s the direction we headed into and we’re super stoked that it worked out the way it did.

I was also playing around trying to make a QuPath command that checks for virtualenvs and allows you to run the AbstractTileableDetectionPlugin logic but I got a bit stuck with how to

  1. Export regions (in parallel if we want)
  2. Run the python script (But just once, this part is not parallel nor run one time per tile)
  3. Make sure that the separate regions with detections are now reimported properly.

But the Groovy script does enough for now let’s say

1 Like

Thanks for the description, I’ll think some more and reply properly later - but quick question: with RoiSet.zip do you mean an ImageJ Roi set?

Is the preferred way to return rois (in some form) or labelled/binary images… or all are needed?

1 Like

Yep a good old ImageJ Roi Set ready for the RoiManager.

We opted for Roi Sets because @uschmidt83 was kind enough to create an ImageJ Roi Set exporter for StarDist, which made sense as StarDists’ output is polygons.
That ended up being our starting point, because that’s what we knew how to do (ImageJ rois to QuPath PathObjects)

However this had been rather tedious for cellpose, as pretty much everything in python works through labelled images and not polygons (even though there are packages available).

Our though was that, because we wanted ROIs (or PathObjects) at the end of the workflow, it was more efficient to go as far as possible in Python rather than export yet another image, which needs to then get loaded into the software again. ROIs were clearly much smaller in size, so faster processing to some extent. (Untested, as converting to polgons from masks in cellpose is rather slow right now)

But now that I know more and more python, the most accepted approach is clearly to provide masks or labeled objects (the latter being more interesting as many objects tend to be touching), despite the extra cost of writing temporary image files somewhere.

Thanks for your time discussing this. Looking forward to hearing from you, and helping in any way that I can.

Good to know, I was wondering how you had got a RoiSet from Pythons! :slight_smile:

So what do you think of:

  • preferred labelled image type/bit-depth
  • cells (which have two rois)
  • overlapping objects
  • overlapping tiles
  • would you also want to make measurements in Python, or only in QuPath?

For cells I am thinking either two labelled images, two channels of one labelled image… although the information might also be packed into a single ARGB image.

Also, a labelled image could be 8-bit (probably too small), 16-bit (possibly too small?), any arbitrary type, or another ‘creative’ ARGB format.

(Questions are a bit scattered, but trying to establish the top priorities and to what extent the task can be simplified while still meeting (most) needs… while typing on my phone)

1 Like

Labelled images could be 32-bit. Leaves more flexibility at the expense of extra disk space. I think that at least this way most scenarios are taken into account.
In our experience, as we are dealing with tiles, 16-bit is also more than enough. With a warning message in case QuPath detects all 65536 bins are taken (“Please reduce the tile size”)

Cells: I like the idea of two-channel labelled images. The pixel ids are shared for each cell-nucleus combination. Issue might be how strict PathCellObjects are. I know cellpose can have a single cytoplasm with multiple nuclei, which happens in biology sometimes.

Overlapping objects: We basically implemented an Intersection Over Union (IoU) scheme like StarDist in order to allow for some overlap. In the case of excessive overlap, we keep the largest one, as per your approach in QuPath.
This has also worked when creating ‘cells’ from overlapping objects using what is done in WatershedCellDetection which we duplicated in order to create a method CreateCells that allows any PathObject to be expanded (though now you have a Expand Annotations Command that does it, but I have not tested it on overlapping objects).

Overlapping tiles Makes sense to have the overlap amount based on the expected size of the objects, to avoid any one object never being complete in any tile.

Measurements We have really not had the need. We hadn’t even considered it before your StarDist implementation provided the possibility to add the probability threshold to the measurements, which is really cool.
This is just a thought, but a JSON file with id, measurement_name, measurement_value where id is the value of the pixel in the labelled image seems like a simple scheme that could easily be implemented by whatever tool would need to use QuPath’s TileImporter?

An extra mention here as it was not clear in my previous post. The reason to prefer polygons here was that this allowed for overlaps! So perhaps that is also something to consider. Perhaps Segmentation Results as GeoJSON are indeed more flexible than masks?

1 Like

Oooh, being able to combine StarDist and CellPose results is something that I’ve been curious about… :slight_smile: