Overlaying a tile-probability heatmap onto a WSI in QuPath

Sample image and/or code

I have a whole slide image (WSI) that I have exported tiles from and produced a probability from each of them using an external deep learning model. It would be great to visualise the probabilities using a heatmap, something like the following, perhaps:

heatmap-tiles

Here, the region shown was split into three overlapping tiles, and I have coloured the central part of each tile with a colour on a red-to-green scale. (Only colouring the central parts to avoid having to figure out what to do with overlapping tiles!) For whatever reason, there was no tile in the bottom right corner, so that stays untouched.

One way I could do this would be to export the whole image, either as a single PNG or as tiles and then overlay a heatmap on top of it using something like OpenCV or Scikit-Image. That has the big downside of losing all of the benefits of working in QuPath (such as being able to zoom in to the image and pan around it), as well as having to create a potentially-large PNG file.

Another way might be to create the overlay as a separate image and then get QuPath to do the overlaying, such as in Pete’s gist. But if I understand this correctly, I would need to create a PNG or TIFF image with the same size as the original WSI, which is huge!

A third way might be to create each of the square patches as an annotation, and to have a handful of classes corresponding to different probability ranges. That could then be viewed quite easily, though it would interfere with existing annotations.

Does anyone have any better suggestions? I surely can’t be the first person who’s wanted to integrate externally-generated heatmaps with existing images!

Best wishes,

Julian

I would recommend detection tiles as they are generally less problematic and slow down QuPath less than a lot of annotations. They can have their own measurements and can be created from a CSV file with a list of coordinates and values (think 3 column CSV file? X, Y, color). That way they could be reloaded if any QuPath action deletes them.

With more coding, you could use that CSV as a image file - essentially it would be a downsampled image where each “pixel” was quite large. It could be visualized a number of ways.

And, now that I brought that up, your idea of creating a PNG file could work - it would be a very small PNG file since each pixel would represent a tile. Then the PNG image would be scaled up so that the pixel size matched the tile size.

Hi @juliang the gist is written for exactly that purpose - no need to save the PNG the same size as the whole slide image, it will be resized according to the dimensions of the ImageRegion.

1 Like

See also Best way to display additional information as overlay? - #6 by poehlmann (there may be a small change in v0.3 to bind the BufferedImageOverlay display to the same shortcut as the pixel classification overlay so they can be toggled on/off in the same way)

1 Like

Amazing @Research_Associate and @petebankhead, thanks! I will have a go with these over the next few days and see what I can make happen!

1 Like

So I’ve taken your advice @Research_Associate and @petebankhead and I’ve written this script to turn a CSV file of results into detection objects:

It would be lovely if the filled detection objects could be set to be partially transparent, but this will do fine for now!

Comments on my code are, of course, welcome!

1 Like

Looks good! Some comments/suggestions:

  • You could avoid creating your own measurement list, and just get the one created by default within the detection object (I forget if there’s a method signature that doesn’t require a measurement list; if there isn’t, you should be able to just pass null). The outcome is the same, but perhaps a little less code.
  • You should close the measurement list when you’re done, although in practice this probably have much impact unless you have an extremely large number of objects
  • You might want to set the pathClass differently depending upon whether the probability is > 0.5 or thereabouts, so that the default visualization is meaningful even without the measurement map display
  • I don’t think there’s a way to control the opacity of the detection (as far as I remember the color assumes RGB rather than ARGB), but you can set the opacity slider at the top of the main QuPath window… or programmatically control it with getCurrentViewer().getOverlayOptions().setOpacity(0.5). You can also use Ctrl/Cmd + the scroll wheel to adjust opacity.
1 Like

Only thought I had is reiterating Pete’s comment here:

About adding each object one at a time is slow vs in bulk. I used to do this the slow way too, and did not realize it was responsible for quite a few of my scripts running much more slowly than they could have.

I think that would look like:

        detection = PathObjects.createDetectionObject(roi, pathclass, prob)
        detection.getMeasurementList().putMeasurement(valuename, value)

without the measurementlistfactory line.

Thanks @petebankhead! Super helpful, and I’ve updated the gist to take these into account (and also to use error dialog boxes instead of throwing exceptions):

  • I’m now not creating a measurement list but using the default one. I’m not sure what it means to “close” a measurement list, though.
  • Interesting idea on setting the pathClass differently, thanks: I’ve implemented that now.
  • Ah, the opacity slider works brilliantly!

And @Research_Associate - using a list and addObjects is fabulous!

1 Like

Just calling detection.getMeasurementList().close() should do it.

When you close a list QuPath can trim any arrays to be the minimum size needed to store the measurements, and if it finds lots of objects have measurements with the exact same names in the exact same order then it will just store that list of names once – and (if I remember correctly) store the index of each measurement name in a map so that it can be found quickly on demand.

Conceivably, this can save quite a bit of space and improve performance if you have a large number of detections, all with a lot of measurements generated at the same time. In practice… it probably won’t make any noticeable difference here.

Ah, OK; added that to the gist! Many thanks as always :slight_smile:

1 Like

And I’ve now realised that it is (relatively) straightforward to handle multiple values per tile, so I’ve modified the gist to do this.

1 Like