QuPath arbitrarily transform detections and annotations

Hello Qupath experts,

Let’s say I have a function which transforms coordinates in space into another coordinate space ( a freeform transform, like something that implements RealTransform. Assuming the transformation behaves correctly - continuous and invertible - is there a way to apply this transform to an annotation or a detection ? If not, do you have any other suggestions in order to achieve this ?

For the context, I’m trying to reuse BigWarp registration results into QuPath.

Thanks!

Nicolas

3 Likes

To avoid dealing with the specifics of ROIs, you have two main options in QuPath: use Java Topology Suite Geometry objects (which can return Coordinate arrays) or Java AWT shape options (which have PathIterators).

You can get either from a QuPath ROI simply by calling ROI.getShape() or ROI.getGeometry().

In general, QuPath uses Shape objects for painting (since it’s needed for Graphics2D rendering) and Geometry objects for analysis/transformation (since there are lots of useful operations found within JTS, and it’s a pretty well-established standard). But converting between shapes and (valid) geometries isn’t easy or fast, so QuPath’s own ROI class handles all that by providing a representation that can easily be converted to either of the others.

The key methods for getting things back are found inside GeometryTools.

So I’d recommend applying your RealTransform to a Geometry. If that works well enough, then it will be useful beyond QuPath. But if you need to be able to support any arbitrary transform, you do need to be very careful about ensuring geometry validity, which can be a huge headache (see here and here). You can test for it using Geometry.isValid(), but for complex geometries this can be extremely slow… so if you can guarantee validity some other way, that’s probably better.

3 Likes

If you get this to work, please post how! I think there’s a bunch of people on this forum who would be interested.

2 Likes

Hi @petebankhead and thanks for the very detailed answer,

I managed to get a RealTransform object from BigWarp into a QuPath plugin within intellij. Now I’m stuck with something very simple : applying this transform to a rectangle (without even thinking about geometry validity for the moment:

       RealTransform rt = realTransformFromBigWarpFile(bwLandmarkFile, false);

        // Let's make a simple rectangle
        Geometry rectangle = GeometryTools.createRectangle(0,0,100,100);
        System.out.println("Initial rectangle : "+rectangle.toString());

        // Apply the transformation
        rectangle.apply((CoordinateFilter) coord -> {
            RealPoint pt = new RealPoint(2);
            pt.setPosition(coord.getX(),0);
            pt.setPosition(coord.getY(),1);
            rt.apply(pt,pt);
            coord.setX(pt.getDoublePosition(0));
            coord.setY(pt.getDoublePosition(1));
        });

        // Triggers geometry changed, necessary as mentioned by the API
        rectangle.geometryChanged();

        System.out.println("Transformed rectangle : "+rectangle.toString());

I can see that the transformed coordinates are well computed, but setting the new coordinates has no effect (coord.setD).

Here’s the output:


Initial rectangle : POLYGON ((0 0, 4 0, 8 0, 12 0, 16 0, 20 0, 24 0, 28 0, 32 0, 36 0, 40 0, 44 0, 48 0, 52 0, 56 0, 60 0, 64 0, 68 0, 72 0, 76 0, 80 0, 84 0, 88 0, 92 0, 96 0, 100 0, 100 4, 100 8, 100 12, 100 16, 100 20, 100 24, 100 28, 100 32, 100 36, 100 40, 100 44, 100 48, 100 52, 100 56, 100 60, 100 64, 100 68, 100 72, 100 76, 100 80, 100 84, 100 88, 100 92, 100 96, 100 100, 96 100, 92 100, 88 100, 84 100, 80 100, 76 100, 72 100, 68 100, 64 100, 60 100, 56 100, 52 100, 48 100, 44 100, 40 100, 36 100, 32 100, 28 100, 24 100, 20 100, 16 100, 12 100, 8 100, 4 100, 0 100, 0 96, 0 92, 0 88, 0 84, 0 80, 0 76, 0 72, 0 68, 0 64, 0 60, 0 56, 0 52, 0 48, 0 44, 0 40, 0 36, 0 32, 0 28, 0 24, 0 20, 0 16, 0 12, 0 8, 0 4, 0 0))
Transformed rectangle : POLYGON ((0 0, 4 0, 8 0, 12 0, 16 0, 20 0, 24 0, 28 0, 32 0, 36 0, 40 0, 44 0, 48 0, 52 0, 56 0, 60 0, 64 0, 68 0, 72 0, 76 0, 80 0, 84 0, 88 0, 92 0, 96 0, 100 0, 100 4, 100 8, 100 12, 100 16, 100 20, 100 24, 100 28, 100 32, 100 36, 100 40, 100 44, 100 48, 100 52, 100 56, 100 60, 100 64, 100 68, 100 72, 100 76, 100 80, 100 84, 100 88, 100 92, 100 96, 100 100, 96 100, 92 100, 88 100, 84 100, 80 100, 76 100, 72 100, 68 100, 64 100, 60 100, 56 100, 52 100, 48 100, 44 100, 40 100, 36 100, 32 100, 28 100, 24 100, 20 100, 16 100, 12 100, 8 100, 4 100, 0 100, 0 96, 0 92, 0 88, 0 84, 0 80, 0 76, 0 72, 0 68, 0 64, 0 60, 0 56, 0 52, 0 48, 0 44, 0 40, 0 36, 0 32, 0 28, 0 24, 0 20, 0 16, 0 12, 0 8, 0 4, 0 0))

As mentioned in the javadoc for the apply(CoordinateFilter) method, it could be that I’m working on a copy of the coordinates. Is this pattern common in QuPath or do you think it’s specific to the way this rectangle is created ? What’s the best way to transform the underlying coordinates ? Or is it better to somehow clone the object to apply the coordinate transforms ?

I’m continuing to look through the JTS api, but if it’s something easy for you, I thought it was better for me to ask already.

Thanks!

1 Like

I’m afraid I don’t know off-hand, it’s deeper into the bowels of JTS than the bit I have stored in my memory. Although I see from the javadocs that for in-place mutation you should really use CoordinateSequenceFilter.

1 Like

Indeed, this works as expected:

File bwLandmarkFile = new File(directory+fileName);
        RealTransform rt = realTransformFromBigWarpFile(bwLandmarkFile, false);

        // Let's make a simple rectangle
        Geometry rectangle = GeometryTools.createRectangle(0,0,100,100);
        System.out.println("Initial rectangle : "+rectangle.toString());

        Geometry transformedRectangle = rectangle.copy();
        // Apply the transformation
        transformedRectangle.apply(
                new CoordinateSequenceFilter() {
                    @Override
                    public void filter(CoordinateSequence seq, int i) {
                        RealPoint pt = new RealPoint(2);
                        pt.setPosition(seq.getOrdinate(i, 0),0);
                        pt.setPosition(seq.getOrdinate(i, 1),1);
                        rt.apply(pt,pt);
                        seq.setOrdinate(i, 0, pt.getDoublePosition(0));
                        seq.setOrdinate(i, 1, pt.getDoublePosition(1));
                    }

                    @Override
                    public boolean isDone() {
                        return false;
                    }

                    @Override
                    public boolean isGeometryChanged() {
                        return true;
                    }
                });

        // Triggers geometry changed, necessary as mentioned by the API
        // rectangle.geometryChanged(); - no need it's specified in the filter
        System.out.println("Transformed rectangle : "+transformedRectangle.toString());
Initial rectangle : POLYGON ((0 0, 4 0, 8 0, 12 0, 16 0, 20 0, 24 0, 28 0, 32 0, 36 0, 40 0, 44 0, 48 0, 52 0, 56 0, 60 0, 64 0, 68 0, 72 0, 76 0, 80 0, 84 0, 88 0, 92 0, 96 0, 100 0, 100 4, 100 8, 100 12, 100 16, 100 20, 100 24, 100 28, 100 32, 100 36, 100 40, 100 44, 100 48, 100 52, 100 56, 100 60, 100 64, 100 68, 100 72, 100 76, 100 80, 100 84, 100 88, 100 92, 100 96, 100 100, 96 100, 92 100, 88 100, 84 100, 80 100, 76 100, 72 100, 68 100, 64 100, 60 100, 56 100, 52 100, 48 100, 44 100, 40 100, 36 100, 32 100, 28 100, 24 100, 20 100, 16 100, 12 100, 8 100, 4 100, 0 100, 0 96, 0 92, 0 88, 0 84, 0 80, 0 76, 0 72, 0 68, 0 64, 0 60, 0 56, 0 52, 0 48, 0 44, 0 40, 0 36, 0 32, 0 28, 0 24, 0 20, 0 16, 0 12, 0 8, 0 4, 0 0))
Transformed rectangle : POLYGON ((-1.567 -1.567, 2.424 2.424, 6.415 6.415, 10.406 10.406, 14.397 14.397, 18.388 18.388, 22.379 22.379, 26.37 26.37, 30.361 30.361, 34.352 34.352, 38.343 38.343, 42.334 42.334, 46.325 46.325, 50.316 50.316, 54.307 54.307, 58.298 58.298, 62.289 62.289, 66.28 66.28, 70.271 70.271, 74.262 74.262, 78.253 78.253, 82.244 82.244, 86.235 86.235, 90.226 90.226, 94.217 94.217, 98.208 98.208, 98.13 98.13, 98.052 98.052, 97.974 97.974, 97.896 97.896, 97.818 97.818, 97.74 97.74, 97.662 97.662, 97.584 97.584, 97.506 97.506, 97.428 97.428, 97.35 97.35, 97.272 97.272, 97.194 97.194, 97.116 97.116, 97.038 97.038, 96.961 96.961, 96.883 96.883, 96.805 96.805, 96.727 96.727, 96.649 96.649, 96.571 96.571, 96.493 96.493, 96.415 96.415, 96.337 96.337, 96.259 96.259, 92.268 92.268, 88.277 88.277, 84.286 84.286, 80.295 80.295, 76.304 76.304, 72.313 72.313, 68.322 68.322, 64.331 64.331, 60.34 60.34, 56.349 56.349, 52.358 52.358, 48.367 48.367, 44.376 44.376, 40.385 40.385, 36.394 36.394, 32.403 32.403, 28.412 28.412, 24.421 24.421, 20.43 20.43, 16.439 16.439, 12.448 12.448, 8.457 8.457, 4.466 4.466, 0.475 0.475, -3.516 -3.516, -3.438 -3.438, -3.36 -3.36, -3.282 -3.282, -3.204 -3.204, -3.126 -3.126, -3.048 -3.048, -2.97 -2.97, -2.892 -2.892, -2.814 -2.814, -2.736 -2.736, -2.658 -2.658, -2.58 -2.58, -2.502 -2.502, -2.424 -2.424, -2.346 -2.346, -2.268 -2.268, -2.191 -2.191, -2.113 -2.113, -2.035 -2.035, -1.957 -1.957, -1.879 -1.879, -1.801 -1.801, -1.723 -1.723, -1.645 -1.645, -1.567 -1.567))
4 Likes

Ok, so with the power of the println function, it looks like I managed to transform ‘in place’ the geometry of QuPath PathObject. But the associated Shape, thus the display, is not being updated. Is there an easy way to update the shape according to the new geometry or do I need to create a new PathObject with this new geometry ?

Oh, that doesn’t sound good… ROIs in QuPath are meant to be immutable. If that was easy to do, can you tell me how? QuPath shouldn’t allow it… at least unless you really really try to thwart the checks, e.g. via reflection.

The intended approach is that you’d create a new object (most of the time) or at least a new ROI and set the ROI for the object (currently only supported for annotations or TMA cores, intended for use when interactively modifying them through the viewer).

You can use GeometryTools.geometryToROI and PathObjectTools.transformObject as a starting point.

If you’re scripting, a call to fireHierarchyUpdate() can resolve many things – but perhaps not the shape caching.

2 Likes

In case it helps, @smcardle has an example of building a geometry into a QuPath object here.

Also, the official docs here: Custom scripts — QuPath 0.2.3 documentation

@petebankhead if he just changed “the geometry” in place, that has nothing to do with the associated QuPath ROI, right? So I think that is all correct.

1 Like

Indeed, no worry, the geometry has not been modified, I guess I’m working on a copy

            Geometry geometry = object.getROI().getGeometry();
            // Print coordinates of the geometry before
            geometry.apply((CoordinateFilter) c -> System.out.println("before:"+c.x+":"+c.y));
            // Transforms the geometry
            GeometryTools.attemptOperation(geometry, (g) -> {
                g.apply(transform);
                return g;
            });
            // Print coordinates of the geometry after the transform -> these are changed
            geometry.apply((CoordinateFilter) c -> System.out.println("after [geometry]:"+c.x+":"+c.y));
            
            // Print coordinates of the geometry after the transform, but the geometry is accessed through getRoi().getGeometry() -> no change
            object.getROI().getGeometry().apply((CoordinateFilter) c -> System.out.println("after [getRoi().getGeometry()]:"+c.x+":"+c.y));

Thanks again for the fast response, and sorry if I have missed some info on the official documentation.

Ah that’s good; yes you should only be able to get either a newly-converted Geometry or a copy.

1 Like

Ok, now everything works, but it’s painfully slow for detections…

Can you link me to the code? Or do you know which is the slow bit…?

Edit: I ask because if you’re firing lots of hierarchy events, it’ll almost certainly be extremely slow. Whereas recursively transforming objects and setting the parent/child relationships directly (assuming those fit with the kind of transform) should be pretty fast. See for example Script to transfer QuPath objects from one image to another, applying an AffineTransform to any ROIs · GitHub

1 Like

Of course, here’s the code:

It’s on the imglib2-include branch.

This class contains the methods to transform the bigwarp landmark file (a demo one is opened from the test resources) into a jts filter + the method to transform PathObject. And I call this simple script to apply the transform to annotation and detection in qupath.

Edit : hmm I didn’t think at all about the hierarchy for the moment. I think no object has a parent.
I didn’t look for the slow bit for the moment, but I’m already very happy that it works, so thanks again for your inputs! :wink:

No problem!

Lots of addObject calls will be slow. You can gather them into a list and call addObjects instead – that should dramatically reduce the number of events fired.

Something like this should do it (unchecked!)

def transformed_detections = all_detections.collect { detection ->
    def transformed_detection = TransformHelper.transformPathObject(detection, transformer, false)
}
addObjects(transformed_detections)

It won’t bother to resolve hierarchical relationships, but you can call resolveHierarchy() later if you need these. Or you can establish them during the transform for maximum efficiency if that becomes important (i.e. set parent/child relationships between objects explicitly).

2 Likes

Oh yes! That changes everything indeed ! It’s super fast now. I just need to make this a bit easier to use, but it’s not too far from a working BigWarp to QuPath bridge. At least, transfering all detections and annotations from one image to a registered one should not be too problematic.

4 Likes

Excellent! That sounds like it will be very useful :smiley:

1 Like

For the record, the way to store and open the transforms is pretty straightforward (json serialization with type adapters, the standard way of storing objects in QuPath, I believe), ‘light’ (few dependencies), and is also fully compatible with what exists in fiji.

(Almost) everything is in this class:

Supports : affine transform (2D and 3D), spline transforms (nD), sequence of tranforms. And if the transforms are invertible, computing the inverse just consists of calling transform.inverse().

Related: Registration of whole slide images at cellular resolution with Fiji and QuPath

4 Likes