Non-rigid ROI transformation

fiji
imagej
imglib2-roi
shaperoi

#1

Hello everybody,

Is there currently a way to do a non-rigid transformation of a ROI which would meet the following requirements (in 2D) ?

  • Float or Double precision (so Sub-pixel accuracy)
  • ‘Holes’ handling (for instance a polygon contour with an excluded polygon inside the first one)
  • Being able to keep the same number of control point before and after deformation. So let’s say if I have an external polygon made of 10 points and an excluded polygon defined by 5 points inside, I want to keep 15 control points before and after the transformation. For instance, solutions that would go through a rasterization step wouldn’t work for me. The underlying reason is that I want the user to be able to adjust as easily the transformed ROI as the non transformed one.
  • Compatible with IJ1 (Shape) ROIs (still happy to hear about pure imglib2 options!)

Here’s an example of a rigid transformation in IJ1:

run("Blobs (25K)");
setAutoThreshold("Default");
setOption("BlackBackground", false);
run("Convert to Mask");
run("Invert");
makeRectangle(72, 32, 85, 81);
run("Crop");
//setTool("zoom");
run("Create Selection");
run("Scale... ", "x=0.5 y=0.5 centered");
run("Rotate...", "rotate angle=15");
run("Blue");
roiManager("Add");

And I don’t even know if it is possible to edit points of the transformed ShapeRoi.

Happy to hear about any hints or direction where to go !


#2

I was playing around with RealMasks and GeomMasks (from imglib2-roi), but got stuck at the point were the masks get converted back to IJ1 ROIs:

#@ ImagePlus imp
#@ ConvertService convertService

import ij.gui.Roi
import ij.gui.PolygonRoi
import ij.gui.ShapeRoi
import net.imglib2.roi.RealMask
import net.imglib2.roi.geom.GeomMasks

// Create IJ1 composite ROI (polygon minus polygon)
polygonRoi1 = new PolygonRoi([10, 40, 40, 10] as float[], [10, 10, 40, 40] as float[], Roi.POLYGON)
polygonRoi2 = new PolygonRoi([30, 35, 30, 25] as float[], [15, 20, 25, 20] as float[], Roi.POLYGON)
compositeRoi = new ShapeRoi(polygonRoi1).xor(new ShapeRoi(polygonRoi2))

// Convert IJ1 to IJ2

convertedRoi = convertService.convert(compositeRoi, RealMask.class)
convertedPolygon1 = convertService.convert(polygonRoi1, RealMask.class)
convertedPolygon2 = convertService.convert(polygonRoi2, RealMask.class)

// Create IJ2 composite ROI
polygonMask1 = GeomMasks.polygon2D([10, 40, 40, 10] as double[], [10, 10, 40, 40] as double[])
polygonMask2 = GeomMasks.polygon2D([30, 35, 30, 25] as double[], [15, 20, 25, 20] as double[])
compositeMask = polygonMask1.xor(polygonMask2)

// Define Affine Transform
import net.imglib2.realtransform.AffineTransform2D
transform = new AffineTransform2D()
transform.set(-1, 0.5, 3, 0, 1, 4)

// Transform wrapped Rois
transformedConvertedRoi = convertedRoi.transform(transform)
transformedConvertedPolygon1 = convertedPolygon1.transform(transform)
transformedConvertedPolygon2 = convertedPolygon2.transform(transform)

// Transform IJ2 Masks
transformedMask1 = polygonMask1.transform(transform)
transformedMask2 = polygonMask2.transform(transform)
transformedCompositeMask = compositeMask.transform(transform)

// Convert IJ2 to IJ1
newRoi = convertService.convert(transformedCompositeMask, Roi.class) // try different of the above masks here

println newRoi

imp.setRoi(newRoi)

The problem is that after transformation, you end up with a net.imglib2.roi.composite.RealTransformUnaryCompositeRealMaskRealInterval (!) that will always get converted to an ImageRoi because a more specific converter doesn’t exist for this class.

The Masks are transformed without losing any information about their source polygons, but that also means that the coordinates are transformed “on-the-fly” when you work with these objects, and only get lost at that last step of converting to an image ROI. I don’t know if a more specific converter can be implemented that evaluates all the coordinates of the transformed components, and then creates an accurate ShapeRoi containing them.

Maybe @awalter17, @tpietzsch or @ctrueden can provide more insight here.


#3

Hello!

Sorry for the delay! I unfortunately haven’t had any time to try and create a converter, but I’ll post my initial thoughts here.

So it may be possible to create a converter from RealTransformUnaryCompositeRealMaskRealInterval to ShapeRoi in special cases. The special case being the operand/source ROI (or all the “leaf” ROIs in case of composites) is a shape which is defined by its “edge points” (i.e. polygons, rectangles, etc. but not circles, ellipses, etc.). There may also be a second condition that requires the RealTransform to be invertible, but I’m not 100% certain on that.

Assuming those conditions are met, a converter (or potentially multiple converters) could be written which essential applies the (inverted?) transform to each of the MaskPredicates edge points. And then uses those “edge points” to create an ImageJ 1.x ROI. This conversion would also be lossy, in the sense that you’d probably lose the information about the ImageJ2 ROI (i.e. no wrapping).

This becomes more complicated when the transformed ROI is not the base ROI. And even more complicated when there are multiple transformations in a single composite ROI. I’d need to think through the idea of multiple transformations a bit more …

A second option would also be some sort of “burn in” operation for transformed ROIs. So in the event that the “leaf” ROIs are also “writable”, then we could just mutate those ROIs by setting their "edge point"s to the transformed points. In the case of composite ROIs, updating the leaves would update the composite. Again this would become a bit of a problem when you have multiple transformations. But if you could just “burn in” the transform, then the current imagej-legacy roi converters should be sufficient.

Those are my initial thoughts. Please let me know your thoughts on these potential solutions. Also feel free to let me know if anything is unclear :smiley_cat:


#4

Hey @NicoKiaru,

clij offers two ways for transforming images in a non-rigid way on the GPU.

a) The affine transform
Use the affine transform to apply rotation/scaling/translation in 2D or 3D. I wrote some example code for transforming a binary image just as in the macro you posted, but doing the operation on the GPU:

run("Blobs (25K)");
setAutoThreshold("Default");
setOption("BlackBackground", false);
run("Convert to Mask");
run("Invert");
makeRectangle(72, 32, 85, 81);
run("Crop");

// define which GPU to use
cl_device = "";

// define image names on GPU
sourceImage = getTitle();
targetImage = "target";

// init GPU
run("CLIJ Macro Extensions", "cl_device=" + cl_device);
Ext.CLIJ_clear();

// send binary image to GPU
Ext.CLIJ_push(sourceImage);

// apply affine transform
Ext.CLIJ_affineTransform(sourceImage, targetImage, "center scale=2 rotate=15 -center");

// get image back from GPU
Ext.CLIJ_pull(targetImage);

// create selection on result image and show it on original image
run("Create Selection");

roiManager("Add");
close();

run("Blue");
selectWindow(sourceImage);
run("Restore Selection");

While writing the example, I realised that the scaling is misconfigured (factor 2 instead of 0.5). This is a bug which will be fixed asap. You find the documentation for the method here:

b) apply a vector field to a binary image
If you have two images describing the local shift (one image where each pixel value enumerates the displacement in X and another image for Y), you can use the method applyVectorField2D().

It is pretty new in clij but will very likely play a more important role in upcoming releases. Thus, feel free to play with it. Here you find some example code:

If you wish, I can also point you to Java code examples doing similar operations. Let me know if you plan to use clij and need any support. I’m happy to help and eager for feedback :wink:

Cheers,
Robert


#5

Very cool, thanks a lot to all of you @imagejan @haesleinhuepf @awalter17 . I did something which works for my own case, but it’s a bit dirty. Writing a proper converter would be nice. Still, for the sake of making a converter, maybe to know what problems I encountered could be useful:

  • Internally to ShapeRoi, an area polygon shape is casted to a java.awt.Polygon object, which is only taking int coordinates (here and here for instance). That’s why sometimes, for IJ1 operator, the ROIs lose their subpixel precision (try an OR command in the ROIManager and you’ll notice it).
  • Inside ShapeRoi this parse method is casting to Integercoordinates and is used several times.

On the bright side, there’s a ShapeRoi constructor which directly take a Shape object.

What I’ve done so far is to make a CompositeFloatPoly class which is solving these issues by spliting any ShapeRoi into a list of polygons. Then I have a list of points that can be modified as wanted. The ShapeRoi is then recomposed when needed. Its working but there’s still something regarding orientation conventions. I assumed a convention where clockwise polygons are defining ‘positive’ area while counterclockwise are ‘negative’ (before recombination). It’s working in most cases but not always… and I have a fix for these.

Depending on how much work it is, I can try to help making a proper converter.