Interactive image alignment

Oh right, that was already done in a couple of places, but I can’t find them right now.
I guess you wanted something like this if you are copying over the whole measurement list:

i = 1
getCellObjects().each{
it.getMeasurementList().putMeasurement("Label", i)
i++
}

Each cell should have a Measurement Label, from 1 to N after running the script. Cells appear to be ordered by XY centroid coordinates.

If the measurements aren’t copied over, this likely won’t work, though, and I haven’t played with the transfer much. Maybe look into it later.

Thank you very much for your kind and prompt reply. I was very surprised to see the reply in just a few hours!
I apologize for this basic question.
I read your posting before and tried to find the ‘</>’ in my mac notebook word but could not find it so I just pasted the script to QuPath.
Would it work if I copy and special paste the script with HTML format in the word file and then paste it to QuPath? (I googled “converting to code from word document”…)

And thank you for the script for labeling cells in the measurement table. I will try it after trying the measurement copy over.

Thank you again.

Thank you very much for your kind and prompt reply. I was very surprised to see the reply in just a few hours!
I really appreciate your taking time to kindly reply in such a detail.

  1. Thank you for your kind explanation. I found the “hidden” annotation by moving the slider.
  2. I thought it would be nice if I could see the original annotation and move the newly copied ROIs to the location I want to overlap the ROIs with the interactive image alignment since I cannot do coding. But I could see only the newly copied ROIs on the interactive image alignment screen.

    This is annotated image.


This is image to compare.


And this is the overlay snapshot with opacity about the half way in the middle.
52|576x130

  1. Thank you. Now I see that 46

</> is here in the reply window… But unfortunately I have no knowledge how to convert the copied script to code using this… Could I please have a recommendation where I can learn this? I apologize for a basic question. I wish I am more familiar with coding…

  1. I totally agree with you and imagine that it must be a complicated task to make a cell ID and make sure it is unique when the images get distorted at least a little bit after a few rounds of stain and destain even on the same tissue. But I greatly appreciate that you have already added some features regarding my opinion. I will look for the next update.

Thank you again.

That icon is used in the same way as bold or italics, select the text, and click it (or CTRL+SHIFT+C on PC). It doesn’t convert to code, just formats it.

Also, now that I am taking the time to read it, I am guessing Pete meant something more like this script from a while back that allows you to input color vectors to generate new intensity mean values for the nucleus and cytoplasm. The new measurements are saved by channel names, so you may want to rename your Stains something like Hematoxylin2, Hematoxylin3, etc.
image
It was original intended for 3plex and higher brightfield imaging, but should work well enough for restaining tissue slices. Obviously sequential slices would have problems due to non-overlapping cells and the same cells frequently not existing even in two sequential slices, so cell by cell data would be of limited usefulness.

You do need to manually paste in the color vector line and the add Intensity features line from the Workflow tab to make the script work, as described by the comments in the script. The script mostly works in m3, but you will need to change closeList() to close() in two locations, around line 55 and line 100.
cellMeasurements.closeList()
=>
cellMeasurements.close()

No, got it first time - I was thinking of the add-an-ID-as-a-measurement trick :slight_smile:

Some of the features in QuPath (including Interactive image alignment) are very provisional and there to show what could be done… but it will take a lot more time/effort to build on them to make them more complete and user-friendly.

@Aimee by replacing the curly quotation marks with straight ones, could you get the script to work?

Thank you very much for the prompt reply.
I tried CTRL+SHIFT+C and it opened the part where I could see the source code so I copied that part and pasted it to QuPath script editor, input the transform part with interactive alignment data, input the annotated file name, checked for the changes to be made comparing it with the webpage and ran.
I got the following error message.
ERROR: Error: startup failed:
Script11.groovy: 25: unable to resolve class qupath.lib.roi.ROITools
@ line 25, column 1.
import qupath.lib.roi.ROITools
^

1 error

ERROR: Script error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:980)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:647)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:596)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:390)
at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:89)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:330)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:327)
at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:325)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:309)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:251)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.getScriptClass(GroovyScriptEngineImpl.java:331)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:153)
at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:766)
at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:696)
at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:676)
at qupath.lib.gui.scripting.DefaultScriptEditor$2.run(DefaultScriptEditor.java:1033)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Following is the code I entered in script editor.

// Get this from 'Interactive image alignment (experimental)
def matrix = [
  1.000, 0.000, -191.457,
  0.000, 1.000, 613.371
]

// SET ME! Define image containing the original objects (must be in the current project)
def otherImageName = '20190625_CD8.svs'

// SET ME! Delete existing objects
def deleteExisting = false

// SET ME! Change this if things end up in the wrong place
def createInverse = true


import qupath.lib.gui.helpers.DisplayHelpers
import qupath.lib.objects.PathCellObject
import qupath.lib.objects.PathDetectionObject
import qupath.lib.objects.PathObject
import qupath.lib.objects.PathObjects
import qupath.lib.objects.PathTileObject
import qupath.lib.roi.ROITools
import qupath.lib.roi.interfaces.ROI

import java.awt.geom.AffineTransform

import qupath.lib.gui.scripting.QPEx

if (otherImageName == null) {
    DisplayHelpers.showErrorNotification("Transform objects", "Please specify an image name in the script!")
    return
}

// Get the project &amp; the requested image name
def project = getProject()
def entry = project.getImageList().find {it.getImageName() == otherImageName}
if (entry == null) {
    print 'Could not find image with name ' + otherImageName
    return
}

def otherHierarchy = entry.readHierarchy()
def pathObjects = otherHierarchy.getRootObject().getChildObjects()

// Define the transformation matrix
def transform = new AffineTransform(
        matrix[0], matrix[3], matrix[1],
        matrix[4], matrix[2], matrix[5]
)
if (createInverse)
    transform = transform.createInverse()

if (deleteExisting)
    clearAllObjects()

def newObjects = []
for (pathObject in pathObjects) {
tempObject = transformObjects(pathObject, transform)
if (tempObject !=null) {
tempObject.setName(pathObject.getName())
newObjects<<tempObject
}}  
addObjects(newObjects)

print 'Done!'

PathObject transformObject(PathObject pathObject, AffineTransform transform) {
    // Create a new object with the converted ROI
    def roi = pathObject.getROI()
    def roi2 = transformROI(roi, transform)
    def newObject = null
    if (pathObject instanceof PathCellObject) {
        def nucleusROI = pathObject.getNucleusROI()
        if (nucleusROI == null)
            newObject = PathObjects.createCellObject(roi2, null, pathObject.getPathClass(), pathObject.getMeasurementList())
        else
            newObject = PathObjects.createCellObject(roi2, transformROI(nucleusROI, transform), pathObject.getPathClass(), pathObject.getMeasurementList())
    } else if (pathObject instanceof PathTileObject) {
        newObject = PathObjects.createTileObject(roi2, pathObject.getPathClass(), pathObject.getMeasurementList())
    } else if (pathObject instanceof PathDetectionObject) {
        newObject = PathObjects.createDetectionObject(roi2, pathObject.getPathClass(), pathObject.getMeasurementList())
    } else {
        newObject = PathObjects.createAnnotationObject(roi2, pathObject.getPathClass(), pathObject.getMeasurementList())
    }
    // Handle child objects
    if (pathObject.hasChildren()) {
        newObject.addPathObjects(pathObject.getChildObjects().collect({transformObject(it, transform)}))
    }
    return newObject
}

ROI transformROI(ROI roi, AffineTransform transform) {
    def shape = ROITools.getShape(roi) 
    shape2 = transform.createTransformedShape(shape)
    return ROITools.getShapeROI(shape2, roi.getImagePlane(), 0.5)
    }


I am sorry to bother you so much.

Thank you again for your great help.

Are you using QuPath v0.2.0-m3 or an earlier version? There was a small change to the script made to make it compatible with v0.2.0-m3.


Is what should have been copied. It looks like the import lines were manually edited.

import qupath.lib.roi.ROITools

should be
import qupath.lib.roi.RoiTools
There seem to be a few other changes made as well, so what was pasted here was definitely not the 0.2.0m3 version. I didn’t dig through all of the problems.

It works!!!
Thank you so much for your great help, all the people here!
It’s amazing how you help a beginner with full of mistakes like me with such kindness and generousness. I really appreciate your great help, Dr. @petebankhead and Dr. @Research_Associate.
Yes… I am so sorry I thought the lines in light colors were just some notes and deleted some of them when copying to the script editor and edited the part with “ROITools” because the script I copied looked like the old version with “PathROIToolsAwt”. (I thought the ones like " /** * Script to transfer QuPath… " were just notes for the users)
I downloaded the QuPath 0.2.0-m3 again today and ran the script and voilà !!!
It can copy over all the annotations and cell detections in the original slide. So exciting to learn!

Since the new slide shows shrinkage or distorsion in some part and the original annotation does not exactly match the cells in the new slide because I copied multiple ROIs at once. But I think I may be able to partly merge that ROI marking and the cell detection in it at the misaligned part and adjust a little bit to align that part.

I will try the next step to get the color and optical density infomation for those cell dections in the new slide and also try numbering the cells while acquiring the cell detection copy. Could you give me any suggestion where in the above script I might want to put the cell numbering script in? or should I run it in separate workflow?

Thank you again and again.

I think you will want three different scripts. From the sounds of it, you want the numbering script run in the first image, then the transformation/transfer (this thread), then the adding new measurements based on color vectors script in the new image.

Thank you so much for your kind reply. I will try it that way.

Is it possible to set the transformation matrix for an image manually? This is not to transfer annotations but for overlaying two images. I’m comfortable with groovy here but I didn’t see anything. I see I can drag the image for translation, but when I change the matrix scaling nothing happens. I know the exact transformation matrix from outside QuPath that I would like to apply… Thanks!

I don’t think it does that at the moment, since I think the primary intent was for analysis and you can’t pick up pixel data from the ghost image. The matrix is the output, the visual alignment is just for convenience and to see how accurate the estimated transform is. Or in other words, the text box is just an output for you to copy and paste from, not an entry.

1 Like

I see, do you know if there is a scripting mechanism to set these values?

There wouldn’t be at this point since that is working in the opposite direction of the intended information flow. You could always try making a feature request on the Github issues page, but that would be a different sort of function entirely. Maybe shift plus mousewheel to allow you to zoom the background image?

This skips the interactive GUI and goes straight to the overlay that’s actually being used:

import qupath.lib.gui.align.ImageServerOverlay
import javafx.scene.transform.Affine

// See https://openjfx.io/javadoc/11/javafx.graphics/javafx/scene/transform/Affine.html
def affine = new Affine(1, 0, 10000, 0, 1, 5000)

// Get viewer & server - you'll probably want to get a server from somewhere else...
def viewer = getCurrentViewer()
def server = getCurrentServer()

// Create an overlay
def overlay = new ImageServerOverlay(viewer, server, affine)

// You can use overlay.getAffine() and make adjustments directly to the Affine object if needed

// Use set rather than add because we might need to run it a few times to get it right...
viewer.getCustomOverlayLayers().setAll(overlay)

I haven’t really looked at the code for this in a long time - it could change a bit when I finally return to it.

2 Likes

Is ImageServerOverlay used only for the alignment of the two images in the current alignment tool? Or is it used elsewhere as well? Wondering how badly I could abuse this for visual functionality :slight_smile: Is there a limit to how many overlays we could load?

It’s intended to be more generally useful. You don’t need to specify an Affine transform if you just want to add another overlay on top.

There are a few different overlay options. If you want to overlay a ‘small’ image over all or part of the image, having it dynamically resized as required, then a BufferedImageOverlay is a better choice.

This could (for example) be used to show a low-resolution heatmap generated by a Python script in the context of the whole slide image. Here’s a fairly pointless demo:

import qupath.lib.gui.viewer.overlays.BufferedImageOverlay
import qupath.lib.regions.RegionRequest
import qupath.imagej.tools.IJTools
import javafx.scene.transform.Affine

// Get viewer & server - you'll probably want to get a server from somewhere else...
def viewer = getCurrentViewer()
def server = viewer.getServer()

// Extract a low-resolution ImagePlus for the current ROI (bounding box) & invert it
def roi = getSelectedROI()
def downsample = 8.0
def request = RegionRequest.createInstance(server.getPath(), downsample, roi)
def imp = IJTools.convertToImagePlus(server, request).getImage()
ij.IJ.run(imp, "Invert", "")
def img = imp.getBufferedImage()

// Show the resulting image on top
def overlay = new BufferedImageOverlay(viewer, request, img)

// Use set rather than add because we might need to run it a few times to get it right...
viewer.getCustomOverlayLayers().setAll(overlay)

It’s all a work in progress though and subject to change… don’t think there’s any inherent limit to the number of overlays, but can’t promise it all works very smoothly if you try adding a lot.

2 Likes

That looks suuuper useful for a few people who wanted to do some things with images I didn’t think were possible in QuPath. Was doing some terrible TMA downsampling, extraction of DAB “channels” and creating quasifluorescent overlay images in FIJI.

1 Like

This works for me, thanks! However, the Brightness and contrast settings seem to be set to the viewer image and sometimes these aren’t appropriate for the server image. For instance, one of the fluorescence images becomes so faint it’s barely visible when overlaid against a brightfield image but I can go the other way quite well. Might there by a way to load the B&C dialog for the overlaid image and adjust after overlay or at least keep the settings as they are when looking at the images individually?

Again, thanks a lot. I’ll keep exploring on my own too