Image registration in Python

What’s the best way to determine the best transformation to align two images in python?

Essentially, I have a binarized version of a scanned slide, as well as the binarized output of a deep learning classifier run on that slide. The classifier output is slightly translated and scaled (due to the nature of FCNs and UNets and the like). I would like to figure out how to automate the determination of the transformation to register the slides.

(I created the binarized slide image by applying a grayscale threshold filter of 0.75, and the binarized output image by mapping all background classes to 0 and all tissue classes to 1.)

Unfortunately, the binary images, while matching in shape, are nowhere close to being exact translations of each other - I can try to use opencv erode/dilate/blur to make them closer but I can never succeed exactly, as our models behave unpredictably on edges.

Here is an example set of images:

Slide:

Classifier

Superimposed (note the large gap in the bottom right edge):

I’ve looked into skimage's registration module, but it doesn’t seem like it can handle this. I’ve also looked at imreg_dft, but I was unable to get it to work (it was seemingly stuck for over an hour before I killed it).

How can I effectively determine the registration transformation between these two images?

Hi Arjun,

Do you have an example of the images before thresholding?

Perhaps the empty areas inside would be easier to align in this case.

Thanks for the response!

This slide is an HED-stained image and looks like

The classifier I am currently testing this on (proof of concept of sorts) is a very simple artifact classifier that only classifies patches as “tissue” or “background/artifact”, so the binarized image above is essentially the entire output of the classifier. However, my goal is to write something that’s applicable to future models, which will aim to differentiate the types of cells in the tissue.

If the issue is the large amounts of gaps in the slide, that can be remedied pretty easily by an OpenCV erode (iterations=3):

I’m more lost about the process of going from images to the registration transformation. I think I can figure out a good sequence of preprocessing transforms (opencv erode/dilate, inverting, etc) once I have some code that performs semi-working registration.

I’m hoping you or somebody else can point me to a library/module or even instructions for writing my own registration routine.

The problem is that you don’t have pixels that correlate in value, so you have to figure out something else to align. You can look at, e.g., the shape outline, or you can look at the gaps in between.

Based on what you write above, it looks like you want the shapes roughly aligned. If so, the closer you can get the shapes with preprocessing the better. Then, finding scale / translation to get the best overlap can be done using scipy.optimize and skimage.transform.SimilarityTransform by defining a cost function that takes into account overlap.

2 Likes

Hey @Arjun.Vikram,

the simple Itk framework comes with sophisticated methods specifically for image registration. Example here:
https://simpleitk.readthedocs.io/en/master/link_ImageRegistrationMethod3_docs.html

Let us know if this helps!

Best,
Robert

2 Likes