Image Registration Suite

imglib2
imagej
affine-transform
registration
phase-correlation

#1

Hi All,

We are in the process of converting the Correct3D_Drift script to a proper Java plugin.

While doing so, in the new plugin we would like to give the user the option to chose via a drop-down menu between different image registration methods, such as

  • Translation_PhaseCorrelation (basically the Correct3D_Drift)
  • Translation_Elastix
  • Affine_Elastix
  • LocalDeformable_Elastix
  • a.s.o.

The user would have the choice to select a 5D reference region against which to register.

Also registrations could be saved, loaded and reapplied to other images.

Technically, we thought of just having a TransformationInterface and then above choices would be implementations of this interface; on top of this would be a SequenceRegistration class that would implement for instance logic to register a time lapse sequence.

We were wondering if that would be useful for the community and/or what would be additional/other suggestions regarding this.

Looking forward to reading your opinions!
Christian and Aliaksandr


#2

@Christian_Tischer,

This is awesome!

Please consider having your classes implement the imglib2 interfaces RealTransform or InvertibleRealTransform as applicable. This would make it much easier for some developers (cough, me) to take advantage of what you do…and you might find other stuff in that library useful as you work :smile:

btw, If you’re not already aware of it, the fiji-cmtk-gui will probably be of interest to you.

Looking forward to hearing more as you progress,
John


#3

As a Correct3D_Drift user I have to say that this sounds great!
I would thoroughly enjoy having the possibility to choose between methods and to use Elastix based methods.

The ability to save and apply would add lots of power for me.
To expand on that, I could use an option of only applying xy and not z, and one for only z but not xy. I posted something similar to that here: http://forum.image.sc/t/correct-only-z-drift-with-correct-3d

I could also use a registration tool that allows me to apply the same xy transform to all slices in a stack of different dimensions than the one in which the registration was originally calculated on.
An application example is: I ask the plugin to calculate the registration for the xy movement of my sample on a 2D maximum intensity projection of only one of the channels. Then I ask the plugin to apply the xy shifts to my original, multi-channel, 3D stack.
Another application example: Use the plugin to calculate the shift between channels based on images of fluorescent beads. Then apply the calculated shifts to a multi-channel xyzt stack.

Looking forward to using more of your progress,

Rafael


#4

I had a look at it and it seems that these interfaces are meant to work on single coordinate vectors; is that right? So far, we were rather thinking of an interface like this:

public void applyTransform( Image input, Image output );

…giving an image as an input and getting the whole transformed image as an output. But the approach you mention looks very interesting! Could you point us to example code where your suggestion is implemented and used to actually transform a whole image and display it? I have to admit here that we are not at all skilled in imglib2, but maybe this is the right moment to get into it!


#5

We were entertaining the idea that the user could specify something like a registration axis; such that one could say whether one wants to register, e.g.

  • 2D-xy slices along z
  • 3D-xyz cubes along t
  • 3D-xyz cubes along c, i.e. align two channels with respect to each other.
  • a.s.o.

Does that sound like what you would like to have?


#6

You are correct - that’s the “most low-level” interface. The code that transforms an image given a RealTransform is called RealTransformRandomAccessible, with convenience methods in RealViews.

Code would look like:

Img myImage = ...
RealTransform myTransform = ...
Img myTransformedImage = RealViews.transform( Views.interpolate( myImage, myIterpolator), myTransform);

where there’s a necessary interpolation step. (Later, I’ll put together some specific code that is actually runnable and post back)

There is not (as far as I know) an image-to-image interface as you describe, but anything that can transform images should also be able to transform points, and anything that can transform points can also transform images.

John


#7

I have to admit this makes lots of sense!
I feel this might be the moment for me to finally embrace imglib2 :slight_smile:


#8

:star_struck::clap::clap::fireworks:

I’ll be suuuuper happy to help if you run into roadblocks - I’d love an elastix wrapper/interface :smile:


#9

Yes. It sounds very nice.


#10

Thank you!

Are there already also interfaces, not for applying, but for finding the best transformation?

If not, would something like this make sense?

public RealTransform findTransform( RandomAccessibleInterval fixed, RandomAccessibleInterval moving )

As I said, I have hardly any clue about imglib2, but my idea would be that the RAI could be good because this could be a way to specify on which parts (intervals) of the two images the user wants to find the best transformation.


#11

There is not such an interface in imglib2.

There is something like that in mpicbg: the Model interface, but it doesn’t operate on images, but rather on point correspondences, so not the same thing.

Yup, something like that looks good to me.

John


#12

Here’s a beanshell script that builds a simple RealTransform (2d affine) and transforms and shows an image
using that transform.

It makes heavy use of convenience functions in the RealViews and Views classes.

import ij.IJ;
import net.imglib2.img.display.imagej.ImageJFunctions;
import net.imglib2.interpolation.randomaccess.NLinearInterpolatorFactory;
import net.imglib2.realtransform.*;
import net.imglib2.view.*;

img = ImageJFunctions.wrap( IJ.getImage());

AffineTransform2D xfm = new AffineTransform2D();
// rotate by some amount 
// note that the rotation is about the origin (0,0) in the top left of the image
xfm.rotate( Math.PI / 8 ); 
xfm.scale( 0.8 );  // scale by some amout

// real random accessible:
// extend: what are the values outside the image
// interpolate
// transform with the affine above
rra = RealViews.affine( 
    Views.interpolate( Views.extendZero( img ), new NLinearInterpolatorFactory() ),
	xfm );

// random accessible make it a raster image ,
// the same size as the original interval( _, interval)
ra = Views.interval( Views.raster( rra ), img );
ImageJFunctions.show( ra );

#13

Great! Works for me.
It looks like there still is this “bug” that the interpolator has trouble dealing with 255 in 8bit images:

        AffineTransform2D xfm = new AffineTransform2D();
		xfm.rotate(0.0 );
		xfm.translate(100.5, 100.0 );

		ExtendedRandomAccessibleInterval erai = Views.extendZero( img );
		RealRandomAccessible rra = Views.interpolate( erai, new NLinearInterpolatorFactory()  );
		AffineRandomAccessible ara = RealViews.affine( rra, xfm );

		RandomAccessibleInterval rai = Views.interval( Views.raster( ara ), img );
		ImageJFunctions.show( rai );


#14

We have another question. In case we measure the drift sequentially, i.e. 1->2, 2->3, 3->4, a.s.o.; in order to apply the drift correction for time-point 4 we need to apply all the drifts to up this time-point (I hope this is understandable). I found below class, which does that, but I was a bit worried that this is maybe slow because, as I understand it, it has to for each pixel compute the combined transformation over and over again, isn’t it? If that’s true, is there some way to actually really “get” the combined transformation rather than recomputing it on the fly for every pixel?


#15

I’ll look into that interpolation issue and report back.

You are correct that the RealTransformSequence will do what you describe. Whether you can “get” the whole transform depends on what it is. For Affine’s (and linear transforms) it’s easy: they all have concatenate and preConcatenate methods.

For (most, that I’m aware of) non-linear transforms, there’s not a straightforward way of concatenating one transform to another. It is possible to compose displacement fields, but to do it, you’d end up implementing RealTransformSequence, effectively. And, I don’t think there would be a speed improvement if you only visit each location once (unless I’m overlooking something).

On the plus side for speed: rendering transformed images is not hard to parallelize across space e.g. an undocumented example from BigWarp

@tpietzsch renders in parallel over several resolutions for bigdataviewer.

John


#16

Yea, that nasty behaviour is due to overflows.
Its fixed by this ClampingNLinearInterpolator, but it lives in the bigdataviewer-core at the moment and hasn’t made it into imglib2-core yet.

(Edit: since Fiji has the bigdataviewer built in, you can just replace NLinearInterpolatorFactory with ClampingNLinearInterpolatorFactory in the script above and it’ll just work :slight_smile:)


#17

I think this project is a great idea! I use Correct 3D drift a lot and all of the things mentioned in this thread would be really useful for me.


#18

Hi All, @stelfrich @imagejan

During a very productive hackathon in Dresden I managed to create version 0.1.2 of the Image Registration Suite :slight_smile:

It obviously is super beta, but maybe you can give it a try and give me feedback, both regarding usability and code.

  • [Fiji > Update > Manage Update Sites > EMBL-CBA
  • Restart Fiji
  • [Fiji > Plugins > Registration > N-D Sequence Registration]

Feel free to post issues here:


#19

@Christian_Tischer, sorry for being a noob, but it throws up an error (below) when trying to compute registration. What have I not got?

[INFO] ImageRegistrationPlugin started
[INFO] ## TransformationFinderRotationTranslationPhaseCorrelation
[INFO] ### TransformationFinderTranslationPhaseCorrelation
org.scijava.module.MethodCallException: Error executing method: de.embl.cba.registration.gui.ImageRegistrationPlugin#computeRegistration
	at org.scijava.module.MethodRef.execute(MethodRef.java:73)
	at org.scijava.module.AbstractModuleItem.callback(AbstractModuleItem.java:229)
	at org.scijava.widget.DefaultWidgetModel.callback(DefaultWidgetModel.java:187)
	at org.scijava.ui.swing.widget.SwingButtonWidget$1.actionPerformed(SwingButtonWidget.java:83)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
	at java.awt.Component.processMouseEvent(Component.java:6535)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
	at java.awt.Component.processEvent(Component.java:6300)
	at java.awt.Container.processEvent(Container.java:2236)
	at java.awt.Component.dispatchEventImpl(Component.java:4891)
	at java.awt.Container.dispatchEventImpl(Container.java:2294)
	at java.awt.Component.dispatchEvent(Component.java:4713)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
	at java.awt.Container.dispatchEventImpl(Container.java:2280)
	at java.awt.Window.dispatchEventImpl(Window.java:2750)
	at java.awt.Component.dispatchEvent(Component.java:4713)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.awt.EventQueue$4.run(EventQueue.java:729)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.scijava.module.MethodRef.execute(MethodRef.java:69)
	... 39 more
Caused by: java.lang.NoClassDefFoundError: net/imglib2/algorithm/phasecorrelation/PhaseCorrelation2
	at de.embl.cba.registration.transformationfinders.TransformationFinderTranslationPhaseCorrelation.findTransform(TransformationFinderTranslationPhaseCorrelation.java:109)
	at de.embl.cba.registration.transformationfinders.TransformationFinderRotationTranslationPhaseCorrelation.computeCrossCorrelationAndTranslation(TransformationFinderRotationTranslationPhaseCorrelation.java:177)
	at de.embl.cba.registration.transformationfinders.TransformationFinderRotationTranslationPhaseCorrelation.computeCrossCorrelationAndTranslation(TransformationFinderRotationTranslationPhaseCorrelation.java:140)
	at de.embl.cba.registration.transformationfinders.TransformationFinderRotationTranslationPhaseCorrelation.findTransform(TransformationFinderRotationTranslationPhaseCorrelation.java:77)
	at de.embl.cba.registration.ImageRegistration.run(ImageRegistration.java:231)
	at de.embl.cba.registration.gui.ImageRegistrationPlugin.registrationThread(ImageRegistrationPlugin.java:247)
	at de.embl.cba.registration.gui.ImageRegistrationPlugin.access$000(ImageRegistrationPlugin.java:48)
	at de.embl.cba.registration.gui.ImageRegistrationPlugin$1.run(ImageRegistrationPlugin.java:281)
	at java.lang.Thread.run(Thread.java:745)
	at de.embl.cba.registration.gui.ImageRegistrationPlugin.computeRegistration(ImageRegistrationPlugin.java:285)
	... 44 more
Caused by: java.lang.ClassNotFoundException: net.imglib2.algorithm.phasecorrelation.PhaseCorrelation2
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 54 more

#20

I don’t have that class in my up-to-date Fiji either.
A quick search on maven.imagej.net told me that it is contained in the net.preibisch::BigStitcher artifact.

@Christian_Tischer, your plugin apparently requires having the BigStitcher update site enabled as well, right?!