Registration of whole slide images at cellular resolution with Fiji and QuPath

Dear community,

Our team is happy to share a whole slide image registration workflow that can reach cellular resolution over a large area (an area that typically requires more than an affine transformation). We bridge QuPath, Fiji (in particular BigWarp) and elastix, in an attempt to merge these 3 worlds.

Briefly, a Project is defined in QuPath, and can be opened in Fiji to be registered either automatically, manually (BigWarp), or semi-automatically (BigWarp is used to correct the result of the automated registration). The resulting transformation is saved in a QuPath Project folder as a JSON transformation file, which QuPath can then retrieve and use to transfer and transform PathObjects (annotations and detections) from one image to another, in one direction or the other.

The full documentation of the workflow can be found here, as well as some video tutorials:

It’s still quite experimental and the UI lacks polishing, but things should work. If you find this useful, we’re happy to get your feedback and issues on the forum,

Cheers,

BIOP-IPAs (@oburri, @stoffelc, @arne, @romainGuiet)

ping @petebankhead @bogovicj @smcardle @Research_Associate @Kimberly_Meechan

26 Likes

Great work! Lots of quality effort on display.

1 Like

Thank you very much @NicoKiaru and team!
This looks very good and has already great documentation.

I just tried with some MRXS images when I start the registration I get the following error.


java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/ejml/data/ReshapeMatrix64F
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:307)
	at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
	at ij.IJ.runPlugIn(IJ.java)
	at ij.Executer.runCommand(Executer.java:151)
	at ij.Executer.run(Executer.java:69)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/ejml/data/ReshapeMatrix64F
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:303)
	... 5 more
Caused by: java.lang.NoClassDefFoundError: org/ejml/data/ReshapeMatrix64F
	at bigwarp.landmarks.LandmarkTableModel.<init>(LandmarkTableModel.java:163)
	at bigwarp.BigWarp.<init>(BigWarp.java:335)
	at sc.fiji.bdvpg.sourceandconverter.register.BigWarpLauncher.run(BigWarpLauncher.java:135)
	at ch.epfl.biop.bdv.command.register.Wizard2DWholeScanRegisterCommand.run(Wizard2DWholeScanRegisterCommand.java:166)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: java.lang.ClassNotFoundException: org.ejml.data.ReshapeMatrix64F
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 13 more

R Converting:ch.epfl.biop.wrappers.elastix.RegistrationParameters->java.io.File
java.io.FileNotFoundException: C:\Users\myname\AppData\Local\Temp\reg-out3077483254649407265\TransformParameters.0.txt (The system cannot find the file specified)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at java.io.FileReader.<init>(FileReader.java:72)
	at itc.transforms.elastix.ElastixTransform.load(ElastixTransform.java:227)
	at ch.epfl.biop.sourceandconverter.register.Elastix2DAffineRegister.run(Elastix2DAffineRegister.java:161)
	at ch.epfl.biop.bdv.command.register.Elastix2DAffineRegisterCommand.run(Elastix2DAffineRegisterCommand.java:79)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Error during registration

Best regards,
Mario

1 Like

Yes, sorry the warning was written in too little characters in the documentation. Unfortunately only bio-formats supported files are working, and it looks like supporting 160 file formats is not enough.

QuPath uses openslide I think for this file format, and openslide is not supported in fiji.

Your only option with this workflow is consisting of converting the files…

From the full documentation:

"
If your image can’t be loaded in QuPath using the Bio-Formats builder, you can convert your slides in ome.tiff format. Several options are available, for instance by using bfconvert with Kheops, or bioformatsf2raw for a fast conversion.
"

Ah, no, sorry, it’s another error, let me check…

Can you check whether you have the following jar with this version in your fiji jars (Fiji.app\jars\) folder ?

You can also check in fiji using help>update>advanced mode and typing ejml in the search bar.

Also, do you have extra update sites enabled ? Which ones ?

I had ejml-0.21 from update site GDSC-SMLM installed.
After removing this update site, FIJI installed ejml-0.24.

Now the registration goes a bit further. I can see the windows for “bigwarp fixed” and “bigwarp moving” windows as well as “landmarks”. Registration is stuck at this point and I still get the error:

java.io.FileNotFoundException: C:\Users\myname\AppData\Local\Temp\reg-out3077483254649407265\TransformParameters.0.txt (The system cannot find the file specified)

I think elastix is not working. Did you check with the command line that running elastix.exe was working ? (cf doc Installation#Fiji in ⚡ Whole Slide Image Registration With ImageJ/Fiji and QuPath) ?

yes, I did. And I got the normal help menu dialogue.

1 Like

Quick feedback from a debug session with @marioK : the issue indeed comes from the mrxs files. Bio-formats can read them, but very badly : there’s no units and only a very low resolution image is loaded, without pyramids. This is at least one reason for the problem.

1 Like

Hi @NicoKiaru and team,

I’m curious to try your new amazing tool. Thanks for all the detailed instructions. I installed everything and did the obvious checks. However, I have issues with importing a QuPath project in Fiji:
First I created a test QuPath project containing two jpg files (that normally just open in Fiji). Upon opening this QuPath project in Fiji I get:

file:/C:/Users/Bram/surfdrive/Shared/Bram&Rolf/Registration/QuPath%20registration%20project/project.qpproj
[ERROR] Unsupported qupath.imagej.images.servers.ImageJServerBuilder class name provider
[ERROR] Unsupported qupath.imagej.images.servers.ImageJServerBuilder class name provider
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at ch.epfl.biop.bdv.bioformats.export.spimdata.BioFormatsConvertFilesToSpimData.getSpimDataInstance(BioFormatsConvertFilesToSpimData.java:262)
	at ch.epfl.biop.bdv.bioformats.export.spimdata.BioFormatsConvertFilesToSpimData.getSpimData(BioFormatsConvertFilesToSpimData.java:291)
	at ch.epfl.biop.bdv.command.importer.QuPathProjectToBDVDatasetCommand.run(QuPathProjectToBDVDatasetCommand.java:145)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[ERROR] Module threw exception
java.lang.NullPointerException
	at ch.epfl.biop.bdv.command.importer.QuPathProjectToBDVDatasetCommand.run(QuPathProjectToBDVDatasetCommand.java:148)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Then I made a QuPath project with your demo data, exactly like in the movie, and tried to open it in Fiji, but that now seems to contain only the DAB image, and not the two fluorescent channels:
image

The console reads as follows:

file:/C:/Users/Bram/surfdrive/Shared/Bram&Rolf/Registration/QuPath_demo_data_project/project.qpproj
sourceFile:C:\Users\Bram\surfdrive\Shared\Bram&Rolf\Registration\DAB.ome.tiff[bf:0 - qp:0]
[ERROR] Unsupported pyramidize server builder
Attempts to set opener settings for file format OME-TIFF; data location = C:\Users\Bram\surfdrive\Shared\Bram&Rolf\Registration\DAB.ome.tiff
No name found for serie 0 ch 0
identifier.entryID = 1
BdvSpimDataPostprocessor:SpimData found.
[INFO] Source already registered

Any ideas?
Thanks,
Bram

1 Like

Not sure since I have not tested this, but I am not sure regular jpg are typically opened by BioFormats? QuPath usually opens them with ImageJ server, I think, which would not work according to the above posts.

2 Likes

Hi @Research_Associate,

Thanks for thinking along here.
I just checked it. BioFormats also opens the .jpg files, but it converts them into an 8-bit composite with three channels (R, G and B).

In fact I made a mistake stating that they were .jpg files. I thought they were but they are actually .tif files. Nevertheless, Bio-Formats seems to treat them the same way: converted to 3 channel 8-bit composite.

EDIT: The same happens to the DAB.ome.tif in the demo data, so this cannot be it.

Bram

2 Likes

Ok, it does have something to do with the file format.
I converted my .tif files to .ome.tif using Bio-Formats Exporter, created a new QuPath project. This project opens in OK in Fiji!

Next question (but this may just be my inexperience with BDV):
The images are both brightfield. If I show them together in a BDV window everything is completely white. I can show one or the other with Shift-1, Shift-2, but that doesn’t allow inspecting / doing manual alignment.
And changing numbers in the brightness&color window (S) doesn’t do anything.
:thinking:

1 Like

Ok, here’s something at least: when I open the same two .ome.tif files together in a new BDV window via Plugins → BigDataViewer → Open Current Image, the brightness & contrast does work, and now also allows me to adjust the two images separately.

Bram

Thanks for your feedback!

In bdv, you are trying to use the old interface, that’s why it doesn’t work. For B&C, you now need to open the right panel: hover your mouse at the right part of the bdv window, then a blue arrow will show up:

Just click on it. I believe pressing ‘P’ can also work. You can then modify the max value to obtain an effect of transparency.

I’ll also check the `Pyramidize’ image server. My bet is that the pyramidize image server just wraps somehow another image server, a bit like a rotated image server rotates a bioformats image server. I’ll try to make this work.

For the jpg files, I will also check. The issue I have is that I’m not sure the pixel size can be set correctly for jpeg files. In this case, I’ll look whether I can retrieve the pixel size which is set in QuPath.

2 Likes

Also, can you share one of your tif file ? I think in your case I would need to support
ImageJServerBuilder, which shouldn’t be too complicated.

Converting the files is really annoying and makes the workflow much less interesting, because you waste a lot of time.

Hi @NicoKiaru, thanks for the support!

Indeed, the blue arrow or pressing ‘P’ in BDV works. (The BDV ‘show help’ (F1 or via the menu) could do with an update, because it only mentions ‘S’ for brightness&color.)

The .jpg files were a mistake; The files I used were actually .tif files, and I realize now that they are re-saved low-res versions from a larger slide scanner format file, that I made last year for another registration try. (The original files presumably are JPEG-XR; I’ll try to get my hands on them again.)
In the meantime, here is the download link to the .tif files that I tried to include. I believe there is actually nothing special about it.
https://surfdrive.surf.nl/files/index.php/s/8vUrGIbEKnW2535/download

Thanks again!
Bram

1 Like

Ok, I tested quickly, and indeed you won’t get anything useful at the cellular level, because it’s too downsampled. Apart from that, everything works with your images if you import them with these options in QuPath:

So : Bio-Formats builder, and do not check Auto-generate pyramids.

2 Likes

Thanks @NicoKiaru, it works like a charm now!

Also, if I use ‘Bio-Formats builder’ to import the demo files, they are both shown in as BDV sources.
(In your demo movie the image provider is set to ‘Let QuPath decide’. In my case this leads to the fluorescence image being lost.)

Bram

1 Like