\(•ᴗ•)/ QuPath scripting (#4): Image Combiner (register images and create project entry)

Triggered by the post

I have modified @petebankhaeds QuPath feature ‘Interactive image alignment’ and packed it as QuPath extension ‘ImageCombiner’.

The ImageCombiner can be used to align images and create new project entries with the combined and aligned image channels.

Download qupath-extension-imagecombiner-0.2.3.jar.zip (33.2 KB)
Extract the jar file and save in the QuPath extensions folder (or simply Drag&Drop the jar file onto the QuPath GUI).

The new function is available in the QuPath menu Analyze>Image Combiner.
ImageCombiner_1

The main difference between the dialog of the ‘Interactive image alignment’ and the ImageCombiner are

  • option to scale the overlay image
  • button to create a combined image and add it as new entry into the project

I have tested the combiner successfully with fluorescence and brightfield images.

For brightfield images, the RGB image channels are separate channels in the newly created image and the ColorDeconvolution channels are added.

Here is a combination of two identical brightfield images. One is scaled, shifted ad rotated. The channels LUTs of both are inverted.

I would be happy if someone could provide some more complex test data.

4 Likes

Hey Peter @phaub ,

wow, that’s an amazing initiative! Great work :slight_smile:

Friends and colleagues of mine have released histological whole brain slices of mice open access:

The data is big, but worth a download!

CC @EL_Pollo_Diablo

Best,
Robert

4 Likes

Hi Robert,
Thanks for this source.
With this I could test the alignment of fluorescence and brightfield images of different magnification and I learned that it is not easy to work with different image types. The contrast is very low in the alignment overlay. An improvement would be nice here.
And - as mentioned here -

a 100% alignment on cell level is hardly possible.

Anyway, here is an example showing the overlay of Hematoxylin channel (from P2A_B6_M10_HE_0001-Scene-1-ScanRegion0.czi) and DAPI channel (from 2020_04_25_GNI__0097-Scene-1.czi)

On cell level some deviantions are to be expected.

2 Likes

Hi Peter @phaub ,

thanks for using the data :slight_smile: Maybe a note on the sections: The stainings are from subsequent sections (5µm if I’m not misstaken), so individual cells will probably never align ideally. But I think if you can get the outlines and the brain regions to align, this is of great help!

You would have to adjust the evaluation process to take the remaining missmatch into account, for instance by counting cells or doing colocalization analysis in a tile-wise fashion.

What transformations does the ImageCombiner allow?

2 Likes

Hi Johannes, thank you for that hint.

Interactive alignment:
Affine transformations include translation, scaling, rotation.
scaleX = scaleY
(I’m not sure if the opencv powered AutoAlign function supports reflections and scaleX != scaleY.)

In some cases the visual contrast between image and overlay is weak in the ImageCombiner (and during Interactive image alignment).

Here is an example of the data

The R and B channels of the 4-channel fluorescence image are overlaid with an H&E image in original color.

I have created a second version of the ImageCombiner which optionally applies the display settings (channel selection, Brightness&Contrast) of the overlay image if this image is opened in a separat viewer.

The principle is shown in the following example.
The H&E image is opened in a second viewer and the ‘optical density sum’ channel is selected.
This OD channel is now used for the overlay instead of the original H&E RGB colors.

This improvement is especially helpful when mixing images of different types and in a multi-channel 16-bit fluorescence overlay, where the contrast enhancement can be very pronounced and helpful.

Download the qupath-extension-imagecombiner-0.2.3_v0.2.jar.zip (33.3 KB) (Unpack the zip file and copy the qupath-extension-imagecombiner-0.2.3.jar to your QuPath extension directory.)

@petebankhead @melvingelbard
During my effort to find a solution I got totally lost in the code. After many attempts, I finally found a way out of the labyrinth, a simple and very straightforward one. It is clear and unambiguous and already prepared in the origin.

Assuming that this modification is also interesting for ImageAlignmentPane.java, I will create a PR.

3 Likes

ImageCombiner supporting channel selection

The idea was suggested to support channel selection in image concatenation:

There seem to be some use cases where this could be helpful.

Here I provide a 3rd version of ImageCombiner, which offers the possibility to combine only certain image channels into the concatenated image.
The approach uses the viewer’s Brightness&Contrast settings to derive the channel selection for all images displayed in a separate viewer.

The following examples show 4 images displayed in different viewers with different channel selection. For the base image, the channel selection is R, G, and B. For the overlay images, stain 1, 2, and 3 are displayed.

The approach to use the viewers’s channel selection is not intended to be ‘optimal’ overall, but it is the best I could achieve within the current QuPath code layout.

To understand my ‘workaround’, some background should be explained:

I) In QuPath brightfield images are handled in a special way. From the RGB channels - the only available channels in the brightfield image data - several additional channels are dynamically generated, i.e. stain channels, ODsum channel, normalized OD channel, hue/saturation/chromaticity channels. These additional channels are not present in the image data itself.
If such channels are to be inserted into the concatenated image, they must be ‘regenerated’ by ImageCombiner.
Currently, ImageCombiner only supports the ‘regeneration’ of stain channels.
(Supporting other channels as well would be possible in principle, but the complexity of the current approach would be overloaded and it would be better to look for an alternative design).

II) If stain channels are added to the concatenated image, only the relevant stain channels are added.
This means that for the 2-color brightfield types Brightfield (H-DAB) or Brightfield (H&E), the remaining stain channel is not added to the concatenated image.
For the Brightfield (other) image type, all 3 stains are added to the concatenated image.

III) Channel selection in brightfield images is limited to a single channel at a time or a combination of R, G, B channels. Therefore, a decision was required on how to ‘interpret’ brightfield channel selection in ImageCombiner.
The details can be found below.

ImageCombiner Functionality / important notes:

  • Any number of images can be combined.
  • Images of different types (fluorescence, brightfield), (RGB, nChannel, 8bit, 16bit …) can be combined.
  • The channels of brightfield images are added as separate channels, even if the original data format was packed RGB.
  • The image format of concatenated images is always ‘Fluorescence’, because all channels are treated as separate channels. (Do not change the image type to Brightfield!).
  • The base image is the image of the current viewer.
  • Overlay images may or may not be displayed in their own viewer.
  • When an overlay image is displayed in its own viewer, the viewer’s channel selection is used to display the overlay image and when the new concatenated image is created.
  • The channel selection of overlay images displayed in their own viewer can be changed between the alignment phase and image creation. The channel selection is updated in ImageCombiner by ‘re-choosing’ the overlay images (just press button ‘Choose images from project’ and confirm with ‘OK’).
  • If an overlay image is not displayed in its own viewer, all* image channels are added to the concatenated image. (all*: depending on the image type, see below).
  • The overlay images are aligned one after the other. Each overlay image has its own affine transformation.
  • The new concatenated image is added as a new persistent project entry and displayed directly in the current viewer.
  • The base and overlay images can be removed from the project after the concatenated image is created.
  • The channel names in the concatenated image are derived from the base and overlay image names and the channel names.

Channel selection in fluorescence images
In fluorescence images, each channel is already present in the image data. No channel is created dynamically. Multiple channel selection is possible in the viewer. The single or multiple channel selection of the viewer is used for overlay display and image concatenation.
If a fluorescence image is used as an overlay image and is not displayed in a separate viewer, all channels are added to the concatenated image.

Channel selection in brightfield images
In brightfield images, channel selection is more complex for the reason stated above.
The following logic is currently used in ImageCombiner to “interpret” channel selection:
If one or a combination of R, G, B channels is selected, only the selected R, G, B channels are added to the concatenated image.
If the ‘Original’ channel is selected, the R,G,B channels and all relevant stain channels are added to the concatenated image.
With any channel selection other than a combination of R, G, B or ‘Original’, only the relevant stain channels are added to the concatenated image.
(As mentioned above, ‘relevant’ means 2 stains or 3 stains, depending on the brightfield type.)
Currently there is no possibility to add only a single stain channel to the concatenated image.
If a brightfield image is used as an overlay image and is not displayed in a separate viewer, the R,G,B channels and all relevant stain channels are added to the concatenated image (similar to channel selection ‘Original’).

Despite intensive testing of this new ImageCombiner version, it cannot be ruled out that errors have crept in or that the behavior deviates from the above definition.
Please let me know if you notice any irregularity.

Here is the 3rd version of ImageCombiner
qupath-extension-imagecombiner-0.2.3_v0.3.1.jar.zip (36.1 KB)
(EDIT: v0.3.1 → Bug in viewer update fixed)

Download and upack the zip file. To install either

  • Drag&Drop the ImageCombiner.jar onto your QuPath GUI
    or
  • save the qupath-extension-imagecombiner-0.2.3.jar to your QuPath extension directory (make sure no other versions exist in the extension directory) and restart QuPath.

Here is the result of the above example that was concatenated according to the described scheme:

2 Likes