Color Deconvolution plugin IJ2

BoudewijnvanLangerak · July 31, 2020, 8:09am

Hi All,

We are working on a deconvolution plugin for IJ2 so we can use it also in KNIME.
It is based on de IJ1 deconvolution plugin.
The thing is we don’t understand what is happening with the images.
We have a working plugin but the result is different in IJ2, KNIME from the IJ1 plugin.

Original image:

The IJ1 deconvolution:

Loading the image in IJ2 results in a 3 channel image. After deconvolution the three resulting images also have 3 channels.
The IJ2 deconvolution:

Running the deconvolution in KNIME results in the same images with the show() function but when they are put in the KNIME table they seem different.

The code for the IJ2 plugin we have created is:

@Override
    public void run() {
        StainMatrix sm = new StainMatrix();
        sm.init("Our stain", R1, G1, B1, R2, G2, B2, 0, 0, 0);
        Img<ByteType> img = (Img<ByteType>) dataset.getImgPlus().getImg();
        ImageStack[] imageStacks = sm.compute(false, true, ImageJFunctions.wrap(img, dataset.getName()));
        ImagePlus imp = new ImagePlus("Image " + 0, imageStacks[0]);
        imp.show();
        ImagePlus imp1 = new ImagePlus("Image " + 1, imageStacks[1]);
        imp1.show();
        ImagePlus imp2 = new ImagePlus("Image " + 2, imageStacks[2]);
        imp2.show();
        deconvolutedImage1 = ImageJFunctions.wrap(imp);
        deconvolutedImage2 = ImageJFunctions.wrap(imp1);
        deconvolutedImage3 = ImageJFunctions.wrap(imp2);
    }

The questions we have are.

Why are the deconvoluted images in IJ1 different from the IJ2 images?
Why do we get 3 channels in IJ2 (and KNIME)?
Why is are the images “strange” when we put them in the table.

I hope you can help us with these questions.

Boudewijn

sunsear · June 30, 2020, 11:13am

You hints in order to use Knime have proven very helpful so far @ctrueden! We already have something working that does the histogram calculation there, so our researchers were very happy with that step. This next step is going to help them extremely, doing the colour deconvolution from the same workflow.

Hope someone can shed some light on why these images look so completely different from their normal results. The ImageJ1 code basically opens an image and passes that to a calculation that does a colour deconvolution. We need to do that from ImageJ2, so the difference is we call ImageJ1 code with ImageJ2 images wrapped by ImageJFunctions.wrap. I hope someone can tell why that causes such a huge difference in output.

imagejan · June 30, 2020, 11:18am

@BoudewijnvanLangerak I changed the topic title to include “Color”, because just “Deconvolution” means an entire different thing to most (image analysis) people.

I guess this is just an issue with the thumbnail rendering and channel composition in KNIME tables, the actual image data (i.e. pixel values) should be unaffected.

fiji/Colour_Deconvolution/blob/dca9a107f1d90b2f15dd7fac4e737534984e64f1/src/main/java/sc/fiji/colourDeconvolution/StainMatrix.java

package sc.fiji.colourDeconvolution;

import java.awt.Color;
import java.awt.Font;
import java.awt.Rectangle;
import java.awt.image.IndexColorModel;
import java.util.regex.Pattern;

import ij.IJ;
import ij.ImagePlus;
import ij.ImageStack;
import ij.gui.NewImage;
import ij.gui.PolygonRoi;
import ij.gui.Roi;
import ij.process.ImageProcessor;

/**
 * This abstract class contains all the library methods to perform image deconvolution based on custom vector.
 * <code>
 * boolean showMatrix=false;//To not display the Matrix used to do the deconvolution in a popup at the end of the process

This file has been truncated. show original

For some reason ImageJ2 images use byte[] internally instead of int[] in ImageJ1

BoudewijnvanLangerak · June 30, 2020, 2:48pm

Did some more testing.
IJ1 and IJ2 show the same image in a different way.
See screenshot.
IJ1 opens the image as RGB and IJ2 as 8-bit unsigned integer with 3 channels.
Can anyone explain why this happens?

phaub · June 30, 2020, 5:30pm

In the StainMatrix.java code (line 294-296) the RGB pixel value is splitted from 24bit int into its byte channels.

        int R = (pixels[j] & 0xff0000)>>16;
        int G = (pixels[j] & 0x00ff00)>>8 ;
        int B = (pixels[j] & 0x0000ff);

If you use byte then this transformation is useless/wrong.

The main problem is mentioned in this post

Either you should ‘force’ IJ2 to treat the data as 24bit int or you should adapt the StainMatrix.java accordingly.

ctrueden · June 30, 2020, 8:43pm

With ImageJ2 we made the decision not to support packed RGB/ARGB/RGBA sample values. I.e.: data is always split to separate channels, with channel treated as a dimension.

The best thing to do here would be to recast the StainMatrix code to work on the ImgLib2 RandomAccessibleInterval interface, not raw byte[] arrays etc. Otherwise, you’ll always only be supporting data stored in memory on the heap, and no larger than 2Gpix—it won’t work e.g. with ImgLib2’s cached cell images that move data to/from disk intelligently on demand.

sunsear · July 1, 2020, 5:56am

Wow, @phaub, @ctrueden, thanks a lot, I feel like a prize idiot now. The code looked so compatible that we were treating it as a black box without thinking any further on it.

To our purposes, this whole wrapping layer is then useless, guess we do have to put in the labour that Curtis is talking about and actually understanding and rewriting what that code is doing. Thanks for the answers, will get back once we have something that works!

sunsear · July 1, 2020, 5:19pm

Ok, put in a day of learning and working to resolve code issues. The result can be found here and is not satisfying yet. I am missing something here, probably either related to colour tables or I am doing something out of order.

github.com

sunsear/colour_deconvolution_IJ2/blob/0468e92a70b9a638740d27c8c78acf15ecee97fc/src/main/java/sc/fiji/colourDeconvolution/StainMatrixIJ2.java

package sc.fiji.colourDeconvolution;

import net.imagej.ImageJ;
import net.imagej.ImgPlus;
import net.imglib2.RandomAccess;
import net.imglib2.display.ColorTable8;
import net.imglib2.img.Img;
import net.imglib2.img.array.ArrayImgs;
import net.imglib2.type.numeric.integer.UnsignedByteType;

public class StainMatrixIJ2 extends StainMatrix {
    private final ImageJ ij = new ImageJ();

    /**
     * Compute the Deconvolution images and return a Stack array of three 8-bit
     * images. If the specimen is stained with a 2 colour scheme (such as H &amp;
     * E) the 3rd image represents the complimentary of the first two colours
     * (i.e. green).
     *
     * @param doIshow    :    Show or not the matrix in a popup

This file has been truncated. show original

I have moved all of the ImageJ version agnostic code into a StainMatrixBase class and now have an ImageJ1 plugin and an ImageJ2 plugin that reuse most of the code.

My new version of the plugin processes the image as a RandomAccessibleInterval as @ctrueden suggested. I haven’t found a way to write it out as such yet, so it still creates a byte and uses that to initialise 3 new images.

That is question 1 I now have: how can I write out an image using something like an RAI?

Then the plugin does all of the plugin magic that I think it should do. In ImageJ1 the code first creates 3 ImageStacks and assigns ColorTables to those. Then it calls addSlice to add byte into those ImageStacks and writes them out.

for (int j = 0; j < 256; j++) { //LUT[1]
    rLUT[255 - j] = (byte) (255.0 - (double) j * cosx[i]);
    gLUT[255 - j] = (byte) (255.0 - (double) j * cosy[i]);
    bLUT[255 - j] = (byte) (255.0 - (double) j * cosz[i]);
}
IndexColorModel cm = new IndexColorModel(8, 256, rLUT, gLUT, bLUT);
outputstack[i] = new ImageStack(width, height, cm);
//A little further on it adds a slice
outputstack[0].addSlice(label, newpixels[0]);

That seems to work fine, the images are as I expect them to be.

When looking at the ImageJ2 version of the code I found I needed to do things in a different order. I create ImgPlus’s and then assign ColorTables to them to the best of my ability:

outputImages[0] = new ImgPlus(ArrayImgs.unsignedBytes(newpixels[0], width, height));
for (int j = 0; j < 256; j++) { //LUT[1]
     rLUT[255 - j] = (byte) (255.0 - (double) j * cosx[channel]);
     gLUT[255 - j] = (byte) (255.0 - (double) j * cosy[channel]);
     bLUT[255 - j] = (byte) (255.0 - (double) j * cosz[channel]);
}
outputImages[channel].initializeColorTables(3);
outputImages[channel].setColorTable(new ColorTable8(rLUT), 0);
outputImages[channel].setColorTable(new ColorTable8(gLUT), 1);
outputImages[channel].setColorTable(new ColorTable8(bLUT), 2);

Question 2: Is the ImageJ2 code I wrote supposed to do the same thing as the ImageJ1 version?

The output of both is very different. This is one of the images from version 1:

expectedDeconvoluted2.tif (11.0 MB)

This is an image from version 2:

outputConvoluted2.tif (11.0 MB)

Hope to hear from you!

Martin and Boudewijn

gabriel · July 1, 2020, 10:45pm

Disclaimer: I never programmed plugins for IJ2, but I am the author of the CD plugin so my comments might help. Apologies if you already know what follows.

The LUTs are for an index colour model that is assigned to each resulting channel image so it approaches the colours of the dyes, but they are not really needed to process the image further, so you can skip that and work with the greyscale results.

However the result is still not correct. If you compare the histograms of the result given by the IJ plugin and your version, you will see that the results are very different, so there is something wrong elsewhere (in addition LUT computation has not been applied, because the image still has a greyscale LUT).

I guess that the problem might be when you cast the image value byte to double.
Please check what values you are getting in the Rlog, Glog and Blog variables (lines 53 to 55). I do not understand why you get a 0 background in the result.

Edit: why don’t you just read the image into double R, G and B variables. I wonder if by adding 1 in line 53 to 55 overflows the byte and that is why you get a 0 background.

sunsear · July 2, 2020, 6:28am

Thanks Gabriel, that is actually quite helpful, I will check out why the result is not correct! I think I will try using output files to write out intermediary results so I can compare what I get at various stages to the 1.0 version of things.

BTW, I think I understand what the colour model is for, and I had a sinking feeling that something was incorrect before that. I am just wondering whether my ImageJ2 code that applies the colorTables is in meaning similar to the ImageJ1 code that I have there.

phaub · July 2, 2020, 7:54am

The main CD procedure of your version (lines 53 - 65) is identical to @gabriel original code. If have checked that in detail.

So, the reason for the different results MUST be somewhere in your IJ2 coding.

In addition to gabriel’s hint (to check the Rlog, Glog, Blog values) I would as a simple test use instead of the CD (lines 53 - 65) the following assignment

	newpixels[0][width * y + x] = R;
	newpixels[1][width * y + x] = G;
	newpixels[2][width * y + x] = B;

to make sure that the indexing is working.
And I would skip the initializeColorTables in line 74 during this test.

ADD1:
The original code can work with RGB ImageStacks. Your code seems to work only on single RGB images.

ADD2 (@gabriel):

a) The IJ2 code is identical to the original.
b) An overflow is not possible because at the time of adding 1 the type is already double.
c) Adding 1 to the R|G|B values before calculating the log is questionable.
Especially because this shift is not handle explicit during re-scaling.
(If you like we can discuss details in a separate thread.)

sunsear · July 2, 2020, 8:54am

All right, thanks @phaub and @gabriel, the hints were actually very helpful! I finally have some first basis working! The original code worked with R,G and B ints that ranged between 0 and 255. In ImageJ2 we get bytes and those range between -128 and 127. I simply had to add 256 to every negative value and treat them as ints again. The first “working” version is there, yay!

I realise the code is not similar to that of the 1 version yet, those are for next steps @phaub. I now have a working StainMatrix that only works on single RGB images. I will first put that into a working very basic IJ2 plugin, then worry about picking up the rest of the IJ1 implementation.

sunsear · July 2, 2020, 9:19am

Ok, transformed that into a working plugin as well. Still confused about the ColorTables though. Can anyone point to some hints how those are supposed to work in IJ2?

phaub · July 2, 2020, 9:37am

@sunsear
Have you seen this

leading to this

sunsear · July 2, 2020, 1:03pm

Yeah, I found those but rereading them did help in realising why the color tables are not applied properly when I output images to file directly.

If I use my plugin and view the images in IJ2 itself as Output parameters, the color table looks really strange though, there is only 1 color table apparently and that colours everything red. These are single channel images, so that might be related. The values in the color table are however completely different than the one that is created in IJ1. When I look at the values that I put into the color tables before they are applied, they look the same as the ones I see in IJ1.

Was hoping there was maybe some documentation on how it is supposed to work somewhere. Thanks for these hints at least @phaub!