CLIJ2 alpha release 🥳

Dear #clij users and friends of GPU-accelerated image processing,

I’m happy to announce the first alpha-release of CLIJ2 :partying_face:. If you have the clij2 update site in Fiji activated, you get it with the next update. Its API reference can be found here.

Experimentalists and early adopters are very welcome to give CLIJ2 a try. You know, I :heart: feedback.

clij2
Yet CLIJ2 is experimental. Please don’t use it for production projects yet.

Highlights

  • CLIJ2 currently contains 146 new operations compared to CLIJ. You can now
    • label connected components,
    • analyse label maps,
    • measure distances in point clouds,
    • work with meshes,
    • determine bounding boxes,
    • use new filters such a Sobel and Laplace,
    • apply filters such as top-hat and binary closing in an easier way,
    • draw lines, boxes, spheres and meshes,
    • copy, crop and paste images,
    • pull regions of interest from the GPU instead of binary images and label maps,
    • push and pull tables to/from the GPU,
    • and a lot more.
  • 5 CLIJ methods were renamed or are misssing CLIJ2. The detailed comparision of CLIJ, CLIJ2 and CLIJx can be found online.
  • Existing CLIJ operations were rewritten for CLIJ2 potentially leading to an expected speedup of 1-10 ms per call. This is especially beneficial when processing small images.
  • The CLIJ-API remains unaffected. CLIJ and CLIJ2 can coexist peacefully. They can actually be combined in macros:
// push images to GPU
Ext.CLIJ2_push("A");
Ext.CLIJ_push("B");
// process them
Ext.CLIJ2_addImages("A", "B", "C");
// pull results back
Ext.CLIJ_pull("C");
  • Java/Groovy/Jython-API gets memory management comparable to Ext.CLIJ_clear(); in CLIJ macro:
buffer = clij2.create(100, 100);
// memory contains one image
clij2.clear();
// memory is empty

CLIJ2 <> CLIJx

The upcoming CLIJ2 may affect the experimental CLIJx API. In case CLIJx users experience issues with their workflows using CLIJx, they can downgrade to the previous release by downloading clij-advanced-filters-0.16.2, removing the number at the end of the filename and replacing clij-advanced-filters-0.20.0 in the /plugins/ directory of Fiji and deleting clij2 and clijx jar files. Please report any issues so that I can fix them!

Like about a year ago: Thanks to everyone who helps testing CLIJ! Without your feedback, CLIJ couldn’t fly!

Happy coding!

Cheers,
Robert

14 Likes

Hey friends of CLIJ2,

just to let you know, I released a new version with some major changes.

There are also new methods in CLIJ2:

  • binaryFillHoles
  • concatenateStacks
  • connectedComponentsLabellingBox and connectedComponentsLabellingDiamond
    (connectedComponentsLabelling will be deprecated)
  • entropyBox (thanks to Pit Kludig for providing the code)
  • excludeLabelsOnSurface
  • excludeLabelsSubSurface
  • equalizeMeanIntensitiesOfSlices
  • floodFillDiamond
  • imageToStack
  • invalidateKernelCache
  • multiplyImageStacksWithScalars
  • powerImage
  • setImageBorders
  • setRandom
  • sumImageSliceBySlice
  • sumXProjection
  • voronoiOctagon

Read more in the release notes.

And a excerpt of new methods in CLIJx (which will hopefully make it into CLIJ2):

  • pullTile
  • pushTile
  • skeletonize

Again, feedback is very much appreciated!

Happy coding!

Cheers,
Robert

7 Likes

Hi @haesleinhuepf
Perhaps we already talked about this already, but am I right that CLIJ2 does not have a watershed option yet? There has been one in CLIJx for some time already, but the results are not the same (worse) as the binary watershed algorithm in Fiji:

Fiji watershed:
image

CLIJx watershed:
image

It would be great if you could look at that, because all-CLIJ2 workflows must currently be interrupted by pulling from the GPU, watershed, and pushing to the GPU again. Let me know if I can contribute anything.

Thanks!
Bram

Hey @bramvdbroek,

thanks for spotting this. Yes, the Watershed is broken in some cases. That’s why it lives in CLIJx waiting for a fix. It is a tricky case, because a parallel implementation of Watershed is not trivial. If you or anyone out there want to have a look, the Java code calls a number of othe CLIJ2 plugins and watershed-custom OpenCL kernels. Initially, it generates a distance map which is not exactly the same as in ImageJ but works in 3D. Afterwards, it detects maxima in this map and eliminates “clumped” maxima pixels. I assume the issue is somewhere in there. Afterwards, it labels spots, dilates them and binarizes the labelmap by the end. If the first part would work, I’d know how to speed up the second part. There were some recent improvements in other places.

It is on my todo list but not before the BETA release of CLIJ2 which is scheduled for this weekend.

Thanks again for the feedback! It’s highly appreciated as it helps me keeping an eye on the more important methods and functions from the users perspective. Thanks! :star_struck:

Cheers,
Robert

2 Likes

Hi @haesleinhuepf,

Thanks for picking this up!
I looked at the code, and tried to find out what exactly goes on. My Java is poor, but I have used IJ macro to generate intermediate results. Here’s what I found up to now:

  1. Indeed the distance map made with CLIJ is quite a bit different than the Fiji Distance map. This need not necessarily pose a problem.
  2. CLIJ2_detectMaximaBox then detects many ‘clumped’ maxima, that lie more or less on ridges. But, these maxima are quite differently distributed for distance maps generated by Fiji and CLIJ (overlay of maxima in red):


    When a Mean2DBox is applied to the distance map before maxima detecting things start to become weird. With a meanRadius of 5 pixels, for ‘Fiji’ the ‘real’ maxima remain (but only if the distance map is converted to 32-bit before sending it to the GPU):

    But in the CLIJ case there are maxima all over the place:

Could this perhaps have something to do with it? I hope it makes sense to you; it leaves me sort of clueless.
Here’s the macro that I used to create the images:

run("CLIJ Macro Extensions", "cl_device=");
Ext.CLIJ2_clear();
close("*");

meanRadius = 0;		//optional smoothing of the distance map
maximaRadius = 1;	//optional changing the radius for maxima finding
make_32bit = true;	//convert distance map to 32-bit before sending to GPU

//Generate binary image
open("path/to/nuclei.tif");
run("Gaussian Blur...", "sigma=2");
setAutoThreshold("Default dark");
run("Convert to Mask");
binaryimage = getTitle();

//Fiji distance map and maxima
run("Duplicate...", "title=distance_map");
selectWindow("distance_map");
run("Distance Map");
if(make_32bit==true) run("32-bit");
Ext.CLIJ2_push("distance_map");
Ext.CLIJ2_mean2DBox("distance_map", "smooth_distance_map", meanRadius, meanRadius);
Ext.CLIJ2_detectMaximaBox("smooth_distance_map", "detected_maxima", maximaRadius);
Ext.CLIJ2_pull("smooth_distance_map");
Ext.CLIJ2_pull("detected_maxima");
setMinAndMax(0, 1);
run("Merge Channels...", "c1=smooth_distance_map c2=detected_maxima create");
Stack.setChannel(2);
run("Red");
Stack.setChannel(1);
run("Grays");
setMinAndMax(0, 60);
rename("distance map + maxima Fiji");

//CLIJ distance map and maxima
Ext.CLIJ2_push(binaryimage);
Ext.CLIJ2_distanceMap(binaryimage, "distance_map_CLIJ");
Ext.CLIJ2_mean2DBox("distance_map_CLIJ", "smooth_distance_map_CLIJ", meanRadius, meanRadius);
Ext.CLIJ2_detectMaximaBox("smooth_distance_map_CLIJ", "detected_maxima", maximaRadius);
Ext.CLIJ2_pull("smooth_distance_map_CLIJ");
if(make_32bit==true) run("32-bit");
Ext.CLIJ2_pull("detected_maxima");
setMinAndMax(0, 1);
run("Merge Channels...", "c1=smooth_distance_map_CLIJ c2=detected_maxima create");
Stack.setChannel(2);
run("Red");
Stack.setChannel(1);
run("Grays");
setMinAndMax(0, 60);
rename("distance map + maxima CLIJ");

close("distance_map");
run("Tile");
Ext.CLIJ2_clear();


You can download the image I used for this here. It produces the following watershedded image:


Thanks again for all your eforts!
Bram

By the way, somehow I now have to type the ‘image destination’ in CLIJ commands as a string, instead of a variable pointing to the output image title, or I get an ‘unknown variable’ error.
It had always worked until two days ago, and suddenly stopped. Even macros that normally execute without problems suddenly fail (like your example macros). I believe it was not triggered by updating CLIJ/CLIJ2, but it just stopped working at some point.
Restarting the computer also didn’t help.

My colleague Marjolijn said she had the same issue.

Very strange, isn’t it?
Bram

1 Like

Hey @bramvdbroek,

thanks for the detailed watershed investigation. That’s very helpful! I’ll dig deeper where you made a hole :slight_smile:

That’s a “bug” coming from the CLIJ-CLIJ2 transition. I’ll organise a better error message. In the meantime, please change this line:

run("CLIJ Macro Extensions", "cl_device=");

To this:

run("CLIJ2 Macro Extensions", "cl_device=");

And let me know if this fixes the issue.

Thanks again for the feedback!

Cheers,
Robert

1 Like

Hey @bramvdbroek,

thanks again for pointing at this. Obviously, it’s a 1.5 years old bug in CLIJ: too many maxima were found. I just fixed it in CLIJ2 BETA (not available via update site yet, please wait 2 days :wink: ).

The distance map is more similar to Fiji now and less spots are detected:

With that, the watershed issue is apparently gone:

Thus, I will move Watershed from CLIJx to CLIJ2. Let’s hope it survives BETA testing :wink:

Thanks for your support! You are awesome! :star_struck:

Cheers,
Robert

1 Like

Hi @haesleinhuepf,

Whoohoo, just in time for the BETA release! Thanks for fixing it so fast. :muscle:
Do you ever sleep, by the way?

And indeed the thing with the variables works when I use

run("CLIJ2 Macro Extensions", "cl_device=");

Thanks again and good luck preparing the release!
Bram

Hi again @haesleinhuepf,
Actually I found another rather general issue which may or may not be easily fixed. Perhaps this has been mentioned before elsewhere, but I couldn’t find it:

With every CLIJ/CLIJ2 operation the pixel calibration (um/pixel) and frame interval information is lost. Is this intentional/inherent? It would be nice if this metadata could be preserved.

Best regards,
Bram

Yes, about twice a day these days :wink:

I think we discussed that earlier :slight_smile: and I agree in general. Unfortunately this is not a trivial task. However, I also see the point that transferring meta-data from one image to another is a bit annoying. Thus, I just introduced pushMetaData() and popMetaData() in CLIJx which can help you here by storing and restoring meta data. They follow the LIFO principle. As soon as the BETA is out, you can try this macro:

run("Close All");
run("T1 Head (2.4M, 16-bits)");

getPixelSize(unit, width, height);
print("Pixel size of " + getTitle() + " is " + width + "/" + height + " " + unit );

run("CLIJ2 Macro Extensions", "cl_device=");

// store meta data of currently active image
Ext.CLIJx_pushMetaData();

image1 = getTitle();
Ext.CLIJ2_push(image1);
Ext.CLIJ2_gaussianBlur3D(image1, image2, 1, 1, 1);
Ext.CLIJ2_pull(image2);

// restore meta data to currently active image
Ext.CLIJx_popMetaData();

getPixelSize(unit, width, height);
print("Pixel size of " + getTitle() + " is " + width + "/" + height + " " + unit );

image

Such a functionality is not really related to GPU-accelerated image processing. I could actually imagine that other plugins have the same issue. Thus, I’m not sure. Maybe such a method would be better suited to become part of core ImageJ or Fiji. What do you think?

Cheers,
Robert

1 Like

Ah yes, I didn’t remember that, sorry.
But I see the point. Most plugins nowadays preserve metadata, but some of them don’t. Indeed, I agree that it would be best if ImageJ/Fiji would take care of it.
But I can imagine that this may cause downward compatibility issues, right?

Bram

So I see no way that ImageJ/Fiji can take care of it automaticallly. If any plugin applies an affine transform to an image. The pixel sizes are scrambled afterwards.

What we could suggest though is having the push/pop meta data mechanism part of the macro language…

That should work fine!
With LIFO, do you mean that you can push and pull metadata from different images to the GPU, in LIFO order?

In non-CLIJ macro language this is actually already possible, but it’s a bit cumbersome, and maybe it doesn’t contain all metadata:

// store meta data of currently active image
metaData = getMetadata("Info");
getVoxelSize(pixelWidth, pixelHeight, pixelDepth, unit);
frameInterval = Stack.getFrameInterval();

// restore meta data to currently active image
setMetadata("Info", metaData);
run("Properties...", "unit="+unit+" pixel_width="+pixelWidth+" pixel_height="+pixelHeight+" voxel_depth="+pixelDepth);
Stack.setFrameInterval(frameInterval);

A single command, like you propose in CLIJ, would make it much easier.
Maybe the brand new Property Functions could be of help? (I haven’t tried them yet).

Thanks for all the efforts! CLIJ2 rocks!
Bram

1 Like