Roadmap towards CLIJ 2.5 (Poll)

Hi clij enthusiasts,

following a tradition I’d like to start a discussion about the feature list of the next clij release. Therefore, I would like to know what commands are used by the community. Thus, I made poll (see below).

Background: I’m retracting an earlier announcement: clEsperanto will not have its first stable release in 2021. We just postponed it to 2022. We want to release a really great library with a well-designed API. We can make a better clEsperanto until summer 2022 so let’s not rush. :relieved:

This gives us the opportunity to release a fully backwards-compatible successor of clij2 in summer 2021: clij2.5 :slight_smile: I’m reaching out to hear your opinion on which experimental clijx commands the community would like to see in a stable clij2.5 release.

Here comes a list of commands which IMHO could be relatively easily be deployed as part of clij2.5, because they have been used in a couple of projects and proved useful:

Furthermore, there are some commands which might be interesting as well but weren’t tested in much detail yet. To make them part of the stable clij 2.5 release, support is welcome:

Furthermore, general feedback is welcome as well. For example, we’re maintaining a wish list as github issues here. Add things there or reply in this post if you have a good idea for extending clij. :slight_smile:

Thanks for your opinion!

Cheers,
Robert

8 Likes

At least a bicubic interpolation scheme for geometric transformations would be wonderful. (Don’t see it on the impressive listings.)

2 Likes

Great idea! I just added it to the wish list. However, while linear interpolation is built-in to OpenCL, bi-cubic is not. Thus, the effort might be substantial to make this work while keeping high performance.
Thanks for the suggestion! :slightly_smiling_face:

1 Like

To fulfill this basic request may require substantial investments but isn’t this what science is about? Just using what’s there is …

TransformJ provides various flavours of cubic interpolation and it even provides a quintic scheme.

1 Like

Yep, but dev time is limited, so something being worked on might be several other somethings not being worked on!

3 Likes

A while ago we had a short discussion a while ago about a temporal median filter with regards to an implementation I did on the GPU using clij and some custom OpenCL code at the Holhbein lab.

I think that a proper GPU implementation will still be magnitudes faster and was wondering if one of the features mentioned in the poll was something with regards to an expanded medianZProjection.

If there is some interest (i would be willing to help with an implementation) I can post a detailed workflow in an issue on git (based on my depracted implementation).

Since I also work with rather large datasets, i was wondering if integration with GPUDirect Storage is already present/planned?

1 Like

Hey @DrNjitram ,

amazing initiative, thanks for reaching out!

I quickly went through your code and only found a median-z-projection is there a reason why you are not using the median-filter? Maybe I’m misunderstanding your project though. You do background-subtraction using a median-filter, right?

Regarding potential speedups: We recently did a project with two students (thanks Felix and Hannes!) about benchmarking of some operations in CLIc, CLIJs C++ sibling, made by @StRigaud. It turned out that mean-x-projections are faster than mean-z-projections, at least above a given stack size. Likely that’s related to how pixels are stored in memory.

The question would now be if a Median-X-projection is also faster. We also would need to investigate how much time an axis transposition would take in the context of realistic image sizes. A similar question arises for median-filters in X and Z.

If you could spend some time on this, I owe you a drink! It should be pretty straight-forward to take the java implementation of the MedianZProjection and its corresponding OpenCL-part, put them into a CLIJ2-plugin-template and turn it into a Median-X-Projection. Taking a look into the Mean-X-Projection and its OpenCL-counterpart might help as well.

From this median-x-projection it would then again be pretty straight-forward to program a sliding window that goes in x and computes the median of sub-stacks. I would recommend doing this in the GPU, using OpenCL to avoid repetetive data transfer.

I’m afraid this special memory access is restricted to NVidias CUDA technology and recent GPUs. However, I think if you can wrangle your data before pushing it to the GPU, transfer time can be optimized also without exploting vendor specific stuff. How large are your images? Do they fit in computer RAM?

We can also have a virtual meeting and discuss details.

Let me know if I can help you!

Cheers,
Robert

2 Likes

Hey Robert,

I dont recall the exact reason why i did certain things, it has been a few months and i have been busy with some other projects.

Why i am not using the median filter i wasnt sure, it might have been it was overlooked. The crux was iirc mostly loading data in and out of the GPU in batches. this also answers another question regarding the data size, as datasets could reach >20GB, more than some computers have (~16GB).

It was a rather complex application so I would love to have a quick discussion about the subject!

1 Like

Hi @haesleinhuepf,

I didn’t know about this “Wish List”!
I opened a few new “issues”!

Cheers,

2 Likes

Hi @haesleinhuepf,

Great to see so many features of CLIJx now making it to the next release!
I would also like to suggest (i.e. urgently need) the following methods:

  1. Visualize label numbers on an image, like the ‘Show All with Labels / Labels’ option in the ROI Manager. The numbers could e.g. be printed/oveerlayed on the centroid locations of the labels, or on locations read from a table, to make it more flexible.
    I need this method so that users can easily see which measurements belong to which cell.
    I try to avoid the ROI Manager as much as possible since our last conversation, but the number labeling is something that I have not found an alternative for yet.
    By the way, I remember seeing a CLIJ example of exactly this by your hand, but I cannot find anything like it…

  2. A method called rigidTimelapseRegistration. Much like translationTimelapseRegistration, but then including rotation.
    (I have tried many registration plugins, but find that good old StackReg (or better: HyperStackReg) still works the best in this particular case, but it’s rather slow.)

Thanks a million!
Bram

2 Likes

Hey @haesleinhuepf,
To continue on my previous post: I solved request nr. 1 (visualize label numbers) with the following macro code:

labelFontSize = 16;
Ext.CLIJ2_getMaximumOfAllPixels(labelmap, nrOfLabels);
Ext.CLIJ2_statisticsOfLabelledPixels(labelmap, labelmap);
for (i = 0; i < nrOfLabels; i++) {
	x = getResult("MASS_CENTER_X", i);
	y = getResult("MASS_CENTER_Y", i);
	Overlay.drawString(i+1, x - labelFontSize/2, y + labelFontSize/2);
}

This works for 2D images, but is easily extended to 3D images by including in the for loop:

  • z = getResult("MASS_CENTER_Z", i)
  • a calculation which slice is closest to z (maybe not even necessary)
  • Stack.setSlice(closestZSlice)

Best regards,
Bram

2 Likes

Hey BRam @bramvdbroek ,

that looks great! I think I will make a ImageJ Macro Markdown tutorial for the website out of that. Or do you want to? :slight_smile:

Cheers,
Robert

4 Likes

Hi @haesleinhuepf,

Cool! How about the following IJMMD code? I adapted your labelling example macro and added creating outlines and numbers as overlay. Feel free to adapt for the website of course!

I have two additional questions:

  • Is it possible to display already existing images in the generated HTML without duplicating the image (which I have done in this case)?
  • How do you prevent displaying the huge statistics table?

I also found the resetMinAndMax necessary to keep the outline colors the same as the labels colors, because the autoscaling is different for both images.

Great stuff!
Bram

/*
# Adding label outlines and numbers as overlay 
Authors: Robert Haase and Bram van den Broek, March 2021

[Source](https://github.com/clij/clij2-docs/tree/master/src/main/macro/xxxxxxxx.ijm)

This macro shows how to apply an automated threshold method and label connected components
on an image using GPU. Additionally, label outlines and label numbers are added to the image as overlay.

*/

/* get test data */
run("Blobs (25K)");
input = getTitle();
/*
## Initialize GPU
First we initialize the GPU and push image data to the GPU memory:
*/
run("CLIJ2 Macro Extensions", "cl_device=");
Ext.CLIJ2_clear();

// push data to GPU
Ext.CLIJ2_push(input);

/*
## Blur the image and create a mask using a fixed threshold
*/
Ext.CLIJ2_gaussianBlur2D(input, input_blurred, 1, 1);
Ext.CLIJ2_pull(input_blurred);
Ext.CLIJ2_automaticThreshold(input_blurred, mask, "Otsu");
Ext.CLIJ2_pull(mask);
/*
## Label connected components
*/
Ext.CLIJ2_connectedComponentsLabelingBox(mask, labelmap);

Ext.CLIJ2_pull(labelmap);
run("glasbey on dark");
resetMinAndMax;

/*
## Remove labels touching image borders
*/
Ext.CLIJ2_excludeLabelsOnEdges(labelmap, labels_not_touching_image_borders);
Ext.CLIJ2_pull(labels_not_touching_image_borders);
run("glasbey on dark");
resetMinAndMax;

/*
## Create thick label outlines
*/
Ext.CLIJ2_detectLabelEdges(labels_not_touching_image_borders, labelmap_edges);
Ext.CLIJ2_dilateBox(labelmap_edges, labelmap_edges_dilated);
labelmap_outlines = "label outlines";
Ext.CLIJ2_mask(labels_not_touching_image_borders, labelmap_edges_dilated, labelmap_outlines);
Ext.CLIJ2_pull(labelmap_outlines);
run("glasbey_on_dark");
resetMinAndMax;

/* 
## Add the label outlines as overlay to the original image
*/
selectWindow(input);
run("Add Image...", "image=["+labelmap_outlines+"] x=0 y=0 opacity=100 zero");

/*
## Add label numbers as overlay to the input image
Count labels and measure the label statistics
*/
run("Clear Results");
Ext.CLIJ2_getMaximumOfAllPixels(labels_not_touching_image_borders, nrOfLabels);
Ext.CLIJ2_statisticsOfLabelledPixels(labels_not_touching_image_borders, labels_not_touching_image_borders);

/*
Draw the label numbers as overlays. The x,y coordinates are retrieved from the label statistics in the results table.
*/
labelFontSize = 11;
labelFontColor = "yellow";
style = "bold italic";

setFont("SansSerif", labelFontSize, style);
setColor(labelFontColor);
selectWindow(input);
for (i = 0; i < nrOfLabels; i++) {
	x = getResult("MASS_CENTER_X", i);
	y = getResult("MASS_CENTER_Y", i);
	Overlay.drawString(i+1, x - labelFontSize/2, y + labelFontSize/2);
}
run("Duplicate...", "title=blobs+overlay");

/*
At the end of the macro, clean up:
*/

Ext.CLIJ2_clear();

Hey Bram @bramvdbroek ,

thanks for the great tutorial! I will update the website with it :slight_smile:

I just updated imagejmakromarkdown so that it can do that. For example, this code:

run("Blobs (25K)");
/*# blur*/
run("Gaussian Blur...", "sigma=3");

Now produces this output:

Would you mind trying if it works for your use-case?

2 Likes

Addendum, here comes the link to the tutorial. I rewrote it a bit to focus on the overlay parts.

Thanks again Bram @bramvdbroek ! I learned today about the Overlay.addImage method. Cool! :dark_sunglasses:

Cheers,
Robert

2 Likes

Hi Robert @haesleinhuepf,

Yes, works like a charm now! Great to see the tutorial online.

Incidentally, I noticed that in dilating the outlines for better visibility, I do not dilate the labels, and therefore the outlines are only expanding towards the inside of the label.
To overcome that bias and dilate to both sides we have to extend the labels as well. I found it easiest to use Ext.CLIJx_extendLabelsWithMaximumRadius:

// determine label borders
Ext.CLIJ2_detectLabelEdges(labels_not_touching_image_borders, labelmap_edges);
// make borders a bit wider
Ext.CLIJ2_dilateBox(labelmap_edges, labelmap_edges_dilated);
// Enlarge the labels with 1 pixel as well
Ext.CLIJx_extendLabelsWithMaximumRadius(labels_not_touching_image_borders, labels_not_touching_image_borders_extended, 1);
// bring label numbers (colours) to the label edge image 
Ext.CLIJ2_mask(labels_not_touching_image_borders_extended, labelmap_edges_dilated, labelmap_outlines);
// visualize labeled image
Ext.CLIJ2_pull(labelmap_outlines);
run("glasbey_on_dark");
resetMinAndMax;

The 3-pixel outlines look quite thick on a small image like blobs.gif, but I like them very much on high-resolution images!

The Overlay.addImage is cool indeed; I find myself using it more and more often. Plus, you can have semi-transparant labels by changing the opacity. I had set that to 100, but you may want to change it in the tutorial, just to show the possibilities.

However, for the best user experience we need a button or shortcut command to toggle the overlay visibility in ImageJ. Show Overlay and Hide Overlay both exist, but there is no single command. Would you happen to know how to construct that? I guess a macro tool could do the trick, if only there is a method like Overlay.isVisible that returns the current state…

EDIT: After some source code hunting I found https://imagej.nih.gov/ij/source/ij/plugin/OverlayCommands.java, where the methods show() and hide() are defined. I’m sorry for going so hugely off-topic (we could maybe split), but I would really love to see a function added here like:

void toggle() {
	ImagePlus imp = IJ.getImage();
	if (imp.getHideOverlay()==false) {
		imp.setHideOverlay(true);
		RoiManager rm = RoiManager.getInstance();
		if (rm!=null) rm.runCommand("show none");
	}
	else {
		imp.setHideOverlay(false);
		if (imp.getOverlay()==null) {
			RoiManager rm = RoiManager.getInstance();
			if (rm!=null && rm.getCount()>1) {
				if (!IJ.isMacro()) rm.toFront();
				rm.runCommand("show all with labels");
			}
		}
	}
}

Do you think this code would work?
And then next: how to implement it, and shape it into a macro function / shortcutable command in ImageJ? Do I need to plead to IJ1 developers for this?

Thanks for your advice!
Bram

1 Like

I’m not exactly sure what it is supposed to do? Showing all overlays + all ROI Manger entries? That should be doable.

In macro, there is Overlay.show(); and roiManger.run("Show all"); and friends. So this should be straightforward, I think.

If you can rewrite it as ImageJ macro, you can pin the code to the toolbar or menu as demonstrated in these slides.

If you’re trying to achieve something else than I presume, I would recommend opening a separate thread. You get then support also from experts who don’t follow the clij discussion :wink:

Cheers,
Robert

I just merged the existing show() and hide() methods into a toggle() function. Apparently show() is programmed to show all the ROIs if the image does not have an overlay.

Ah, I didn’t explain it well enough. If a macro now outputs images with segmentation outlines & numbers as overlay, as a user you want to be able to quickly switch the overlay on and off in the UI. Right now this can only be done via two commands in the ImageJ menu (Show Overlay and Hide Overlay). It would be really nice to have a similar command Toggle Overlay Visibility, analogous to the ‘Show All’ checkbox in the ROI Manager), so that people can assign a shortcut key to this command. I think it could be implemented if that little piece of code ends up in https://imagej.nih.gov/ij/source/ij/plugin/OverlayCommands.java.

I’ll ask for it in a separate post and leave this space for CLIJ 2.5 stuff. :slight_smile:

EDIT: Oh, it seems that I have overlooked the macro function Overlay.hidden, which returns the state of the overlay. Then it’s just a piece of cake to make a macro (tool) for it.
And of course, @Wayne Rasband already did it in 2012. :flushed:
(Still, it would be nice to have it implemented by default. :innocent:)

Bram

1 Like