A roadmap towards CLIJ2

Hi fans of GPU-accelerated image processing,

this is a call for feedback. After the clij-paper is out, it’s time to think towards clij2 and I’m wondering about users opinions.

The schedule is basically fixed:

  • CLIJ2 will release in June 2020
  • There will be fully functional beta-test versions in May 2020
  • I expect the first functional prototype in January 2020

What new functionality will be in?

There are new features which will come very likely:

There are things which could be in but it depends if people really would use it:

Furthermore user suggestions:

What does the community think? Is there something decisive missing in CLIJ what I haven’t mentioned yet? Please keep in mind, CLIJ is basically a collection of atomic functions. So, there will likely never come a “TrackMate” or “DeepLearning” operation to CLIJ :wink:

Thanks for your feedback in advance! You’re great guides :slight_smile:

Cheers,
Robert

10 Likes

Hi @haesleinhuepf,

I recently tried to CLIJ-ified our DEVILS plugin but I would need to use a rolling ball subtraction.
Can it be added to CLIJ2?

Best,

Romain

2 Likes

Hi @haesleinhuepf,
Thanks for this, it’s brave, and also a good way to collect feedback.
To chip in on the jython side: please enable using the with statement for automatic clean up, like this:

from __future__ import with_statement

with CLIJx.getInstance("630") as clijx:
    # do whatever with clijx

# When execution exits the 'with' code block, clijx.clear() is called automatically
# even in the event of an error within the 'with' code block.

All that is needed is for the clijx instance to implement __enter__ and __exit__, as in this example: https://stackoverflow.com/questions/3774328/implementing-use-of-with-object-as-f-in-custom-class-in-python/3774396#3774396
These two methods can be implemented in the java side too.

The PEP (Python enhancement proposal) that introduced with is here: https://www.python.org/dev/peps/pep-0343/

3 Likes

Wow cool, clijing devils :slight_smile: I’m happy to put some work into this. However the implementations I found (see also discussion here ) are a bit odd. I’m wondering - isn’t the rolling ball background just a combination of basic filters? (Haven’t read the paper yet) If you find a simplified implementation of it, let me know, I’m happy to translate anything to OpenCL which is useful to people. :wink:

Hey @albertcardona,

wow, this is so beautiful, I will asap try it out. Thaanks for this suggestion and stay tuned!

I’ll update the list above with both of your ideas! Thanks a lot for the feedback @romainGuiet and @albertcardona :slight_smile:

1 Like

Along the same lines, but not specific to Jython, I’d suggest:

  • Implement java.lang.autoCloseable, so that you can use it in try-with-resources statements:
    • from Java:
      try (CLIJx clij = CLIJx.getInstance("630")) {
        // do whatever with clij
      } // no need to close in a catch clause...
      
    • from Groovy:
      CLIJx.getInstance("630").withCloseable { clij ->
        // do whatever with clij
      }
      
2 Likes

Hey @imagejan,

also great idea. We might have to wait for a hard bugfix in an upstream dependency before we should implement autoclosable, but I like the idea and I’m happy to put it on the list. Thanks! :slight_smile:

I’ve got a handful of ideas.

  1. this one is kind of trivial. Typically when I pull processed images from the GPU, the resultant image ends up being stripped of all of it’s metadata (pixel sizes, voxel length, frame intervals, etc.). Any chance it could be made to retain this information (or most of the image metadata)? Or is there a better workaround in ImJ that y’all use?

  2. Would anyone else benefit from having most of the MorphoLibJ features GPU-enabled? There’s a lot of great filtering and segmentation options on there. I find myself constantly using it for most of the workflows I’m developing in our center.

  3. The FIJI native rolling ball background subtraction and denoising (e.g. despeckle) features are really well done. But VERY slow. Seems like a perfect opportunity for CLIJ to shine.

Just a few ideas. I hope that’s helpful.

-Wilson

2 Likes

Hey @Wilson_Adams

great! I was counting on you :wink:

Regarding 1: Meta data. That’s a big effort, but you’re absolutely right, we should dig into this. Thanks for reminding!

I think many (most?) of them are in the list for clij2 (above) already. Some come with different names, e.g. morphological dilation (MorpholibJ) comes a maximum filter in CLIJ. However, would you mind sharing a prioritised list of your favoriite filters? Recently, some students and volunteers asked for filters which need to be implemented. I could point them to this list :wink: Feel free to add filters which are not in MorpholibJ but you like them anyway.

Let’s start a controversy: I know this filter is very useful, I’m a big fan as well. However, I have huge issues in reading its code. Until now I refused to port it because I’m trying to keep code in CLIJ readable and maintainable. I’m aiming to have CLIJ live long term. Therefore, maintainability is key :wink: But if the community decides this filter is important, I will translate it to OpenCL either word by word or I manage simplifying it.

Thanks again for your feedback! I’m happy to update the list on top :slight_smile:

Cheers,
Robert

Addendum: Despeckle is just a median filter?!

I don’t know if this is a particularly useful contribution but when I’ve been wanting a local background subtraction while using Clij I’ve been using the top hat filter and at least on the stuff I work on the results look basically indistinguishable between that and the rolling background subtration.

2 Likes

Hey @lmurphy,

that was a good hint! The rolling ball may indeed be some mix of top-hat and mean filter…

This image shows four different backgrounds which could be subtracted:

  • maximumSphere of the minimumSphere of the original (CLIJ)
  • maximumBox of the minimumBox of the original (CLIJ)
  • rolling ball background (ImageJ)
  • rolling paraboloid (ImageJ)

The rolling paraboloid might be a kind of a blur…

Thanks :slight_smile:

1 Like

Ok, who finds a combination of min/max/mean/blur/… filters which mimic the rolling ball and/or rolling paraboloid on blobs.gif with radii 10, 20 and 50 gets a Fiji or CLIJ t-shirt for free.
Winner is the first who implements it with mean-squared error (MSE) = zero or the one with minimal MSE (averaged over radii 10, 20, 50) by December 31st 2019 23:59 CET :sunglasses:

3 Likes

This is news to me hahaha. Thanks for pointing that out!

-Wilson

1 Like

Thanks Robert!

The way i currently handle the metadata issue in macro scripting is just sandwich-ing the CLIJ commands between getPixelSize() and run(“Properties…”, "pixel/voxel size info ") and it seems to do the trick for volumes and time lapses for the time being. Not sure if that’s something relatively easy to implement on the source code side of things.

As for the MorphoLibJ filters, the morphological convolutions are really handy for segmentation. I personally use the the:

  1. laplacian (already suggested) ,
  2. 2D gradient (sobel, already suggested)
  3. dilation and
  4. closing a lot.

These operations in combiation with
A) disc and
B) square kernals. Perhaps the
C) top hat kernal little more thanks to the genius of @lmurphy for rolling ball bkgd subtraction. Maybe an applet to define
D) custom angled ellipse and
E) irregular polygon kernels?

i dont know if there’s already a good plugin for that yet.

I personally love the marker-based watershed tools, which come in very handy for a lot of my workflows lately - Lots of time lapse imaging of dense cell cultures.

The directional filtering and morphological convolutions are also really nice to mitigate extraneous noise and streaking artifacts.

-Wilson

1 Like

Maybe the rolling ball challenge should be its own thread? I suspect the code could be accelerated on CPU as well if we figure this out.

1 Like

Alright! Opening and closing are indeed missing and super easy to make! Thanks for your list @Wilson_Adams !

Good point! Also: Did you know, you can run clij/OpenCL also on Intel CPUs :wink:

I started a separate topic for the rolling ball challenge:

2 Likes

Maybe you could join forces with @schmid to make the code more readable (as he is the author of at least the sliding paraboloid part of the code of this plugin, if I understood correctly).

3 Likes

Cool to see the with statement already supported. Glad that was an easy feature!

2 Likes

Indeed, it was an easy one (and I learned something about jython :shushing_face:). If anybody wants to give it a try, here is the example script:

https://github.com/clij/clij-advanced-filters/blob/master/src/main/jython/with_statement.py

Thanks @albertcardona for the suggestion! This could indeed make coding much easier!

1 Like