Running CliJ outside of Fiji

Since @haesleinhuepf just put the excellent CliJ paper on BioRXiv, I started playing with CliJ in Micro-Manager. Just loading the CliJ jars (https://github.com/clij/clij/releases) on the classpath (by putting them in the plugins directory) leads to hopeful results, i.e.

import net.haesleinhuepf.clij.CLIJ;
clij = CLIJ.getInstance();
clij.clinfo();

in the Beanshell scripting environment works correctly, but:

p = ij.WindowManager.getCurrentImage();
in = clij.push(p);

results in an exception caused by SciJava (stack trace copied below).

I assume that I need to do some SciJava magic beforehand to register services (Micro-Manager 2.0 already uses SciJava to register plugins, but as far as I know does not scan for Services).

I’d greatly appreciate help from someone in the know (@ctrueden, @imagejan ?) to figure out what needs to be done to get this code to run in the Micro-Manager environment. This should be useful for anyone who wants to use this code outside of Fiji.

Stacktrace:


java.lang.IllegalArgumentException: Invalid service: net.haesleinhuepf.clij.converters.CLIJConverterService
    at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:281)
    at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:249)
    at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:195)
    at org.scijava.service.ServiceHelper.loadServices(ServiceHelper.java:171)
    at org.scijava.Context.<init>(Context.java:280)
    at org.scijava.Context.<init>(Context.java:236)
    at org.scijava.Context.<init>(Context.java:176)
    at org.scijava.Context.<init>(Context.java:162)
    at net.haesleinhuepf.clij.CLIJ.convert(CLIJ.java:395)
    at net.haesleinhuepf.clij.CLIJ.push(CLIJ.java:356)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at bsh.Reflect.invobsh % keMethod(Reflect.java:131)
    at bsh.Reflect.invokeObjectMethod(Reflect.java:77)
    at bsh.Name.invokeMethod(Name.java:852)
    at bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:69)
    at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:96)
    at bsh.BSHPrimaryExpression.eval(BSHPrimaryExpression.java:41)
    at bsh.BSHAssignment.eval(BSHAssignment.java:71)
    at bsh.Interpreter.run(Interpreter.java:471)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No compatible service: org.scijava.plugin.PluginService
    at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:244)
    at org.scijava.service.ServiceHelper.createServiceRecursively(ServiceHelper.java:341)
    at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:270)
    ... 22 more
2 Likes

Hey @nicost,

thanks for the initiative!

@ctrueden / @imagejan I guess you can help us. What is the minimal dependency to just get some SciJava services running. So far, CLIJ derives everything from scijavas parent pom:

And basically it just depends on imagej, and imglib2:

Is there a way for sparing dependencies? We really just need some RandomAccessibleIntervalss, ImagePlusses, and Contexts/ Services.

Any hint is appreciated. Thanks a lot in advance!

Cheers,
Robert

2 Likes

I am not even sure if it is a matter of dependencies (I’ll test that by putting all 256 jars on the classpath;). I suspect that scijava needs to do some kind of scanning to list all available Services (and Plugins, and what not). Hope we will be enlightened soon!

1 Like

Just tested with all jars on the classpath (252;). Same issue.

1 Like

Hey @nicost,

I think I got it!

I found a way to circumvent using the CLIJ internal converter service. All you need are two converters which you can create for yourself. As you suggested, I copied the clij-release files to the plugin folder of MM:

Could you please test if the following beanshell script works in your MM as well?

If this works, we only need to find out how to run the service-initialisation correctly. Having the converter service is quite practical - especially if you want to send a byte-stream directly from a camera towards the GPU, without doing a detour via ImagePlus :wink: But I’m optimistic that we get this running as well.

Cheers,
Robert

Looks like we arrived at the same work-around at the same time (see my private email)!

As for MM data, if I understand correctly, all one needs is a ClearCLBuffer, which should not be too hard, as the native MM storage is a java.nio.Buffer.

The main difficulty will be to untangle the current dependencies of CliJ, so that it can be brought in through maven/ivy, rather than copying jars and hoping that all used dependencies are present at run-time.

1 Like

Shouldn’t need to. The JARs just need to be on the classpath.

Note that SciJava services are plugins of type Service.

The minimal dependency is org.scijava:scijava-common.

The clij-core library depends on much more:

$ git grep -h '^import' | sed 's/\.[A-Za-z0-9\*]*;$//' | sort -u
import clojure.lang
import ij
import ij.gui
import ij.plugin
import ij.process
import java.io
import java.lang.reflect
import java.nio
import java.util
import net.haesleinhuepf.clij
import net.haesleinhuepf.clij.clearcl
import net.haesleinhuepf.clij.clearcl.backend
import net.haesleinhuepf.clij.clearcl.backend.jocl
import net.haesleinhuepf.clij.clearcl.enums
import net.haesleinhuepf.clij.clearcl.exceptions
import net.haesleinhuepf.clij.clearcl.util
import net.haesleinhuepf.clij.converters
import net.haesleinhuepf.clij.converters.implementations
import net.haesleinhuepf.clij.coremem
import net.haesleinhuepf.clij.coremem.enums
import net.haesleinhuepf.clij.coremem.offheap
import net.haesleinhuepf.clij.kernels
import net.haesleinhuepf.clij.test
import net.haesleinhuepf.clij.utilities
import net.imagej
import net.imagej.ops
import net.imglib2
import net.imglib2.img
import net.imglib2.img.array
import net.imglib2.img.display.imagej
import net.imglib2.loops
import net.imglib2.realtransform
import net.imglib2.type.logic
import net.imglib2.type.numeric
import net.imglib2.type.numeric.integer
import net.imglib2.type.numeric.real
import net.imglib2.view
import org.apache.commons.math3.stat.descriptive.summary
import org.jruby
import org.junit
import org.scijava
import org.scijava.plugin
import org.scijava.service
import static junit.framework.TestCase
import static net.haesleinhuepf.clij.test.TestUtilities
import static net.haesleinhuepf.clij.utilities.CLIJUtilities
import static org.junit.Assert

So here we have dependencies on clojure, imagej1, clij-clearcl, clij-coremem, imagej-common, imagej-ops, imglib2, imglib2-ij, imglib-realtransform, commons-math3, jruby, junit, and scijava-common.

You can discover which dependencies are actually being used (as opposed to declared) by running:

mvn dependency:analyze

I did this, and filed a couple of PRs, to get you started:

With this change, the clij-core dependencies drop from 161 to 68. :muscle:
If you make clij depend on the slimmer clij-core, you can similar check it for clij itself (mvn dependency:list| grep ' ' | wc -l).

If your intent is to keep things slim, then CLIJ will need to be modularized further, to isolate the dependencies on other artifacts as mentioned above. Happy to meet during my visit later this month, if you want advice on how best to do that.

As an aside: using a property in a project’s <version> declaration (e.g. <version>${clij_core.version}</version>) is considered evil. The Maven build even warns about it. Best to avoid that.

2 Likes

Thanks @ctrueden!

MM does not have an easy way to put jars on the classpath. Instead, it relies on the ImageJ1 class loader (which is how the clij-core jar was loaded). I assume that difference explains the Exception.

I have trouble understanding the Micro-Manager plugin loading code, but it is quite possible that it does not “scan” the jars in the plugin directory at all (and, regretfully, when I put them in the mmplugins directory, they will be loaded by a different class loader and not be accessible from anywhere else). The plugin loading code is here: https://github.com/nicost/micro-manager/tree/ViewerPlusCV/mmstudio/src/main/java/org/micromanager/internal/pluginmanagement.

Any suggestions on how to make this work are very welcome.

1 Like

Wow, thanks, @ctrueden! This looks awesome!

I was already looking forward to meet you here in Dresden. Let’s have that chat. And thanks again for showing how clean up the dependencies. That’s super helpful!

CU!

Cheers,
Robert

1 Like

Hey @nicost,

I just wrote a hard-wired Fallback-service making clij.push, clij.pull and clij.convert run in MicroManager. This unfortunately hinders boofcv-interoperability for the moment. You find an update in the recent release (which is now no BETA anymore! :partying_face: )

Furthermore, I added installation instructions for MicroManager to the documentation.

The installation via Maven to MicroManager with minimum dependencies stays on my todo-list, but it’s nothing one should do in an overnight-intervention. :wink:

Cheers,
Robert

1 Like

@haesleinhuepf: Works like a charm! I’ll look into writing a utility to push NIO buffers from Micro-Manager directly into a clearCLBuffer. Maybe it would be easiest for now to have this live in CliJ, since it will work with any image that is in an NIO buffer (moreover, it is currently difficult to add a dependency on CliJ to Micro-Manager because of the many dependencies it has). Actually, it looks like most of the code is already in ImagePlusConverters. I’ll hopefully get a pull request to you pretty soon.

@ctrueden: It would still be very nice to be enlightened about using SciJava for dependencies that are loaded through a classloader rather than by being on the classpath, but the need is no longer high, so next time in the bar?

1 Like

The reason I didn’t reply is because it’s not something I can discuss coherently without digging more deeply. So while I look forward to our next bar time, I’m not sure how much light I could shed on the question in that setting.

I know that by default, Fiji has only jars/imagej-launcher-4.0.5.jar on the system classpath. All other JARs are loaded via—I think?—ImageJ1’s plugin class-loader. And somehow, in that situation, all the SciJava plugins still get discovered just fine. Maybe it’s just that when creating an org.scijava.Context, the current thread’s context class loader is used? Do you set the context class loader in Micro-Manager? Since it sounds like Micro-Manager might use a mix of different class loading approaches, the situation may be more complicated. I’m sorry I don’t have time to dig into it and understand how Micro-Manager works and recommend a solution to the issue. I believe @marktsuchida understands the situation though, so perhaps he can comment further on a way forward.

1 Like

Hey @nicost,

Such a converter exists already - almost, it uses StackInterfaces instead of NIO buffers, but they are related to each other:

There are some more converters in the same package bridging CLIJ and ClearControl. Feel free to grab code there. The cool thing about the ConverterService is that having something like clearcontrol-lightsheet on the classpath suddenly enables clij.convert(whatever, ClearCLBuffer.class) converting anything into the right type. We just need to get the converter service running :wink:

@ctrueden: Thanks for the hints! I’m pretty sure we’ll manage that. Btw. also thanks for building and maintaining that amazing service/plugin infrastructure. CLIJ wouldn’t work without it :muscle:

Cheers,
Robert

That would bring in a dependency on ClearControl. I would love to reduce rather than increase dependencies. And I am not a big fan of code that automatically executes differently after bringing in a new jar. That makes it much harder to understand what is actually going on (exemplified by how hard it is to understand what currently is happening with class loading and scijava plugin loading in Micro-Manager). I rather call the correct function myself rather than have some magic figure it out for me (or not, as we just experienced).

I added a super simple java.nio.buffer converter to clij-core, and I created a pull request on github (https://github.com/clij/clij-core/pull/5 ), including a modification of your script that snaps an image in Micro-Manager, pushes the image from a Direct Byte Buffer into a ClearCLBuffer, blurs it on the GPU, and displays using ImageJ. I’ll look into converting it back into a Micro-Manager Image object.

Please have a look and let me know where you would want this (or something similar) to live. It is more a matter of code organization/architecture than anything else.

1 Like

That was not the intention. I meant doing something like for ClearControl. But of course for MM. A clij-mm repository. We could also fix the dependencies in there :wink:

However, if it’s just a converter, I can just pull it into clij-core… Let me think of it…

Have a nice weekend in the meantime!

Cheers,
Robert

Hey @nicost,

I just took your code and put it in a separate repository. (See my message on your github PR).

I would like to dive deeper into the dependency mismatch between MM and CLIJ. Therefore, I need the right maven pom for MM. I would then try to make a CLIJ-pom that is as compatible to the MM-pom as possible minimizing additional dependencies.

Is this the right pom?

<dependency>
  <groupId>org.micromanager</groupId>
  <artifactId>MMCoreJ</artifactId>
  <version>2.0.0-SNAPSHOT</version>
</dependency>

Thanks!

Cheers,
Robert

Micro-Manager is not mavenized. I put something up on a private maven site (http://valelab4.ucsf.edu/~nstuurman/maven/) a long time ago, certainly out of date now, and does not list dependencies in a useful way. There are likely other efforts along those lines, and you may have stumbled on one of them, but none of them is in sync with the main project (which currently seems to be in my github branch ViewerPlusCV - coincidentally started to get a ClearVolume viewer integrated in Micro-Manager;) Getting that all correct is a big project, and not something I will undertake soon, but hopefully there will soon an active maintainer who can take this on.

Also, I really do not think that it is useful to depend on MM. The converter I wrote only knows about Direct Byte Buffers, which happens to be the way MM stores pixel data. By combing that with knowledge about dimensions (as I did in the class NioBuffer), you have all you need to convert to a ClearCLBuffer (and back). No dependencies, all build-in Java classes that are always available. Much cleaner and simpler than a project that depends on 300 outside projects, and is also available to others who store images in Direct Byte Buffers but do not want to depend on Micro-Manager. It makes total sense to me to put that in clij-core as it does not add any dependencies and increases its versatility.

1 Like

Alright. I’ll minimize dependencies on clij side then. Btw, the “IJ” stands for ImageJ. I don’t think making clij independent from ImageJ makes it better. The ImageJ-independent layer behind clij is called ClearCL. Creating any more layers just leeds to lasagne code and it will be harder to maintain it…

Sure! But at the moment clij and clij-core are under code-freeze, read here why. I thought doing everything micromanager related in a repository makes sense and later we merge the right parts of it in clij-core.

Cheers,
Robert

Understood, however, it would be so simple to make this more universally usable. Your kernels and code to manage memory on the OpenCL and Java side are super nice, but bringing that in through maven now comes at the cost of a huge number of dependencies that have nothing to do with the task at hand. This goes not only for Micro-Manager, but any image processing code outside of the ImageJ/imglib2 ecosystem. For Micro-Manager it may be best to copy the relevant code, which is a bit of a pity with respect to re-usability.

Maybe I should try forking Clij-Core, remove all ImageJ/imglib2/SciJava dependencies and put that on maven? That way, this great code will become accessible to more developers.

1 Like

Feel free to do that. It’s BSD licensed. However, before you rip something apart, be aware: It was hard work to get a stable clij release. Don’t touch a running system. I would like to establish CLIJ as a stable base the people can rely on. That’s why I don’t touch clij-core now. But we can do that at the right point in time.

Let’s think and discuss more about the idea. Let’s bring it in context with the other ideas in the air. Then let’s make decisions on priorities. We have one year time until CLIJ 2.0.0 and I want it to be double as awesome as CLIJ 1.0.4. :wink:

If you decide to make a clij fork, please keep the other ideas in mind which may happen during the next year before the next clij release:

  • Reduce dependencies of clij-core to a minimum.
  • Solidify the clij bridge towards boofcv
  • Extend the clij bridge towards imagej-ops
  • Build a clij bridge towards NanoJ. They have super cool stuff implemented on the GPU.
  • Build a clij bridge towards ITK. I worked 7 years with ITK and see big potential here.
  • Improve clij-based deconvolution
  • Get a GPU-accelerated FFT into clij
  • Extend functionality of clij with more advanced clij-filters

Apart from all these things, my higher priority is helping people getting started with CLIJ. I’m happy to help people translating their CPU workflows to GPU workflows because in that way I learn what other functionality extensions people need.

Let’s make clij-micromanager become a super convenient GPU-acceleration toolkit running on microscopes to allow people doing smart microscopy e.g. on OpenSPIMs. I would totally be on board for that project :muscle: