Additional jars for scijava jupyter kernel

maven
scijava
imagej2
jupyter

#1

Hi @hadim

we have a scijava-jupyter kernel running on our jupyterhub server. It’s gorgeous :drooling_face:

I also managed to add the jars to get a connection to OMERO going, by adding the necessairy excutables to the kernel binaries and by editing the kernel.json to include the subdirectory.

So on the server the jars live in:
/opt/anaconda3/envs/scijava_env/opt/scijava-jupyter-kernel/
in there I find ij-1.51s.jar, indicating me the version of ij that the kernel is using. Now we would like to take advantage of the decent server hardware config to apply a classifier created with the weka segmentation to a bunch of images. Our first attemps showed that the Trainable_Segmentation.jar is missing as well as many of its dependencies (weka, imagescience, …).

Finally the following two questions arose:

  1. how is the the ij and deps. that you provide in the scijava-jupyter-kernel different from fiji and why?
    (in our use case the scripts are often elaborated in fiji script editor and not necessairly in a local instance of the scijava-jupyter-kernel)
  2. if I want to avoid version incompatibilities between jars, is there a way using maven to pull the jars necessairy for our endeavour with the ij-[version].jar?

Maybe @iarganda as author of the trainable segmentation would have some insights too?

Best regards,

Felix


#2

Hi @FelixM,

Glad to hear you get to work on jupyterhub!

Unfortunately, the project is not active anymore and I think the idea was to replace it with the kernel it is based on the BeakerX kernel.

That would need to be confirmed by @ctrueden.

The scijava-jupyter kernel was a layer on top of BeakerX anyway and the burden of maintenance was probably not worth compared to using the original beakerX kernel.

In consequence, you won’t be able to use script parameters anymore but would have to specifically instantiate an ImageJ object with ij = new net.imagej.ImageJ().

But the nice thing about BeakerX is that it comes with a mechanism to pull any dependencies you want that can make any notebooks reproducible (in term of ImageJ deps):

//load ImageJ
%classpath config resolver imagej.public 'https://maven.imagej.net/content/groups/public'
%classpath add mvn net.imagej imagej 2.0.0-rc-67

So in your case, you could just use the dependencies that the current Fiji version use and you would end up with the same exact classpaths than in Fiji.

Most of the ImageJ tutorials notebooks are being rewritten with BeakerX: https://nbviewer.jupyter.org/github/imagej/tutorials/tree/op-directory/notebooks/

Happy to help more if needed.

Hope it helps.


#3

thanks @hadim for the ‘mise à jour’

I will miss that…

This sounds very exciting.

What I do not quite understand is what replaces the scijava-jupyter-kernel. Is there something like a scijava-beakerx-kernel that can be added to a jupyterhub installation or can the BeakerX kernel be used directly?

Also how does https://github.com/imagej/imagej-notebook fit into this context?


#4

I don’t think there will be any scijava-beakerx-kernel but once again the situation needs to be clarified with @ctrueden.

So at the moment I would suggest you to use the BeakerX kernel directly.

As for imagej-notebook, the short answer is I don’t know :slight_smile:


#5

Hi @hadim

So I gave the beakerx kernel a go. But beside having some server trouble I need to fix, I found that it is not quite as easy to use.

%classpath add mvn net.imagej imagej 2.0.0-rc-67

Does not give me trainableSegmentation.WekaSegmentation, so I need to load those dependencies separately. Is there a Fiji artifact I could reference to get all the plugins I get on the desktop version?

Also given that the scijava-jupyter-kernel is not maintained anymore I would be very interested if there is a vision for jupyter notebooks and ImageJ and what it might be. Since the tutorials are written as notebooks, what is currently the suggested way for Fiji users to run those on their machine? @ctrueden?


#6

Trying to respond quickly, so please ask if anything is not clear…

When you install the beakerx conda package, you get several JVM-based kernels, including Groovy, Java, Clojure, Jython, and others. We are planning to use the BeakerX Groovy kernel directly going forward, for several reasons:

  • Less maintenance burden for SciJava developers. I.e.: it is less niche than SJJK, maintained by Two Sigma and of more general interest to a wider community. No more falling behind new BeakerX features.
  • Solves the “reproducibility problem” we have with SJJK, because you can use %classpath to specify fixed versions of dependencies, as well as extend the classpath at runtime with additional Java libraries.
  • Script parameters features of SJJK was not beneficial for notebooks from a “lines of code” standpoint; you can access a SciJava service from the ImageJ gateway just as easily—e.g. #@ LogService log vs. ij.log().
  • BeakerX notebooks are closer to executable in “vanilla” i.e. non-notebook contexts as well, because ij = new net.imagej.ImageJ() is perfectly fine standard Groovy code, whereas #@ ImageJ ij requires a SciJava-based execution framework to interpret.

It provides convenience functions for working with ImageJ in a notebook. E.g., ij.notebook().methods(object) will use Java reflection to introspect the object and dump a table of available methods, similar to help(object) or dir(object) in Python. The most important function is ij.notebook().display(object), which is capable of transforming certain objects that would otherwise not render properly when output of a notebook cell.

Also: soon, we will use the BeakerX Displayers capability within imagej-notebook to make the cell output rendering smarter for several ImageJ- and SciJava-based objects such as Dataset, ImageDisplay, etc.

Yes. I recommend using the Groovy kernel or the Jython kernel, but really any of them should be OK, based on your taste.

Yep:

%classpath add mvn sc.fiji fiji 2.0.0-pre-7

I have been trying to publish a new version of this Fiji blanket artifact whenever I publish a new net.imagej:imagej. When we reach net.imagej:imagej:2.0.0, we’ll also release sc.fiji:fiji:2.0.0. (Not sure yet whether we’ll try to keep version numbers in sync after that, but probably not.)

One problem I do not yet have a solution for is how to import pom-scijava as a Bill of Materials so that you can use %classpath with multiple artifacts without specifying versions, such that you get the guarantees of the BOM in the same way as you would with a Maven-based project. It may be that BeakerX’s %classpath and/or Groovy Grape have a feature like this, but I have not found any documentation or examples, so I’m guessing not. I asked in the twosigma/beakerx Gitter channel.

As @hadim said, we are in the process of updating all ImageJ tutorial notebooks to the BeakerX Groovy kernel. In the meantime, they are still on master using SJJK, and you can certainly run them that way.

What you cannot do is use the current SJJK-based notebooks but mix in additional JARs. At least: not from inside the notebook. You have to do the hack you described above of adding JARs on the file system. Switching to the BeakerX Groovy kernel will address that very important issue.

I would be very interested to hear about any other issues or concerns you have regarding SJJK being discontinued.


#7

Hi @ctrueden

thanks for the detailed explanations. Your technical arguments for the beakerx kernel are sound. And between some old features not working and the new ones concerning visualisation and especially dependency management, I see the sum of that being definitly in the positive.
Then #@parameter as input in the context of notebooks are probably less relevant. When it comes to #@service there are other ways such as

import net.imagej.legacy.IJ1Helper
log = IJ1Helper.getLegacyContext().getService(LogService.class)

… to get ij-services in a script. This might be longer to write, but in groovy I get code completion to help me with that.

For now I am struggeling a bit to get the ij groovy scripts going on a beakerx kernel on our jupyterhub. Somehow using %classpath add mvn net.imagej imagej 2.0.0-rc-67 in the groovy notebook resulted in a beakerx-groovy kernel that crashes immediatly after launching a notebook (empty or not). So I will have to get to the bottom of this.

Yep. So I had added them manually to the bin once the SJJK was created (as you discribed). Which is obviously much less convenient than using %classpath and leads to a lot of maven-repository browsing.

I have been very enthousiats learning about @hadim’s SJJK. Notebooks are great for so many reasons in the context of science and teaching. Additionnaly @hanslovsky’s imglyb brings together fiji and scikit…
The important thing for me is notebooks, I do not care what the kernel is called. Thinking about it I found it preferable to have a kernel supporting the common/existing jupyter-notebook featues, such as code completion (maybe interactive figures?), than adding new ones while missing out on the former.
Learning about the change of direction concerning ij-notebooks by reading about SJJK beeing discontinued hade me worrieed for a moment, when I did not see immediatly where the path of ij-notebooks will lead in the future. Thanks for clarifying.


#8

Should be easier than that. At the top of your notebook, you say ij = new net.imagej.ImageJ() and then you can do e.g. ij.log() to access the associated LogService with all functions. For access to non-core services you can do myService = ij.get(MyService.class).

Sorry to hear that. Does it work for you locally? On Binder? If it’s only your local JupyterHub, there must be something odd. Happy to help troubleshoot if you are stumped (although I’m no expert on this tech either).

Agreed! :grin:

Glad to do so. I’ll make an announcement when the tutorials have been updated.