Compatibility of Imagej-tensorflow with ImageJ1

esgomezm · August 6, 2020, 8:11am

Hi all,

Is there any way to use the TensorFlow library manager (imagej-tensorflow) in an ImageJ1 plugin? (@frauzufall, @ctrueden)

There is a shortcode in the Topic Imagej-tensorflow now with command to change TensorFlow library version / switch to GPU, but I guess it works only for ImageJ2.

Any comment/recommendation is highly appreciated!
Thank you

ctrueden · August 6, 2020, 7:30pm

@esgomezm Let’s discuss your requirements a bit before diving in. Why can’t you use ImageJ2 features? DeepImageJ relies on the TensorFlow Java bindings already—so installation into a plain ImageJ 1.x becomes more complex. The ImageJ Updater was invented to help users avoid this complexity. The easiest path forward for both DeepImageJ developers and users would be to require ImageJ2, and use SciJava-style Commands rather than ImageJ1-style PlugIns.

If continued support of plain ImageJ 1.x is a hard requirement for you, you could still spin up a SciJava application container internally and use that for the TensorFlowService and so on. Something like this:

import net.imagej.tensorflow.TensorFlowService;
import org.scijava.Context;
...
private static TensorFlowService tfService;
static {
  Context ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
  if (ctx == null) ctx = new Context(TensorFlowService.class);
  tfService = ctx.service(TensorFlowService.class);
}

And then use tfService in your code however you want.

The weird IJ.runPlugIn invocation is for accessing ImageJ2’s existing SciJava Context. If you’re running from inside an ImageJ1, it will return null, in which case it spins up a tiny one just for you.

frauzufall · August 8, 2020, 2:58pm

I just want to add notes for how you could just use the existing ImageJ2 resources for what you are doing with DeepImageJ. So here is a quick run through how to use the existing ImageJ2 tools.

You need these dependencies - imagej-tensorflow for library loading and tensor - image conversion, imagej-modelzoo which can mostly do what CSBDeep was always able to to (prediction of arbitrary image to image networks, tiling), but can handle multiple input and output nodes plus has the modelzoo specification API:

<dependency>
    <groupId>net.imagej</groupId>
    <artifactId>imagej-tensorflow</artifactId>
    <version>1.1.6</version>
</dependency>
<dependency>
    <groupId>net.imagej</groupId>
	<artifactId>imagej-modelzoo</artifactId>
	<version>0.6.0</version>
</dependency>

Regarding running them from an IJ1 Plugin, @ctrueden already explained how to get the Context in IJ1.

So we do that or create a new Context and get the services we need, in an IJ2 command we can use Parameter annotations instead (example):

Context context = (Context) IJ.runPlugIn("org.scijava.Context", "");
if (context == null) context = new Context();

// get services which we want to use
TensorFlowService tf = context.service(TensorFlowService.class);
LogService log = context.service(LogService.class);
UIService ui = context.service(UIService.class);
OpService op = context.service(OpService.class);
ModelZooService modelZoo = context.service(ModelZooService.class);
ScriptService script = context.service(ScriptService.class);

// only needed for opening and saving via IJ2
// DatasetIOService datasetIO = context.service(DatasetIOService.class);
// DatasetService dataset = context.service(DatasetService.class);

This is how we can load the TensorFlow library - this snippet is not needed because it is executed automatically during prediction, but if you want to somehow deal with it differenly, you can:

// load TF library, show error if it failed
tf.loadLibrary();
if(!tf.getStatus().isLoaded()) {
log.error(tf.getStatus().getInfo());
}

Now we load the input image - I guess you want the IJ1 way, but for completeness I also added the code for IJ2:

// input paths
String imgPath = "/home/random/tmp/deepimagej/blobs.tif";
String outPath = "/home/random/tmp/deepimagej/output.tif";

// load image via IJ2
// Img input = datasetIO.open(imgPath);
// load image via IJ1
Img input = ImageJFunctions.wrap(IJ.openImage(imgPath));
ui.show("input", input);

We use the ModelZooService to load the model archive:

// load model
String modelPath = "/home/random/tmp/deepimagej/model.bioimage.io.zip";
ModelZooArchive modelArchive = modelZoo.open(modelPath);

Here is how you can extract a macro from the archive and execute it via the ScriptService - we are creating a copy of the input image to not modify the original image:

Img inputCopy = op.copy().img(input);
ui.show("input copy", inputCopy);

// macro preprocessing
File preprocessingMacro = modelArchive.extract("preprocessing.ijm");
script.run(preprocessingMacro, true);

Now we create a default imagej-modelzoo prediction instance and also adjust e.g. the tiling options:

// setup prediction
DefaultSingleImagePrediction prediction = new DefaultSingleImagePrediction(context);
prediction.setInput("input", inputCopy, "XY");
prediction.setTrainedModel(modelPath);
// tiling options - if memory is insufficient, tiling will automatically increase, but it helps to start with higher tiling for bigger images
prediction.setNumberOfTiles(1);

Then we run the prediction and display the output:

prediction.run();
RandomAccessibleInterval output = prediction.getOutput();
ui.show("output", output);

Next, the postprocessing, in your case, is executed via macro, extracted from the model:

// macro postprocessing
File postprocessingMacro = modelArchive.extract("postprocessing.ijm");
script.run(postprocessingMacro, true);

And here again the two ways of storing the result - IJ1 and IJ2:

// save via IJ1
IJ.save(ImageJFunctions.wrap(output, "output"), outPath);
// save via IJ2
// datasetIO.save(dataset.create(output), outPath);

That’s it. Additionally, you can adjust the model specification via API, e.g. from your UI, here is an example, you can find a more extensive example here:

// adjust model
modelArchive.getSpecification().setName("my model");
modelArchive.getSpecification().setDescription("my model description");
modelZoo.save(modelArchive, modelPath);

The API is probably incomplete / not fully in sync with the official modelzoo specification and I would really appreciate issues / PRs in the imagej-modelzoo repository to make sure our specifications match, it’s just a first draft which works well for DenoiSeg and N2V, but it would be great to work together on this.

The code could be shortened more if you would use an IJ2 command and I will create an example repository demonstrating custom prediction commands soon. I will also write more documentation about the ImageJ2 display that exists for the model, it just seems like this is not relevant since you want to use your own UI. The code above is hopefully sufficient to demonstrate how to make use of the existing ImageJ2 TF prediction / modelzoo specification backend resources. Please let me know if anything’s unclear / something I wrote is wrong / the code is causing issues.

EDIT Here is the data for the code above.

carlosuc3m · August 17, 2020, 11:33pm

Thanks @ctrueden and @frauzufall for your answers!
The workaround @ctrueden proposed works perfectly.
At DeepimageJ we have decided to maintain support for IJ1 and IJ2 for the moment, although there will be probably a DeepImageJ version for IJ2 in the future.
With respect to your comments @frauzufall, thanks a lot for you help! It will be useful in the near future. But as I have said, we are maintianing IJ1 for the moment.

Yes! We will be glad to work together. DeepImageJ is following the Yaml requirements defined at the Bioimage Model Zoo. These requirements are actively discussed here.

frauzufall · August 18, 2020, 10:25am

It’s not about the specification itself, it’s about how writing and reading the specification is implemented / available via public API in Java. That’s why we invited you to the Hackathon last December where we wrote imagej-modelzoo, but sadly no developer of your group came, why we had multiple Skype calls and why I made this issue trying to get us to use the same code base, especially since the specification is still discussed and the implementation needs to be updated accordingly from time to time.

carlosuc3m · November 25, 2020, 4:09pm

Hello again.
We tried this implementation with our plugin. It works on an eclipse environment but we were not able to make the plugin run on an ImageJ1 distribution.
To develop the plugin we used the following pom:

<repositories>
	<repository>
		<id>scijava.public</id>
		<url>https://maven.scijava.org/content/groups/public</url>
	</repository>
</repositories>

<dependencies>
	<dependency>
		<groupId>com.google.protobuf</groupId>
		<artifactId>protobuf-java</artifactId>
		<version>${protobuf.version}</version>
	</dependency>
	<dependency>
		<groupId>org.tensorflow</groupId>
		<artifactId>proto</artifactId>
	</dependency>
	<dependency>
		<groupId>org.tensorflow</groupId>
		<artifactId>libtensorflow</artifactId>
   		<version>1.15.0</version>
	</dependency>
	<dependency>
		<groupId>org.tensorflow</groupId>
		<artifactId>libtensorflow_jni</artifactId>
   		<version>1.15.0</version>
	</dependency>
	<dependency>
		<groupId>org.yaml</groupId>
		<artifactId>snakeyaml</artifactId>
	</dependency>
	<dependency>
	    <groupId>net.imagej</groupId>
	    <artifactId>imagej-tensorflow</artifactId>
   		<version>1.1.6</version>
	</dependency>
	
	<!-- Test scope dependencies -->
	<dependency>
		<groupId>junit</groupId>
		<artifactId>junit</artifactId>
		<scope>test</scope>
	</dependency>
</dependencies>

To run the plugin in IJ1 we took all the jar downladed with the pom and installed them on the IJ1 distribution.
HOwever,

Context ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");

does not seem to work as it throws the following error:

[WARNING] Class pool is empty: forgot to call Thread#setClassLoader?

 java.lang.IllegalArgumentException: No compatible service: org.scijava.service.Service
	at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:244)
	at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:195)
	at org.scijava.service.ServiceHelper.loadServices(ServiceHelper.java:171)
	at org.scijava.Context.<init>(Context.java:281)
	at org.scijava.Context.<init>(Context.java:237)
	at org.scijava.Context.<init>(Context.java:133)
	at org.scijava.Context.<init>(Context.java:120)
	at org.scijava.Context.<init>(Context.java:109)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.lang.Class.newInstance(Unknown Source)
	at ij.IJ.runUserPlugIn(IJ.java:239)
	at ij.IJ.runPlugIn(IJ.java:204)
	at ij.IJ.runPlugIn(IJ.java:193)
	at deepimagej.TensorFlowModel.<clinit>(TensorFlowModel.java:124)

Do you know why this might be happening?
Thank you in advance for your time.
Carlos

haesleinhuepf · November 26, 2020, 5:34am

Hey @carlosuc3m,

this is unfortunately a known issue: The SciJava service infrastructure is broken in ImageJ and I think also in QuPath, Icy and MicroManager. In order to make my scijava-based stuff work in these environments, I typically use two workarounds:

Catch the exception and initialize the service yourself:

github.com

clij/clijx-assistant-imagej2/blob/8a70a906cdfc51a6bd0244373dec75c6dcf1234a/src/main/java/net/haesleinhuepf/clijx/imagej2/ImageJ2ServiceSingleton.java#L16


import org.scijava.command.CommandService;
public class ImageJ2ServiceSingleton {
    private static OpService opService;
    private static Context ctx;
    static {
        // this initializes the SciJava platform.
        // See https://forum.image.sc/t/compatibility-of-imagej-tensorflow-with-imagej1/41295/2
        ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
        if (ctx == null) ctx = new Context(CommandService.class, OpService.class);
        opService = ctx.getService(OpService.class);
    }
    public static OpService getOpService() {
        return opService;
    }
    public static Context getContext() {
        return ctx;
    }
}

And have fallback services in case you implemented your own service:

github.com

clij/clij-core/blob/75da576507ea85e7eff6cfe9405adb0a9538e64c/src/main/java/net/haesleinhuepf/clij/CLIJ.java#L470


public <S, T> T convert(S source, Class<T> targetClass) {
    if (targetClass.isAssignableFrom(source.getClass())) {
        return (T) source;
    }
    synchronized (this) {
        try {
            if (converterService == null) {
                converterService = new Context(CLIJConverterService.class).service(CLIJConverterService.class);
            }
        } catch (RuntimeException e) {
            converterService = FallBackCLIJConverterService.getInstance();
        }
        converterService.setCLIJ(this);
        CLIJConverterPlugin<S, T> converter = (CLIJConverterPlugin<S, T>) converterService.getConverter(source.getClass(), targetClass);
        converter.setCLIJ(this);
        T result = converter.convert(source);
        // this is because of the disabled cleaner thread in clij-coremem 0.5.2:
        //RessourceCleaner.cleanNow();
        //System.out.println("Cleaning2");

Let me know if this helps!

Cheers,
Robert

carlosuc3m · November 26, 2020, 11:51am

Hey, @haesleinhuepf,
Thank you for your prompt answer. However I am still sruck with this error. How do you could I initialize the Tensorflow service myself?
It seems that I am not able to get the context.
Regards,
Carlos

haesleinhuepf · November 26, 2020, 12:02pm

Can you try catching the exception and then creating a context like in the example above? You may want to keep this context in a static variable…

  try {
     ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
  } catch (RuntimeException e) {
      // ignoring exception
  }
  if (ctx == null) ctx = new Context(CommandService.class, TensorflowService.class);

carlosuc3m · November 26, 2020, 12:15pm

I have already tried this but I still cannot find the service:

Caused by: java.lang.IllegalArgumentException: No compatible service: org.scijava.command.CommandService
	at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:244)
	at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:195)
	at org.scijava.service.ServiceHelper.loadServices(ServiceHelper.java:171)
	at org.scijava.Context.<init>(Context.java:281)
	at org.scijava.Context.<init>(Context.java:237)
	at org.scijava.Context.<init>(Context.java:177)
	at org.scijava.Context.<init>(Context.java:163)
	at deepimagej.TensorFlowModel.<clinit>(TensorFlowModel.java:130)
	... 2 more

haesleinhuepf · November 26, 2020, 12:28pm

Can you please point us to the exact code you ran and the exact line where the error happens? It’s otherwise hard to debug remotely.

carlosuc3m · November 26, 2020, 12:34pm

Yes sorry!

github.com

deepimagej/deepimagej-plugin/blob/b84bd98e5c60b3ae836aaa8ff00cb0d0678a28d0/src/main/java/deepimagej/TensorFlowModel.java#L124


												   	     "tf.saved_model.signature_constants.REGRESS_METHOD_NAME",
												   	     "tf.saved_model.signature_constants.REGRESS_OUTPUTS",
												   	     "tf.saved_model.signature_constants.DEFAULT_TRAIN_SIGNATURE_DEF_KEY",
												   	     "tf.saved_model.signature_constants.DEFAULT_EVAL_SIGNATURE_DEF_KEY",
												   	     "tf.saved_model.signature_constants.SUPERVISED_TRAIN_METHOD_NAME",
												   	     "tf.saved_model.signature_constants.SUPERVISED_EVAL_METHOD_NAME"};

	
	private static TensorFlowService tfService;
	static {
	  Context ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
	  if (ctx == null) ctx = new Context(TensorFlowService.class);
	  tfService = ctx.service(TensorFlowService.class);
	}
	
	public static String loadLibrary() {
		if (!tfService.getStatus().isLoaded()) {
			tfService.initialize();
			tfService.loadLibrary();
			if (tfService.getStatus().isLoaded()) {
				return tfService.getStatus().getInfo();

This is the line that fails. If I change it by what you suggested it, gives the error mentioned above.

haesleinhuepf · November 26, 2020, 12:40pm

Can you try to catch the exception?

carlosuc3m · November 26, 2020, 12:48pm

I have tried this but still no success.

github.com

deepimagej/deepimagej-plugin/blob/aad7e9edfaaf49ecd57730dcee3139cbb2323e8f/src/main/java/deepimagej/TensorFlowModel.java#L130


												   	     "tf.saved_model.signature_constants.SUPERVISED_EVAL_METHOD_NAME"};


	private static TensorFlowService tfService;
    	private static Context ctx;
	static {
		try {
			ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
		} catch (Exception ex) {
		}
		if (ctx == null) ctx = new Context(CommandService.class, TensorFlowService.class);
		tfService = ctx.service(TensorFlowService.class);
	
	}
	
	public static String loadLibrary() {
		if (!tfService.getStatus().isLoaded()) {
			tfService.initialize();
			tfService.loadLibrary();
			if (tfService.getStatus().isLoaded()) {
				return tfService.getStatus().getInfo();

In lines 127 and 130 is when the errors happen.

haesleinhuepf · November 26, 2020, 12:52pm

Ok, if it complains about the CommandService, can you remove it?

if (ctx == null) ctx = new Context(TensorFlowService.class);

carlosuc3m · November 26, 2020, 12:57pm

In that case it complains about the TensorFlowService

Caused by: java.lang.IllegalArgumentException: No compatible service: net.imagej.tensorflow.TensorFlowService

carlosuc3m · November 26, 2020, 12:59pm

This is the code:

github.com

deepimagej/deepimagej-plugin/blob/4cf2285121ac5dbb295423373d63dc02d619eb1e/src/main/java/deepimagej/TensorFlowModel.java#L130


												   	     "tf.saved_model.signature_constants.SUPERVISED_EVAL_METHOD_NAME"};


	private static TensorFlowService tfService;
    	private static Context ctx;
	static {
		try {
			ctx = (Context) IJ.runPlugIn("org.scijava.Context", "");
		} catch (Exception ex) {
		}
		if (ctx == null) ctx = new Context(TensorFlowService.class);
		tfService = ctx.service(TensorFlowService.class);
	
	}
	
	public static String loadLibrary() {
		if (!tfService.getStatus().isLoaded()) {
			tfService.initialize();
			tfService.loadLibrary();
			if (tfService.getStatus().isLoaded()) {
				return tfService.getStatus().getInfo();

haesleinhuepf · November 26, 2020, 1:04pm

Arrg. Alright, then you might need to implement your own FallbackTensorflowService. I had to do this with my stuff as well. Or can you initialize the Tensorflow service traditionally by calling new TensorflowService (); ?

carlosuc3m · November 26, 2020, 1:08pm

It seems that I cannot instantiate the TensorFlowService, so I will try to create a FallbackTensorflowService as you suggest.
Thank you for your help Robert!

haesleinhuepf · November 26, 2020, 1:13pm

You’re welcome! Don’t give up! You’re almost there. And from own experience I can tell you, the community will appreciate your efforts if you keep your stuff backwards compatible to ImageJ.