Running multiple instances of ImageJ

A lot of the plugins I use (stabilization, etc) can only make use of one processor core and it seems like the computing speed of the processor is the limiting factor when it comes to how fast it can process images. Since we have a multi-core procesor, it feels like a bit of a waste to just use one of all the cores. I understand that programming for multi-threading is hard and maybe even not possible in a lot of cases.

I often split say four channels and process them separately before adding them together again. So instead of running the plugins on one channel at a time I have tried to open several instances of ImageJ to let each of them use one core, thus saving time since the processing is done in parallel. This doesn’t work. I can open the images and start running the plugins, but if I run more than one at a time, I get java-errors a couple of time when it works through the pictures of each channel. The plugin I use is in this case MultiStackReg and the stabilization is not correctly done if I run more than one instance at a time.

Questions: Is this known already? Is there a reason it’s impossible to run more than one stabilization at a time? Is the “fault” in the plugin or in ImageJ itself?

Could you please paste the error message you are seeing?

Apologies for the late reply, been really busy.

I run multistackreg as a macro that makes it register and save transformation matrices on all stacks in a folder. At some point when it’s working through the files, it opens a window name “Exception” with the following contents:

(Fiji Is Just) ImageJ 2.0.0-rc-41/1.50d; Java 1.6.0_24 [64-bit]; Windows NT (unknown) 6.2; 281MB of 98209MB (<1%)

java.lang.NullPointerException
    at turboRegImage.<init>(TurboReg_.java:3393)
    at TurboReg_.transformImage(TurboReg_.java:1254)
    at TurboReg_.run(TurboReg_.java:777)
    at ij.IJ.runUserPlugIn(IJ.java:212)
    at ij.IJ.runPlugIn(IJ.java:176)
    at ij.IJ.runPlugIn(IJ.java:165)
    at MultiStackReg_.registerSlice(MultiStackReg_.java:2290)
    at MultiStackReg_.processDirectives(MultiStackReg_.java:605)
    at MultiStackReg_.run(MultiStackReg_.java:327)
    at ij.IJ.runUserPlugIn(IJ.java:212)
    at ij.IJ.runPlugIn(IJ.java:176)
    at ij.Executer.runCommand(Executer.java:132)
    at ij.Executer.run(Executer.java:65)
    at ij.IJ.run(IJ.java:292)
    at ij.macro.Functions.doRun(Functions.java:601)
    at ij.macro.Functions.doFunction(Functions.java:96)
    at ij.macro.Interpreter.doStatement(Interpreter.java:227)
    at ij.macro.Interpreter.doBlock(Interpreter.java:605)
    at ij.macro.Interpreter.doStatement(Interpreter.java:269)
    at ij.macro.Interpreter.doIf(Interpreter.java:963)
    at ij.macro.Interpreter.doStatement(Interpreter.java:245)
    at ij.macro.Interpreter.doBlock(Interpreter.java:605)
    at ij.macro.Interpreter.runUserFunction(Interpreter.java:308)
    at ij.macro.Interpreter.doStatement(Interpreter.java:230)
    at ij.macro.Interpreter.doBlock(Interpreter.java:605)
    at ij.macro.Interpreter.doStatement(Interpreter.java:269)
    at ij.macro.Interpreter.doIf(Interpreter.java:976)
    at ij.macro.Interpreter.doStatement(Interpreter.java:245)
    at ij.macro.Interpreter.doBlock(Interpreter.java:605)
    at ij.macro.Interpreter.doStatement(Interpreter.java:269)
    at ij.macro.Interpreter.doFor(Interpreter.java:527)
    at ij.macro.Interpreter.doStatement(Interpreter.java:251)
    at ij.macro.Interpreter.doBlock(Interpreter.java:605)
    at ij.macro.Interpreter.runUserFunction(Interpreter.java:308)
    at ij.macro.Interpreter.doStatement(Interpreter.java:230)
    at ij.macro.Interpreter.doStatements(Interpreter.java:215)
    at ij.macro.Interpreter.runMacro(Interpreter.java:138)
    at ij.macro.MacroRunner.run(MacroRunner.java:152)
    at java.lang.Thread.run(Thread.java:662)

The “registered” stacks are not aligned as they are if I run the macro in only one instance at the time. Instead, there are very strange changes made to the stacks that looks like it has aligned some of the slices in the stacks but then put in a slice from some other position halfway and things like that.

Sorry for dropping the ball on this thread.

From the stack trace, it looks like a bug in TurboReg. To my knowledge, the author (Philippe Thévenaz) does not monitor this forum. So if you are still stuck, I would suggest writing to the ImageJ mailing list and CC him directly in your mail.

1 Like

I contacted Philippe and he explained the problem to me:

What happens is that one of the communication channels between TurboReg and MultiStackReg involves files. These files are stored at a single location that corresponds to what is returned by ImageJ’s macro command ‘getDirectory(“temp”);’. If you run multiple instances, then these files may overwrite each other, leading to unpredictable results.

Do you think this is possible to script your way out of? Somehow creating unique temporary directories?

I think the best way would be to change MultiStackReg (and probably also the original StackReg) to write unique file names, e.g. by changing these lines (in @Kota’s version of MultiStackReg_java):

to something like:

import java.util.UUID;

[...]

final String sourcePathAndFileName = IJ.getDirectory("temp") + UUID.randomUUID().toString();
[...]
final String targetPathAndFileName = IJ.getDirectory("temp") + UUID.randomUUID().toString();
1 Like

Note that the SciJava Common library, a foundational part of ImageJ2, includes the FileUtils.createTemporaryDirectory methods for such purposes. You may also find useful FileUtils.deleteRecursively(File) for cleaning up afterwards.

1 Like

Hi David, Curtis & Jan,

I updated the code according to @imagejan’s suggestion (random names given to files). Not tested.

A useful tip from @ctrueden, thanks a lot! For now, I think making a specific directory for each slice would be a bit too much and otherwise I need to change the code more to use a specific directory for all slices. Please think of this as a rescue for the problem.

By the way, I’m wondering what would be the easy way to send a jar for situation like this? A proper way probably would be to register it to a maven repository and provide a download link, which I cannot do quickly, or much easier by putting the jar file in github?

Cheers,
Kota

1 Like

@Kota I thought that’s why you have this? https://github.com/cmci/MultiStackRegistration
So it can live on GitHub, as well as include your changes?
I would not commit a JAR to revision control—only sources.

I think using Dropbox as you did to share binary builds is fine.

@ctrueden Thanks again for the suggestion.

Yes, I asked the question because of this, I had been disturbed by committing binary every time when I compiled and I stopped doing so long time ago. So I wanted to know any general solution these days for distributing the builds.

I just now looked again in the Github about the releases: It seems that the policy of Github has changed several times (see alsp here), and now there is an upload option for release pages - rather manually and separately done aside from the revision control itself.

… so I tried this option.

2 Likes

Thanks for the amazing response on this! As soon as I can find the time, I will put the new MultiStackRegistration to the test and keep you updated on the results!

2 Likes

@Kota I can confirm that your fix works. I was able to run two different batch jobs of multistackreg at the same time with no errors.

I did notice that 1.46 of multistackreg requires java 8 while some other plugins doesn’t seem to run on that version. “Correct 3D drift”, for instance, doesn’t run on the same java version so I have to use two different “versions” of ImageJ (just two different cataloges really) in order to use them both. Not a big problem but I thought someone might want to know.

@David thanks for letting me know. @Christian_Tischer recently worked with Correct 3D drift and he might have some comments on its compatibility with Java8 … Do you maybe @Christian_Tischer?

I just tried it and for me Correct 3D drift works with Java 1.8

The file Correct_3D_drift.py is served by both the Fiji and the Java-8 update sites (the latter one shadowing the former). @David If you’re running on Java 8, please make sure you have the Java-8 update site enabled.

If you download Fiji (bundled with Java), the Java-8 update site is enabled by default.

Thanks, after enabling the Java-8 update site, Correct 3D drift works with java 8.

But instead I got another problem. After enabling that site and updating, I can no longer run multiple instances of ImageJ despite having unchecked the box in Options->Misc. Very strange. It detects the running instance and doesn’t open another one. Also ran it from a windows shortcut with “–allow-multiple” added, but no difference.

This stops me of doing what I wanted to be able to do in my first post. What could be wrong?

This seems to be a bug. Could you please open an issue (or a new forum thread) and describe your system (OS, Java version etc.) as well as a minimal recipe how to reproduce the issue, so that it doesn’t get buried in the posts of this thread here?

1 Like

Happy to report that your fix seems solid, works every time. But today I discovered 533 GB of StackRegSource-files in the Windows temp folder. Perhaps you need to make the plugin “clean up” after itself somehow? It doesn’t seem to delete the files as it is now at least.