Image J Macro that calls Trainable Weka Segmentation functions well but fails after several repetitions, with a Null Pointer Exception and the message "could not apply classifier"

Hi everyone/anyone that can help :slight_smile:

I have a macro that loops through many folders, each containing a set of .tiff images. Each folder is opened as a stack, and after some pre-processing, I run the Trainable WEKA Segmentation plugin, load a pre-trained classifier and get the resultant probabilities for saving.

The first iteration of the loop is always successful, however at some point in the later iterations it will fail. The problem is not specific to the stack being worked on as beginning the loop again on the failed folder is always successful. The bug happens after a varying number of iterations, but at the earliest in the 2nd and at the latest by around the 10th iteration.

I guessed that it might be a memory issue, but, although I normally run this script on stacks of 36 images, each 32-Bit and 1392x1040 pixels, I have tried the same on stacks of only 3 images of 1/4 the size and I get the same problem. I also increased the memory allotted to Fiji to 97GB and added the run("Close All"); and run("Collect Garbage"); commands to the end of each loop to free up more memory, but it did not improve the situation.

Whenever the loop fails, an exception appears upon calling the Trainable WEKA plugin (below), but it will continue to run for some time before several Fiji error windows appear saying “Could not apply classifier” and the console gives an error report (also below). I am using Fiji with Image J 1.53c and the Trainable WEKA plugin version 3.2.34 which, as far as I can tell, are updated versions.

Below I copy an abridged version of the Fiji macro with the relevant commands, and the exception message that appears.

Macro:

rootPath = “Z:/rootDir”
dirList = getFileList(rootPath);
for (iDir = 0; iDir<dirList.length; iDir++){

open(rootPath+"/"+dirList[iDir]);
run(“Trainable Weka Segmentation”);
wait(10000);
call(“trainableSegmentation.Weka_Segmentation.loadClassifier”, “Z:/classifierDir/classifier.model”);
wait(10000);
call(“trainableSegmentation.Weka_Segmentation.getProbability”);
wait(10000);
selectWindow(“Probability maps”);
saveAs(“Tiff”, “Z:/resultsDir/results_dirList[iDir]_.model”);
run(“Close All”);
run(“Collect Garbage”);

}

Exception message:

(Fiji Is Just) ImageJ 2.1.0/1.53c; Java 1.8.0_172 [64-bit]; Windows 10 10.0; 29394MB of 97675MB (30%)

java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at ij.macro.Functions.call(Functions.java:4515)

at ij.macro.Functions.getStringFunction(Functions.java:276)

at ij.macro.Interpreter.getStringTerm(Interpreter.java:1475)

at ij.macro.Interpreter.getString(Interpreter.java:1453)

at ij.macro.Interpreter.doStatement(Interpreter.java:333)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doIf(Interpreter.java:1090)

at ij.macro.Interpreter.doStatement(Interpreter.java:299)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doFor(Interpreter.java:634)

at ij.macro.Interpreter.doStatement(Interpreter.java:305)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doIf(Interpreter.java:1090)

at ij.macro.Interpreter.doStatement(Interpreter.java:299)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doFor(Interpreter.java:634)

at ij.macro.Interpreter.doStatement(Interpreter.java:305)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doIf(Interpreter.java:1090)

at ij.macro.Interpreter.doStatement(Interpreter.java:299)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doFor(Interpreter.java:634)

at ij.macro.Interpreter.doStatement(Interpreter.java:305)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doIf(Interpreter.java:1090)

at ij.macro.Interpreter.doStatement(Interpreter.java:299)

at ij.macro.Interpreter.doBlock(Interpreter.java:712)

at ij.macro.Interpreter.doStatement(Interpreter.java:323)

at ij.macro.Interpreter.doFor(Interpreter.java:634)

at ij.macro.Interpreter.doStatement(Interpreter.java:305)

at ij.macro.Interpreter.doStatements(Interpreter.java:264)

at ij.macro.Interpreter.run(Interpreter.java:160)

at ij.macro.Interpreter.run(Interpreter.java:93)

at ij.macro.Interpreter.run(Interpreter.java:104)

at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)

at ij.IJ.runMacro(IJ.java:153)

at ij.IJ.runMacro(IJ.java:142)

at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148)

at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144)

at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095)

at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144)

at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145)

at org.scijava.script.ScriptModule.run(ScriptModule.java:157)

at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)

at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)

at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)

at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException

at trainableSegmentation.Weka_Segmentation$CustomWindow.setButtonsEnabled(Weka_Segmentation.java:1153)

at trainableSegmentation.Weka_Segmentation.getProbability(Weka_Segmentation.java:2677)

… 61 more

Console error report:

java.lang.NullPointerException
hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
trainableSegmentation.WekaSegmentation$8.call(WekaSegmentation.java:7021)
trainableSegmentation.WekaSegmentation$8.call(WekaSegmentation.java:6973)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

at hr.irb.fastRandomForest.FastRandomForest.distributionForInstance(FastRandomForest.java:646)
at trainableSegmentation.WekaSegmentation$8.call(WekaSegmentation.java:7021)
at trainableSegmentation.WekaSegmentation$8.call(WekaSegmentation.java:6973)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException
trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6923)
trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6361)
trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6288)
trainableSegmentation.Weka_Segmentation.getProbability(Weka_Segmentation.java:2678)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
ij.macro.Functions.getStringFunction(Functions.java:276)
ij.macro.Interpreter.getStringTerm(Interpreter.java:1475)
ij.macro.Interpreter.getString(Interpreter.java:1453)
ij.macro.Interpreter.doStatement(Interpreter.java:333)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doIf(Interpreter.java:1090)
ij.macro.Interpreter.doStatement(Interpreter.java:299)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doFor(Interpreter.java:634)
ij.macro.Interpreter.doStatement(Interpreter.java:305)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doIf(Interpreter.java:1090)
ij.macro.Interpreter.doStatement(Interpreter.java:299)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doFor(Interpreter.java:634)
ij.macro.Interpreter.doStatement(Interpreter.java:305)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doIf(Interpreter.java:1090)
ij.macro.Interpreter.doStatement(Interpreter.java:299)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doFor(Interpreter.java:634)
ij.macro.Interpreter.doStatement(Interpreter.java:305)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doIf(Interpreter.java:1090)
ij.macro.Interpreter.doStatement(Interpreter.java:299)
ij.macro.Interpreter.doBlock(Interpreter.java:712)
ij.macro.Interpreter.doStatement(Interpreter.java:323)
ij.macro.Interpreter.doFor(Interpreter.java:634)
ij.macro.Interpreter.doStatement(Interpreter.java:305)
ij.macro.Interpreter.doStatements(Interpreter.java:264)
ij.macro.Interpreter.run(Interpreter.java:160)
ij.macro.Interpreter.run(Interpreter.java:93)
ij.macro.Interpreter.run(Interpreter.java:104)
ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
ij.IJ.runMacro(IJ.java:153)
ij.IJ.runMacro(IJ.java:142)
net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148)
net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144)
net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095)
net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144)
net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145)
org.scijava.script.ScriptModule.run(ScriptModule.java:157)
org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6923)
at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6361)
at trainableSegmentation.WekaSegmentation.applyClassifier(WekaSegmentation.java:6288)
at trainableSegmentation.Weka_Segmentation.getProbability(Weka_Segmentation.java:2678)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at ij.macro.Functions.call(Functions.java:4515)
at ij.macro.Functions.getStringFunction(Functions.java:276)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1475)
at ij.macro.Interpreter.getString(Interpreter.java:1453)
at ij.macro.Interpreter.doStatement(Interpreter.java:333)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doIf(Interpreter.java:1090)
at ij.macro.Interpreter.doStatement(Interpreter.java:299)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doFor(Interpreter.java:634)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doIf(Interpreter.java:1090)
at ij.macro.Interpreter.doStatement(Interpreter.java:299)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doFor(Interpreter.java:634)
at ij.macro.Interpreter.doStatement(Interpreter.java:305)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doIf(Interpreter.java:1090)
at ij.macro.Interpreter.doStatement(Interpreter.java:299)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doFor(Interpreter.java:634)
at ij.macro.Interpreter.doStatement(Interpreter.java:305)
at ij.macro.Interpreter.doBlock(Interpreter.java:712)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doIf(Interpreter.java:1090)
at ij.macro.Interpreter.doStatement(Interpreter.java:299)
at ij.macro.Interpreter.doStatement(Interpreter.java:323)
at ij.macro.Interpreter.doFor(Interpreter.java:634)
at ij.macro.Interpreter.doStatement(Interpreter.java:305)
at ij.macro.Interpreter.doStatements(Interpreter.java:264)
at ij.macro.Interpreter.run(Interpreter.java:160)
at ij.macro.Interpreter.run(Interpreter.java:93)
at ij.macro.Interpreter.run(Interpreter.java:104)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:153)
at ij.IJ.runMacro(IJ.java:142)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145)
at org.scijava.script.ScriptModule.run(ScriptModule.java:157)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)