Need help with exiting Python Script

imagej
python
headless
scripting

#1

HI guys,

I have a python script that is running fine when called from an external application in headless mode. It is called using:

ImageJ-win64.exe --ij2 --headless --console --run c:\Users\Public\Documents\Fiji\scripts\script.py "JSONPARAMFILE='c:\Users\Test\Test.json'"

Since the external application is waiting for the python script to exit it is required that it really does this, but I juts do not know how to achieve this. When started from commandline it never exits as well. At the end of my python script is tried:

# finish and exit the script
os._exit()
#sys.exit()
#quit()
#raise SystemExit

But nothing worked so far. What else could I try or what am I doing wrong here? Here is the console output, where it stays forever …

c:\Users\Public\Documents\Fiji
λ ImageJ-win64.exe --ij2 --headless --console --run c:\Users\Public\Documents\Fiji\scripts\script.py "JSONPARAMFILE='c:\Users\Test\Test.json'"

c:\Users\Public\Documents\Fiji
λ Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
15:10:46,483 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback-test.xml] at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml]
15:10:46,484 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs multiple times on the classpath.
15:10:46,485 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/plugins/Mosaic_ToolSuite/MosaicSuite-1.0.8_Full.jar!/logback-test.xml]
15:10:46,485 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml]
15:10:46,512 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@41a6d121 - URL [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml] is not of type file
15:10:46,650 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
15:10:46,653 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
15:10:46,660 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
15:10:46,667 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
15:10:46,734 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to ERROR
15:10:46,734 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
15:10:46,735 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
15:10:46,736 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4f449e8f - Registering current configuration as safe fallback point

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[INFO] Overriding BIOP Run Macro...; identifier: command:ch.epfl.biop.macrorunner.B_Run_Macro; jar: file:/c:/Users/Public/DOCUME~1/Fiji/plugins/BIOP/B_Run_Macro-1.0.0-SNAPSHOT.jar
[INFO] Overriding Save Image As Tiff Without Prompt; identifier: script:ZenIntegration/Save_Image_As_Tiff_Without_Prompt.java; jar: file:/c:/Users/Public/DOCUME~1/Fiji/jars/scijava-common-2.65.0.jar
[INFO] Fiji Script Directory : c:\Users\Public\Documents\Fiji\scripts
[INFO] Filename                      : c:\Users\Test_raw_stitched.czi
[INFO] BinFactor                     : 4
[INFO] Filter Radius                 : 3
[INFO] Min. Partice Size             : 10000
[INFO] Min. Circularity              : 0.01
[INFO] Max. Circularity              : 0.99
[INFO] Threshold Method              : Triangle
[INFO] Threshold Background          : black
[INFO] Headless Mode                 : True
[INFO] Add Particles to ROI-Manger   : False
[INFO] Save Particles as Image       : True
[INFO] Save Format                   : ome.tiff
[INFO] Save Results                  : True
[INFO] ------------  START IMAGE ANALYSIS ------------
[INFO] Opening Image ...
[INFO] File Extension   : .czi
[INFO] Resolution Count : 1
[INFO] SeriesCount      : 2
[INFO] Getting Image Series : 0
[INFO] Start Processing ...
[INFO] Apply Binning ...
[INFO] Apply Filter ...
[INFO] Convert to 8-bit ...
[INFO] Apply Threshold ...
        Label   Area    BX      BY      Width   Height  Slice
1       Test_raw_stitched.czi      7452239.59      27976.80        1573.58 3238.65 3513.11 1
2       Test_raw_stitched.czi      6170972.64      12277.59        1701.66 3147.16 2598.24 1
3       Test_raw_stitched.czi      6361806.76      19267.22        1848.04 3183.76 2653.13 1
4       Test_raw_stitched.czi      4634590.62      2689.73 2579.94 2689.73 2250.59 1
5       Test_raw_stitched.czi      6508782.51      19578.27        10155.08        3238.65 2653.13 1
6       Test_raw_stitched.czi      7581805.91      27629.15        10246.57        3476.52 3019.08 1
7       Test_raw_stitched.czi      5577043.31      11161.44        11015.07        2945.89 2488.45 1
8       Test_raw_stitched.czi      4201363.70      3220.35 11051.66        2488.45 2232.29 1
[INFO] Saving Processed Image to : c:\Users\Test_raw_stitched_PA.ome.tiff
[INFO] Save Results to: c:\Users\Test_raw_stitched_RESULTS.txt
[INFO] Updated Metadata and writing JSON to: c:\Users\zenoutput.json
[INFO] Done.

#2

HI guys,

I still did not find a solution for this… It is really blocking my workflow and it would be really cool if somebody could at least point me into the right direction.

Sebi


#3

Dear @sebi06,

I agree that the behavior you are describing seems undesirable. When an IJ instance is created to execute a script, this instance should be properly closed once the script has been executed. For the time being, you could check if the following does the trick for you:

from java.lang import System;

System.exit(0);

Best,
Stefan


#4

Hi Stefan,

I tried it out but

System.exit(42)

does not work either. The Commandlline, which open up when I start FIji in headless mode just stays open. The script does what it should do. When I start the script in headless mode from the commandline the script also does not exit automatically. I always have to press a key until the script “returns” … It looks like this:

c:\Users\Public\Documents\Fiji
λ ImageJ-win64.exe --ij2 --headless --console --run c:\Users\Public\Documents\Fiji\scripts\GuidedAcq_fromZEN.py "JSONPARAMFILE='c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched.json'"

c:\Users\Public\Documents\Fiji
λ Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
08:16:24,453 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback-test.xml] at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml]
08:16:24,455 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs multiple times on the classpath.
08:16:24,455 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/plugins/Mosaic_ToolSuite/MosaicSuite-1.0.8_Full.jar!/logback-test.xml]
08:16:24,455 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml]
08:16:24,485 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@41a6d121 - URL [jar:file:/c:/Users/Public/DOCUME~1/Fiji/jars/mitobo-1.8.4.jar!/logback-test.xml] is not of type file
08:16:24,707 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
08:16:24,713 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
08:16:24,727 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
08:16:24,739 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
08:16:24,850 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to ERROR
08:16:24,850 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
08:16:24,851 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:16:24,852 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4f449e8f - Registering current configuration as safe fallback point

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[INFO] Overriding BIOP Run Macro...; identifier: command:ch.epfl.biop.macrorunner.B_Run_Macro; jar: file:/c:/Users/Public/DOCUME~1/Fiji/plugins/BIOP/B_Run_Macro-1.0.0-SNAPSHOT.jar
[INFO] Overriding Save Image As Tiff Without Prompt; identifier: script:ZenIntegration/Save_Image_As_Tiff_Without_Prompt.java; jar: file:/c:/Users/Public/DOCUME~1/Fiji/jars/scijava-common-2.65.0.jar

[INFO] Fiji Script Directory : c:\Users\Public\Documents\Fiji\scripts
[INFO] Filename                      : c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched.czi
[INFO] BinFactor                     : 4
[INFO] Filter Radius                 : 3
[INFO] Min. Partice Size             : 10000
[INFO] Min. Circularity              : 0.01
[INFO] Max. Circularity              : 0.99
[INFO] Threshold Method              : Triangle
[INFO] Threshold Background          : black
[INFO] Headless Mode                 : True
[INFO] Add Particles to ROI-Manger   : False
[INFO] Save Particles as Image       : True
[INFO] Save Format                   : ome.tiff
[INFO] Save Results                  : True
[INFO] ------------  START IMAGE ANALYSIS ------------
[INFO] Opening Image ...
[INFO] File Extension   : .czi
[INFO] Resolution Count : 1
[INFO] SeriesCount      : 2
[INFO] Getting Image Series : 0
[INFO] Start Processing ...
[INFO] Apply Binning ...
[INFO] Apply Filter ...
[INFO] Convert to 8-bit ...
[INFO] Apply Threshold ...
        Label   Area    BX      BY      Width   Height  Slice
1       OverViewScan_Test_raw_stitched.czi      7452239.59      27976.80        1573.58 3238.65 3513.11 1
2       OverViewScan_Test_raw_stitched.czi      6170972.64      12277.59        1701.66 3147.16 2598.24 1
3       OverViewScan_Test_raw_stitched.czi      6361806.76      19267.22        1848.04 3183.76 2653.13 1
4       OverViewScan_Test_raw_stitched.czi      4634590.62      2689.73 2579.94 2689.73 2250.59 1
5       OverViewScan_Test_raw_stitched.czi      6508782.51      19578.27        10155.08        3238.65 2653.13 1
6       OverViewScan_Test_raw_stitched.czi      7581805.91      27629.15        10246.57        3476.52 3019.08 1
7       OverViewScan_Test_raw_stitched.czi      5577043.31      11161.44        11015.07        2945.89 2488.45 1
8       OverViewScan_Test_raw_stitched.czi      4201363.70      3220.35 11051.66        2488.45 2232.29 1
[INFO] Saving Processed Image to : c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched_PA.ome.tiff
[INFO] Save Results to: c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched_RESULTS.txt
[INFO] Updated Metadata and writing JSON to: c:\Output\Guided_Acquisition\zenoutput.json
[INFO] Done.

And here I have to press a key to get to:

...
[INFO] Saving Processed Image to : c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched_PA.ome.tiff
[INFO] Save Results to: c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched_RESULTS.txt
[INFO] Updated Metadata and writing JSON to: c:\Output\Guided_Acquisition\zenoutput.json
[INFO] Done.

c:\Users\Public\Documents\Fiji
λ

I think this is exactly the same thing when I start the Fiji headless script from within my application. Here I use the following code to start Fiji in headless (which works fine):

# start Fiji script in headless mode
app = Process()
app.StartInfo.FileName = IMAGEJ
app.StartInfo.Arguments = option
app.Start()
# wait until the script is finished
app.WaitForExit()
excode = app.ExitCode

print('Exit Code: ', excode)
print('Done.')

In that case I can see from the scrip log output, that it “arrived” at the exit call.

When I wait for about 30-40s, the the script really exits and my application return the correct exit code 42 provided by the Fiji python script.

So it looks like the script really “exited”. This “waiting” does not help if I start the script directly from CMD … Also it makes not difference, what exit command i use. The behavior is the same when I use

os._exit(42)

at the end of the python script running in headless mode.

Conclusion: I am still confused about what is going on here. My expectation would be that the Fiji scripts right after calling the respective command. And that should also close the CMD window automatically. Could this be due what I do within my script? That does not sound plausible to me, but …


#5

I only now realized that you are on Windows! Could you please add --dry-run to the aforementioned command line invocation and replace javaw with java in the reported invocation? This has solved the issue for me…


#6

Hi Steffen,

I did not use javaw in my call from the commandline, so I am really sure where I should replace it. Or maybe I misundertsood you here?

And you are right, all of this is done on Windows.


#7

If you run the following command

ImageJ-win64.exe --dry-run --ij2 --headless --console --run c:\Users\Public\Documents\Fiji\scripts\GuidedAcq_fromZEN.py "JSONPARAMFILE='c:\Output\Guided_Acquisition\OverViewScan_Test_raw_stitched.json'"

ImageJ will (thanks to --dry-run) output the call to the javaw executable that ultimately starts up ImageJ. In that output, you should replace javaw with java.

Best,
Stefan


#8

Hi Steffen,

java -Dpython.cachedir.skip=true -Dplugins.dir=c:\\Users\\Public\\DOCUME~1\\Fiji -Xmx12204m -Djava.awt.headless=true -Dapple.awt.UIElement=true -Xincgc -XX:PermSize=128m -Djava.class.path=c:\\Users\\Public\\DOCUME~1\\Fiji/jars/imagej-launcher-4.0.5.jar -Dimagej.dir=c:\\Users\\Public\\DOCUME~1\\Fiji -Dij.dir=c:\\Users\\Public\\DOCUME~1\\Fiji -Dfiji.dir=c:\\Users\\Public\\DOCUME~1\\Fiji -Dfiji.defaultLibPath=bin/server/jvm.dll -Dfiji.executable=c:\\Users\\Public\\Documents\\Fiji\\ImageJ-win64.exe -Dij.executable=c:\\Users\\Public\\Documents\\Fiji\\ImageJ-win64.exe -Djava.library.path=c:\\Users\\Public\\DOCUME~1\\Fiji/lib/win64;c:\\Users\\Public\\DOCUME~1\\Fiji/mm/win64;c:\\Users\\Public\\DOCUME~1\\Fiji/lib/fftw;c:\\Users\\Public\\DOCUME~1\\Fiji/lib/jcuda-6.5 -Dscijava.context.strict=false net.imagej.launcher.ClassLauncher -ijjarpath jars -ijjarpath plugins net.imagej.Main --run c:\\Users\\Public\\Documents\\Fiji\\scripts\\GuidedAcq_fromZEN.py JSONPARAMFILE='c:\\Output\\Guided_Acquisition\\OverViewScan_Test_raw_stitched.json'

Yes, that works. But I need to use the original “short” command line version to be able to start Fiji in headless mode from within my application.
Otherwise I would have to build up the output from --dry-run programmatically … Thiss might be doable, but scares me a bit :-).

Is there any way to permanently change javaw to java. Can that be changes inside a configuration file or something similar?


#9

Dear @sebi06,

I have created an issue in the imagej-launcher project to track your suggestion:

https://github.com/imagej/imagej-launcher/issues/47

I am not entirely sure about your use case, but you should be able to only change the --run ... part at the end of the invocation and keep the rest fixed (under the assumption that your folder configuration is consistent between machines). I fear that’s your only chance for now, since the fallback ImageJ.sh script requires a Cygwin environment on Windows. Plus, it doesn’t seem to support --run (but I might be wrong here).

Best,
Stefan


#10

Hi Stefan,

I already figured out how the “manipulate” that massive string from within my application before using it to call Fiji in headless mode. It works, but this is clearly a “hack”.

The use case is to start Fiji in headless mode from within a ZEN Blue python script to use it for “fancy” image analysis tasks and read back the results back into ZEN as soon as Fiji is finished.
One application for that would be GuidedAcquistion (other people call it Adaptive Feedback Microscopy) amd that works very well.

In case our own image analysis cannot “do it”, i use Python-Power and Fiji to solve the problem. Finally the respective lines inside the ZEN python script are:

# start Fiji script in headless mode
app = Process()
app.StartInfo.FileName = "java"
app.StartInfo.Arguments = fijistr
app.Start()
# wait until the script is finished
app.WaitForExit()
excode = app.ExitCode
print('Exit Code: ', excode)

And fijistr is rally long … :-). Basically I created an dedicated function to “compose” it based on the output of the --dry-run.

Thanks for filing the request on GitHub.

Sebi


#11

I had the same issue recently (strange that I did not find your post earlier).
By removing the --console argument, the external program managed to detect that the script execution was over.
However, I did not do it myself rather some collaborators, so I don´t know if you still get the print statements to the console redirected to your external program (if it is not the case then right the output to a file)