I am trying to run CellProfiler.exe from command line (from a .php file using the exec command). The pipeline contains some ImageJ calls.
I would like to run CP headless, but when I run CP using the -c command line flag there are two problems:
I am running CP 2.1.1 (rev d1aeb88) on Windows 7 (64 bit).
Let me start with the obvious (sorry if this is too obvious!). Did you include a switch like this, as noted by the Help?
–ij-plugins-directory=IJ_PLUGINS_DIRECTORY
(I think this works with the compiled version, but certainly with the developer’s version)
Cheers,
David
Hi David,
Thanks for pointing this out. I was not aware of this switch. I will try it.
/Petter
Hi David,
I have tried the following command:
“C:/Program Files/CellProfiler/CellProfiler.exe” --jvm-heap-size=8g -c -r -o D:/xampp/htdocs/LargeImageHandling/uploaded/petter/t2/output -p D:/xampp/htdocs/LargeImageHandling/uploaded/petter/t2/prtest140227_A_tmp.cppipe --plugins-directory=“C:/Program Files/CellProfiler/plugins” -i “D:/xampp/htdocs/LargeImageHandling”
It works fine from the command line, but not when I am Calling it using the exec function in php. In the latter case the plugins are not found.
Pettar - could you post the code you use for the exec function in PHP?
Hi Lee,
I have used php code like this:
Any news on the problem when running the command from php?
Another solution for me might be to run it showing the CP interface, but closing it when the pipeline finishes. Is there such an option?
Sorry Petter, I didn’t see this post. I think you have --plugins-directory when you should have --ij-plugins-directory on the command-line. Could you try that?
Hi Lee,
Thanks for pointing this out, but it still does not work!
I have tried the following command:
“C:/Program Files/CellProfiler/CellProfiler.exe” -d C:/tmp/cpdone.txt --jvm-heap-size=8g -c -r -o D:/xampp/htdocs/TissueMaps/uploaded/petter/t5/test -p D:/xampp/htdocs/TissueMaps/uploaded/petter/t5/prInsitu20140821_test_tmp.cppipe --ij-plugins-directory=“D:/xampp/htdocs/TissueMaps/CellProfilerPlugins” -i “D:/xampp/htdocs/TissueMaps”
The done file says “Failure”. If I run it without the -c it works fine.
If I run it from the command line I get this output. Does it make any sense to you?
C:\Program Files\CellProfiler>“C:/Program Files/CellProfiler/CellProfiler.exe”
-d C:/tmp/cpdone.txt --jvm-heap-size=8g -c -r -o D:/xampp/htdocs/TissueMaps/uplo
aded/petter/t5/test -p D:/xampp/htdocs/TissueMaps/uploaded/petter/t5/prInsitu201
40821_test_tmp.cppipe --ij-plugins-directory=“D:/xampp/htdocs/TissueMaps/CellPro
filerPlugins” -i “D:/xampp/htdocs/TissueMaps”
[INFO] Created service: org.scijava.log.StderrLogService
[INFO] Created service: org.scijava.thread.DefaultThreadService
[INFO] Created service: org.scijava.event.DefaultEventService
[INFO] Created service: org.cellprofiler.ijutils.CellProfilerAppEventService
[INFO] Found 487 plugins.
[INFO] Created service: org.scijava.plugin.DefaultPluginService
[INFO] Created service: org.scijava.object.DefaultObjectService
[INFO] Created service: imagej.module.DefaultModuleService
[INFO] Created service: imagej.command.DefaultCommandService
[INFO] Created service: imagej.script.DefaultScriptService
[INFO] Created service: io.scif.services.DefaultFormatService
[INFO] Found 11 DataType plugins.
[INFO] Created service: imagej.data.types.DefaultDataTypeService
[INFO] Created service: imagej.data.DefaultDatasetService
[INFO] Created service: imagej.display.DefaultDisplayService
[INFO] Created service: imagej.data.display.DefaultImageDisplayService
[INFO] Created service: imagej.data.lut.DefaultLUTService
[INFO] Created service: imagej.console.DefaultConsoleService
[INFO] Found 2 Platform plugins.
[INFO] Configuring platform: imagej.plugins.platforms.windows.WindowsPlatform
[INFO] Configuring platform: imagej.platform.DefaultPlatform
[INFO] Created service: imagej.platform.DefaultPlatformService
[INFO] Created service: imagej.io.DefaultRecentFileService
[INFO] Created service: imagej.io.DefaultIOService
[INFO] Found 13 InputWidget plugins.
[INFO] Created service: imagej.widget.DefaultWidgetService
[INFO] Created service: imagej.text.DefaultTextService
[INFO] Created service: imagej.menu.DefaultMenuService
[INFO] Found 3 applications.
[INFO] Created service: org.scijava.app.DefaultAppService
[INFO] Created service: org.scijava.app.DefaultStatusService
[INFO] Created service: imagej.tool.DefaultToolService
[INFO] Created service: imagej.options.DefaultOptionsService
[INFO] Created service: org.scijava.event.DefaultEventHistory
[INFO] Created service: imagej.data.animation.DefaultAnimationService
[INFO] Created service: imagej.data.operator.DefaultCalculatorService
[INFO] Created service: imagej.ui.common.awt.AWTRenderingService
[INFO] Created service: imagej.data.display.DefaultOverlayService
[INFO] Created service: imagej.data.threshold.DefaultThresholdService
[INFO] Created service: imagej.data.autoscale.DefaultAutoscaleService
[INFO] Created service: imagej.data.display.DefaultInputService
[INFO] Created service: imagej.data.display.DefaultZoomService
[INFO] Created service: imagej.data.display.DefaultWindowService
[INFO] Created service: imagej.data.DefaultImgPlusService
[INFO] Created service: imagej.data.measure.DefaultStatisticsService
[INFO] Created service: imagej.data.measure.DefaultMeasurementService
[INFO] Created service: imagej.data.sampler.DefaultSamplerService
[INFO] Created service: net.imglib2.meta.units.DefaultUnitService
[INFO] Created service: io.scif.codec.DefaultCodecService
[INFO] Created service: io.scif.DefaultMetadataService
[INFO] Created service: io.scif.formats.qt.DefaultQTJavaService
[INFO] Created service: io.scif.formats.tiff.DefaultTiffService
[INFO] Created service: io.scif.gui.DefaultGUIService
[INFO] Created service: io.scif.refs.DefaultRefManagerService
[INFO] Created service: io.scif.img.cell.cache.MapDBCache
[INFO] Created service: io.scif.img.converters.DefaultPlaneConverterService
[INFO] Created service: io.scif.img.DefaultImgUtilityService
[INFO] Created service: io.scif.io.DefaultNIOService
[INFO] Created service: io.scif.services.DefaultFilePatternService
[INFO] Created service: io.scif.services.DefaultLocationService
[INFO] Created service: io.scif.services.DefaultTranslatorService
[INFO] Created service: io.scif.services.JAIIIOServiceImpl
[INFO] Created service: io.scif.xml.DefaultXMLService
[INFO] Created service: imagej.ui.DefaultUIService
[INFO] Created service: imagej.ui.dnd.DefaultDragAndDropService
[INFO] Created service: imagej.updater.core.DefaultUploaderService
[INFO] Created service: imagej.plugins.commands.restructure.SplitChannelsContext
Monitor
[INFO] Created service: imagej.plugins.commands.app.CoreAppEventService
[WARNING] Ignoring non-existent scripts directory: C:\Program Files\CellProfiler
\imagej\scripts
[INFO] Found 0 scripts
[INFO] Found 400 legacy plugins (plus 53 ignored).
[INFO] Created service: imagej.legacy.DefaultLegacyService
[INFO] Created service: imagej.plugins.uis.swing.SwingIconService
[INFO] Created service: imagej.plugins.uis.swing.overlay.JHotDrawService
[INFO] Created service: imagej.ui.common.awt.AWTScreenCaptureService
[INFO] Created service: imagej.render.DummyRenderingService
[INFO] Created service: imagej.data.display.DummyScreenCaptureService
[INFO] Created service: io.scif.services.DefaultInitializeService
[INFO] Found 6 ScriptLanguage plugins.
Setting Java temp dir to C:\Users\Petter\AppData\Local\Temp\multistackreg9075703
310927688491.tmp
[INFO] Discovered user interface: imagej.plugins.uis.swing.sdi.SwingUI
[INFO] Discovered user interface: imagej.plugins.uis.swing.mdi.SwingMdiUI
[INFO] Launching user interface: imagej.plugins.uis.swing.sdi.SwingUI
Exception in thread “Thread-0” java.util.concurrent.ExecutionException: org.mozi
lla.javascript.WrappedException: Wrapped java.awt.HeadlessException (#3)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
Caused by: org.mozilla.javascript.WrappedException: Wrapped java.awt.HeadlessExc
eption (#3)
at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:17
54)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:22
5)
at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java
:85)
at org.mozilla.javascript.gen._java_python_bridge__14._c_anonymous_1(:3)
at org.mozilla.javascript.gen._java_python_bridge__14.call()
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:3
94)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:309
1)
at org.mozilla.javascript.gen._java_python_bridge__14.call()
at org.mozilla.javascript.Context$1.run(Context.java:478)
at org.mozilla.javascript.Context.call(Context.java:489)
at org.mozilla.javascript.Context.call(Context.java:476)
at org.mozilla.javascript.JavaAdapter.callMethod(JavaAdapter.java:573)
at adapter1.run()
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:44
1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
Caused by: java.awt.HeadlessException
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:1
59)
at java.awt.Window.(Window.java:432)
at java.awt.Frame.(Frame.java:403)
at javax.swing.JFrame.(JFrame.java:202)
at imagej.plugins.uis.swing.SwingApplicationFrame.(SwingApplicatio
nFrame.java:53)
at imagej.plugins.uis.swing.AbstractSwingUI.createUI(AbstractSwingUI.jav
a:165)
at imagej.ui.AbstractUserInterface.show(AbstractUserInterface.java:94)
at imagej.ui.DefaultUIService.showUI(DefaultUIService.java:187)
at imagej.ui.DefaultUIService.showUI(DefaultUIService.java:172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
… 15 more
stopping worker thread 0
stopping worker thread 1
stopping worker thread 2
stopping worker thread 3
stopping worker thread 4
stopping worker thread 5
stopping worker thread 6
stopping worker thread 7
stopping worker thread 8
Sorry I didn’t get around to answering this sooner. The problem is that you’re executing an ImageJ 1.0 macro in RunImageJ.
Caused by: java.awt.HeadlessException
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:1
59)
ImageJ 1.0 isn’t headless, so we have to start up the ImageJ UI to run the macro - perhaps there is some work-around that we could do to run the macro headless and hope that no AWT calls are made. The error occurs when we start the UI and AWT rejects our request because Java is headless. ImageJ 2.0 is designed to ensure that headless operations work, so for scripts other than macros and ImageJ 2.0 plugins, you can truly run headless.
There’s a trick we use here on our cluster to run ImageJ 1.0 macros. If you define the environment variable, “CELLPROFILER_USE_XVFB”, then CellProfiler itself will run in headless-style on the command-line, but it will start Java in non-headless mode so that UI can be done. On our Linux cluster, we use XVFB (http://en.wikipedia.org/wiki/Xvfb) to mimic the display. On Windows, you’ll get a lot of annoying windows popping up, but other than that, it should operate as if it were headless.
Let me know if defining CELLPROFILER_USE_XVFB works for you. If not, maybe we can put in something to disable ImageJ 1.0 UI display and we can see if the macros execute correctly.
We have discovered that CellProfiler runs much slower headless. I have a project that takes about 30 minutes headless, but only 6 minutes otherwise. When running headless only 4 out of 8 processors are doing any work, and none of them near 100%. Do you have any solution to this?
We run multiple jobs per machine here, usually one job per core. CellProfiler headless is designed to run more or less single-threaded with the idea that you can scale up predictably by running multiple instances. So the behavior you’re seeing is by design.
We saw the multiprocessing solution in the GUI as a way for people who don’t have much IT support to get faster processing without understanding the infrastructure. You’re almost certainly better off running an analysis via the GUI than you are running a single analysis headless - a good strategy is to run the CP GUI on a large machine and tune the number of workers appropriately to your pipeline via the preferences.
OK. This does not really fit how we designed our tool around CellProfiler including post analysis outside CellProfiler. One improvement for us would be if it was possible to close CellProfiler automatically after the pipeline has finished when running via the GUI.
I will try to modify our code to split the image sets and start multiple threads. This should work for us.
I have got it working now. I had to split the input and then merge the results afterwards.
Thanks for reporting back, and glad you got it working. Sorry for the hassle!
David