CellProfiler does not exit when running headless

We’re running headless CellProfiler on our Torque/PBS cluster with a command similar to

cellprofiler -c -r -p pipeline_file.cppipe --data-file=data_file.csv

When the pipeline is complete after running correctly, CellProfiler spits out the following error:

Traceback (most recent call last):
File “/data/aics/software/anaconda2/envs/cellprofiler/bin/cellprofiler”, line 9, in
load_entry_point(‘CellProfiler’, ‘console_scripts’, ‘cellprofiler’)()
File “/data/aics/software/CellProfiler/cellprofiler/main.py”, line 146, in main
run_pipeline_headless(options, args)
File “/data/aics/software/CellProfiler/cellprofiler/main.py”, line 706, in run_pipeline_headless
return exit_code
UnboundLocalError: local variable ‘exit_code’ referenced before assignment

and hangs indefinitely. This seems like a similar problem to what @alextruong described in this thread. Furthermore, given erroneous input, such as

cellprofiler -c -r -p this_file_does_not_exist.cppipe --data-file=data_file.csv

similar behavior is demonstrated, albeit with a different error. The error is as follows:

Traceback (most recent call last):
  File "/data/aics/software/anaconda2/envs/cellprofiler/bin/cellprofiler", line 9, in <module>
    load_entry_point('CellProfiler', 'console_scripts', 'cellprofiler')()
  File "/data/aics/software/CellProfiler/cellprofiler/__main__.py", line 146, in main
    run_pipeline_headless(options, args)
  File "/data/aics/software/CellProfiler/cellprofiler/__main__.py", line 647, in run_pipeline_headless
    pipeline.load(options.pipeline_filename)
  File "/data/aics/software/CellProfiler/cellprofiler/pipeline.py", line 882, in load
    raise IOError("Could not find file, " + fd_or_filename)
IOError: Could not find file, /home/derekt/projects/cellprofiler_executor/cp_pipelines/czi2tif/czi2tif_.cppipe

but CellProfiler also hangs indefinitely.

We had a similar problem where the javabridge vm was not being terimated correctly, and caused Python to hang. It’s a long shot but this may be related.

Are there any workarounds to this?

If you’re able to update the version of CP on your cluster, a recent pull request should fix the first issue.

I’m not sure about the second one- I can see in the pipeline.py file where the error is being tripped but I’m not sure exactly which changes to make to have it behave correctly and terminate the process. Best to check with @allen_goodman here or on GitHub.

Hi Beth,

Thanks for your suggestion. We had issues compiling CellProfiler from source on our Linux cluster since it runs a pretty old version of CentOS. I did find a temporary work around for this issue. We added the --DONE_FILE flag when executing headless and that change seems to prevent the process from hanging.

1 Like