Predict.py issue on cluster

Hi,

I’m creating a pipeline in Windows using the GUI (Windows 10 64-bit) and then utilising the CreateBatchFiles module to export to a cluster running Centos 6.

Previously, I’ve managed to get this working without issue. However now I wish to use the PixelClassifier function from Ilastik, utilising the predict.py plugin module. I think I installed Ilastik correctly from the instructions on their website. However, once the pipeline reaches the predict module it throws up the following error:

JVM will be started with AWT in headless mode
Creating JVM object
Signalling caller
Enabled Bio-formats directory cacheing
Created temporary file /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsgEGDZ2.hdf5
/home/nel56/.conda/envs/cellprofiler_edv1_1/bin/cellprofiler: (10 ): sys.exit(main())
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/main.py: (171 main): exit_code = run_pipeline_headless(options, args)
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/main.py: (683 run_pipeline_headless): initial_measurements = cellprofiler.measurement.load_measurements(options.pipeline_filename, image_numbers=ima
ge_set_numbers)
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/measurement.py: (1853 load_measurements): image_numbers=image_numbers)
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/measurement.py: (271 init): for frame in traceback.extract_stack():
HDF5Dict.init(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsgEGDZ2.hdf5, temporary=True, copy=<HDF5 group “/Measurements/2019-10-22-16-10-21” (2 members)>, mode=w
backend Qt5Agg version 5.9.2
Could not load runimagej
Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/modules/init.py”, line 325, in add_module
m = import(mod, globals(), locals(), [‘all’], 0)
File “/home/nel56/WORKING_DATA/CellProfiler/CellProfiler-plugins/runimagej.py”, line 11, in
import imagej
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/imagej/init.py”, line 1, in
from .imagej import *
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/imagej/imagej.py”, line 10, in
import jnius_config
ImportError: No module named jnius_config
Could not load measureimagefocus
Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/modules/init.py”, line 325, in add_module
m = import(mod, globals(), locals(), [‘all’], 0)
File “/home/nel56/WORKING_DATA/CellProfiler/CellProfiler-plugins/measureimagefocus.py”, line 8, in
import microscopeimagequality.miq
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/microscopeimagequality/miq.py”, line 14, in
import tensorflow.contrib.slim
ImportError: No module named contrib.slim
Using TensorFlow backend.
could not load these modules: runimagej,measureimagefocus
Created temporary file /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5
/home/nel56/.conda/envs/cellprofiler_edv1_1/bin/cellprofiler: (10 ): sys.exit(main())
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/main.py: (171 main): exit_code = run_pipeline_headless(options, args)
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/main.py: (753 run_pipeline_headless): initial_measurements=initial_measurements
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/pipeline.py: (1570 run): copy=initial_measurements)
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/measurement.py: (271 init): for frame in traceback.extract_stack():
HDF5Dict.init(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5, temporary=True, copy=<HDF5 group “/Measurements/2019-10-22-16-10-37” (2 members)>, mode=w
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/utilities/hdf5_dict.py:539: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it
will be treated as np.int64 == np.dtype(int).type.
np.issubdtype(hdf5_type, int) or
/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/utilities/hdf5_dict.py:541: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it wi
ll be treated as np.float64 == np.dtype(float).type.
hdf5_type_is_float = np.issubdtype(hdf5_type, float)
HDF5Dict.flush(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5, temporary=True
Set MySQL transaction isolation to “READ COMMITTED”: 0L
HDF5Dict.flush(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5, temporary=True
Times reported are CPU and Wall-clock times for each module
Tue Oct 22 16:20:43 2019: Image # 1, module Images # 1: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Tue Oct 22 16:20:43 2019: Image # 1, module Metadata # 2: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Getting image reader for: CyQ, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF488_ORG.tif
Falling back to Java reader.
Getting image reader for: None, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF488_ORG.tif
Getting image reader for: AV, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF568_ORG.tif
Falling back to Java reader.
Getting image reader for: None, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF568_ORG.tif
Tue Oct 22 16:20:43 2019: Image # 1, module NamesAndTypes # 3: CPU_time = 3.77 secs, Wall_time = 1.50 secs
Tue Oct 22 16:20:45 2019: Image # 1, module Groups # 4: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Getting image reader for: CyQ, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF488_ORG.tif
Tue Oct 22 16:20:45 2019: Image # 1, module ImageMath # 5: CPU_time = 0.90 secs, Wall_time = 0.24 secs
Getting image reader for: AV, None, file:/home/nel56/WORKING_DATA/CellProfiler/ACTIVE_RUN/input/1105-Experiment-781-Stitching-01-Scene-001-C3-C03-Image%20Export-01_AF568_ORG.tif
Tue Oct 22 16:20:45 2019: Image # 1, module ImageMath # 6: CPU_time = 1.02 secs, Wall_time = 0.20 secs
Warning: Ignoring your non-empty LD_LIBRARY_PATH
2019-10-22 16:20:58,367 main-ERROR: An unexpected error occurred. Your installation might be corrupt!
Traceback (most recent call last):
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/ilastik_install/cli.py”, line 114, in
main()
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/ilastik_install/cli.py”, line 108, in main
args.root / “conda-meta”, args.root, prefix_config.prefix, args.root
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/ilastik_install/core.py”, line 58, in replace_prefixes
mode,
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/ilastik_install/external/_constructor.py”, line 148, in update_prefix
new_prefix.encode(“utf-8”),
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/ilastik_install/external/_constructor.py”, line 118, in binary_replace
assert new_placeholder in res
AssertionError
Traceback (most recent call last):
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/ilastik/ilastik.py”, line 134, in
main()
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/ilastik/ilastik.py”, line 128, in main
hShell = ilastik_main.main(parsed_args, workflow_cmdline_args)
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/ilastik/ilastik_main.py”, line 132, in main
load_fn = _prepare_auto_open_project(parsed_args)
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/ilastik/ilastik_main.py”, line 326, in _prepare_auto_open_project
from lazyflow.utility.pathHelpers import PathComponents, isUrl
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/lazyflow/lazyflow/init.py”, line 31, in
from . import utility
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/lazyflow/lazyflow/utility/init.py”, line 25, in
from .memory import Memory
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/ilastik-meta/lazyflow/lazyflow/utility/memory.py”, line 27, in
import psutil
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/psutil/init.py”, line 95, in
from . import _pslinux as _psplatform
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/site-packages/psutil/_pslinux.py”, line 9, in
import base64
File “/sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/base64.py”, line 11, in
import binascii
ImportError: /sharedlustre/users/nel56/CellProfiler/Ilastik/ilastik-1.3.3-Linux/lib/python3.7/lib-dynload/binascii.cpython-37m-x86_64-linux-gnu.so: file too short
Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/logging/init.py”, line 868, in emit
msg = self.format(record)
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/logging/init.py”, line 741, in format
return fmt.format(record)
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/logging/init.py”, line 465, in format
record.message = record.getMessage()
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/logging/init.py”, line 329, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file predict.py, line 189
Error detected during run of module Predict
Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 1782, in run_with_yield
self.run_module(module, workspace)
File “/home/nel56/.conda/envs/cellprofiler_edv1_1/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 2034, in run_module
module.run(workspace)
File “/home/nel56/WORKING_DATA/CellProfiler/CellProfiler-plugins/predict.py”, line 191, in run
raise cpe
CalledProcessError: Command ‘[u’/home/nel56/WORKING_DATA/CellProfiler/Ilastik/ilastik-1.3.3-Linux/run_ilastik.sh’, ‘–headless’, ‘–project’, u’/home/nel56/WORKING_DATA/CellProfiler/Ilastik/ilastik-1.3.3-Linux/IlastikAF488.ilp’, ‘–outpu t_format’, ‘hdf5’, ‘–export_source’, ‘Probabilities’, ‘–output_filename_format’, ‘/localscratch/user_scratch_files/1437355.1.all.q/tmpjwSeUY.h5’, ‘/localscratch/user_scratch_files/1437355.1.all.q/tmpB2taum.h5’]’ returned non-zero exit
status 1
Tue Oct 22 16:20:45 2019: Image # 1, module Predict # 8: CPU_time = 3.20 secs, Wall_time = 42.69 secs
HDF5Dict.flush(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5, temporary=True
HDF5Dict.del(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsJQqaYF.hdf5, temporary=True
HDF5Dict.del(): /localscratch/user_scratch_files/1437355.1.all.q/CpmeasurementsgEGDZ2.hdf5, temporary=True
Closing reader <bioformats.formatreader.ImageReader object at 0x2af6bf7a0050>
Closing reader <bioformats.formatreader.ImageReader object at 0x2af6bf7a0390>

Sorry that’s quite a lot. Does anyone have any ideas where this error is coming from or why? Or suggest a workaround to use the pixel classification functionality from ilastik in my cellprofiler pipeline on the cluster?

My pipe line is here: Ilastik_Lynsey_CLuster_1_error.cppipe (40.6 KB)

Cheers,

Ed

Hi @edward.law,

I dont know whether this is a copy paste error, or a genuine problem, but in your output this line popped into my attention:

CalledProcessError: Command ‘[u’/home/nel56/WORKING_DATA/CellProfiler/Ilastik/ilastik-1.3.3-Linux/run_ilastik.sh’, ‘–headless’, ‘–project’, u’/home/nel56/WORKING_DATA/CellProfiler/Ilastik/ilastik-1.3.3-Linux/IlastikAF488.ilp’, ‘–outpu t_format’, ‘hdf5’, ‘–export_source’, ‘Probabilities’, ‘–output_filename_format’, ‘/localscratch/user_scratch_files/1437355.1.all.q/tmpjwSeUY.h5’, ‘/localscratch/user_scratch_files/1437355.1.all.q/tmpB2taum.h5’]’ returned non-zero exit
status 1

By this line I can say that ilastik didn’t like the supplied command line arguments. The ‘–outpu t_format’, should not contain a space (there is one after between “outpu” and “t_format”. But this might be a copy paste issue. In order to get closer to the problem you could maybe look at the ilastik log file, which should be in your user home and have the name ilastik_log.txt. Maybe delete the file first, and then repeat whatever you have done to produce that error and then look at the file. Maybe you can post the contents then here.
Cheers!

Hi @k-dominik,

Turns out it was a simple installation corruption issue. I reinstalled ilastik on the cluster and it works without problem.

I should say I found it’s important to feed the Predict module with the path to the “run_ilastik.sh” script on the cluster, rather than asking CellProfiler to remap from a local mapping to cluster mapping in the CreateBatchFiles module. I think CreateBatchFiles will only perform mappings for the “input” and “output” folders, but I may be wrong. Might be nice to have that feature available.

Cheers,

Ed