Cellprofiler pipeline error when running in Jupyter Lab

Hello,

I created a pipeline using the CellProfiler 3.0.0 app on my Mac OS X. It runs fine on a sample set of images and produces the desired output. I then exported the pipeline file and copied it over to a Conda environment on a Ubuntu machine that has cellprofiler installed. The pip freeze and java versions are given below. Whenever I try to run the pipeline which ran fine on my Mac OS, I get the following error. What am I doing wrong? Any help will be appreciated. I am using the sample notebook cellprofiler_demo.ipynb at https://github.com/CellProfiler/notebooks and modifying it to run my pipeline.


AttributeError Traceback (most recent call last)
in ()
----> 1 workspace = run_pipeline(pipeline_filename, images)

in run_pipeline(pipeline_filename, image_dict)
32 )
33
—> 34 module.prepare_run(workspace)
35 module.run(workspace)
36 module.post_run(workspace)

/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/images.pyc in prepare_run(self, workspace)
322 expression = self.filter.value_text
323 env = J.get_env()
–> 324 ifcls = J.class_for_name(“org.cellprofiler.imageset.ImageFile”)
325 scls = env.find_class(“java/lang/String”)
326 iffilter = J.make_instance(

/home/jagadish/.local/lib/python2.7/site-packages/javabridge/jutil.pyc in class_for_name(classname, ldr)
1737 if ldr == “system”:
1738 ldr = static_call(‘java/lang/ClassLoader’, ‘getSystemClassLoader’,
-> 1739 ‘()Ljava/lang/ClassLoader;’)
1740 return static_call(‘java/lang/Class’, ‘forName’,
1741 ‘(Ljava/lang/String;ZLjava/lang/ClassLoader;)’

/home/jagadish/.local/lib/python2.7/site-packages/javabridge/jutil.pyc in static_call(class_name, method_name, sig, *args)
932 ‘’’
933 env = get_env()
–> 934 fn = make_static_call(class_name, method_name, sig)
935 args_sig = split_sig(sig[1:sig.find(’)’)])
936 ret_sig = sig[sig.find(’)’)+1:]

/home/jagadish/.local/lib/python2.7/site-packages/javabridge/jutil.pyc in make_static_call(class_name, method_name, sig)
903 ‘’’
904 env = get_env()
–> 905 klass = env.find_class(class_name)
906 if klass is None:
907 jexception = get_env().exception_occurred()

AttributeError: ‘NoneType’ object has no attribute ‘find_class’

pip freeze:

alabaster==0.7.12

appdirs==1.4.3

asn1crypto==1.3.0

attrs==19.3.0

Babel==2.8.0

backports-abc==0.5

backports.functools-lru-cache==1.6.1

backports.shutil-get-terminal-size==1.0.0

bleach==3.1.0

boto3==1.12.12

botocore==1.15.12

cellh5==1.3.0

CellProfiler==3.0.0rc3

centrosome==1.1.6

certifi==2019.11.28

cffi==1.13.2

chardet==3.0.4

cloudpickle==1.2.2

configparser==4.0.2

contextlib2==0.6.0.post1

cryptography==2.8

cycler==0.10.0

Cython==0.29.14

cytoolz==0.10.1

dask==1.2.2

decorator==4.4.1

defusedxml==0.6.0

deprecation==2.0.7

docutils==0.15.2

entrypoints==0.3

enum34==1.1.6

functools32==3.2.3.post2

future==0.18.2

futures==3.3.0

h5py==2.9.0

hmmlearn==0.2.2

idna==2.8

imagecodecs-lite==2019.12.3

imageio==2.6.1

imagesize==1.2.0

importlib-metadata==1.3.0

inflect==3.0.2

ipaddress==1.0.23

ipykernel==4.10.0

ipython==5.8.0

ipython-genutils==0.2.0

ipywidgets==7.5.1

javabridge==1.0.18

Jinja2==2.11.1

jmespath==0.9.4

joblib==0.14.1

jsonschema==3.2.0

jupyter==1.0.0

jupyter-client==5.3.4

jupyter-console==5.2.0

jupyter-core==4.6.1

jupyterlab==0.33.12

jupyterlab-launcher==0.11.2

kiwisolver==1.1.0

libtiff==0.4.2

linecache2==1.0.0

lxml==4.4.2

mahotas==1.4.9

MarkupSafe==1.1.1

matplotlib==2.2.3

mistune==0.8.4

mkl-fft==1.0.15

mkl-random==1.1.0

mkl-service==2.3.0

more-itertools==5.0.0

MySQL-python==1.2.5

mysqlclient==1.4.6

nbconvert==5.6.1

nbformat==4.4.0

networkx==2.2

notebook==5.4.1

numpy==1.16.0

olefile==0.46

packaging==20.1

pandas==0.24.2

pandocfilters==1.4.2

pathlib==1.0.1

pathlib2==2.3.5

pexpect==4.7.0

pickleshare==0.7.5

Pillow==6.2.1

prokaryote==2.4.1

prompt-toolkit==1.0.15

ptyprocess==0.6.0

pyamg==3.1.1

pycparser==2.19

Pygments==2.5.2

pyOpenSSL==19.1.0

pyparsing==2.4.6

pyrsistent==0.15.6

PySocks==1.7.1

python-bioformats==1.5.2

python-dateutil==2.8.1

pytz==2019.3

PyWavelets==1.0.3

pyzmq==15.3.0

qtconsole==4.6.0

raven==6.10.0

requests==2.22.0

s3transfer==0.3.0

scandir==1.10.0

scikit-image==0.14.2

scikit-learn==0.20.3

scipy==1.1.0

Send2Trash==1.5.0

simplegeneric==0.8.1

singledispatch==3.4.0.3

sip==4.19.13

six==1.13.0

snowballstemmer==2.0.0

Sphinx==1.8.5

sphinxcontrib-websupport==1.2.0

subprocess32==3.5.4

terminado==0.8.3

testpath==0.4.4

tifffile==2019.7.26.2

toolz==0.10.0

tornado==4.5.3

traceback2==1.4.0

traitlets==4.3.3

typing==3.7.4.1

unittest2==1.1.0

urllib3==1.25.7

wcwidth==0.1.8

webencodings==0.5.1

widgetsnbextension==3.5.1

wxPython==3.0.2.0

wxPython-common==3.0.2.0

zipp==0.6.0

java version:
openjdk version “1.8.0_112”
OpenJDK Runtime Environment (Zulu 8.19.0.1-linux64) (build 1.8.0_112-b16)
OpenJDK 64-Bit Server VM (Zulu 8.19.0.1-linux64) (build 25.112-b16, mixed mode)

Can you explain what your bigger picture goal is? There may be easier ways to accomplish it than trying to run the pipeline in the jupyter notebook.

@bcimini Thanks for your response. Hope you and everyone you know is safe and healthy!

I have a custom pipeline that takes in 4D (x,y,z,c) image stacks of live budding yeast cells from the Microscope and segments them to generate segmentation masks and cut out per-cell (x,y,z,c) mini stacks. I want to run these mini stacks through a measurement pipeline. Since my entire pipeline can be run by a scientist from a Jupyter notebook, I felt it would be most seamless to integrate it this way without having to move data, load stuff on a UI etc. Please advise if this is the best approach.

Thanks
Jagadish

@bcimini Just checking again if you had any thoughts. Thanks.

Ah, sorry, missed this the last time around!

I personally would probably just run CellProfiler with its usual shell flags using subprocess, rather than trying to bring the pipeline inside the notebook, but that’s just me.

Your packages all look fine, as does your java version.

In the notebook you linked, image loading is being handled separately by skimage and added to the workspace directly, which is presumably how they’re getting around having to deal with javabridge, which seems to be causing the issues you’re seeing. You can try to just load javabridge and figure out how to get it to nicely load stuff inside an Anaconda jupyter notebook; I’m afraid I don’t have specific advice to offer on that, though maybe others do?

If that doesn’t work, the subprocess solution still should.

Good luck! I’d definitely be interested in hearing an update on how this works out :slight_smile:

Thanks @bcimini. So, here is what I tried. I am attaching a simple pipeline file I created to measure object sizes of a bunch of binary masks. The Analyze Images works great on my Mac OS X Mojave with Cellprofiler 3.0.0. I exported the cppipe file to my Ubuntu machine and ran cellprofiler (v3.0.0rc3) from the command line as follows and run into the error below. My pipeline file as well a sample folder of masks is attached. Can you please advise what I am doing wrong? I already have run ‘sudo apt install python-mysqldb’ and the package exists already.

cellprofiler -p mask_measurement.cppipe -c -r -i <input_folder_with_binary_masks> -o <output_folder>

MySQL could not be loaded.
Traceback (most recent call last):
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 87, in
import MySQLdb
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/MySQLdb/init.py”, line 23, in
(version_info, _mysql.version_info))
ImportError: this is MySQLdb version (1, 2, 5, ‘final’, 1), but _mysql is version (1, 4, 6, ‘final’, 0)
/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/utilities/hdf5_dict.py:527: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type.
np.issubdtype(hdf5_type, int) or
/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/utilities/hdf5_dict.py:529: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
hdf5_type_is_float = np.issubdtype(hdf5_type, float)
/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/importlib/init.py:37: H5pyDeprecationWarning: The h5py.highlevel module is deprecated, code should import directly from h5py, e.g. ‘from h5py import File’.
import(name)
Times reported are CPU times for each module, not wall-clock time
Mon Apr 13 00:12:15 2020: Image # 1, module Images # 1: 0.00 sec
Mon Apr 13 00:12:15 2020: Image # 1, module Metadata # 2: 0.00 sec
Mon Apr 13 00:12:15 2020: Image # 1, module NamesAndTypes # 3: 0.47 sec
Mon Apr 13 00:12:15 2020: Image # 1, module Groups # 4: 0.00 sec
Error detected during run of module MeasureImageAreaOccupied
Traceback (most recent call last):
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 1763, in run_with_yield
self.run_module(module, workspace)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 2007, in run_module
module.run(workspace)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/measureimageareaoccupied.py”, line 281, in run
statistics += self.measure_images(op, workspace)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/measureimageareaoccupied.py”, line 363, in measure_images
image = workspace.image_set.get_image(operand.binary_name.value, must_be_binary=True)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/measurement.py”, line 1596, in get_image
raise ValueError(“The %s image is missing from the pipeline.” % name)
ValueError: The None image is missing from the pipeline.
Mon Apr 13 00:12:15 2020: Image # 1, module MeasureImageAreaOccupied # 5: 0.00 sec

mask_measurement.cppipe (3.6 KB) mask_images.zip (6.3 KB)

Is there a reason you’re using rc3 rather than the main 3.0.0 release (or, even better, 3.1.9)? It’s hard for me to say for sure if the issue is due to rc3-vs-release or linux-vs-mac; we fixed a lot of bugs between rc3 - release (which was rc8)

Hi Beth,

I did a fresh install with the pip freeze (attached). I now have 3.1.8 running on Ubuntu. On my Mac, For some reason, I am only able to get 3.0.0 to run despite having Java 8 (tried both 241 and 152), that too only by opening it from the terminal. I did have success with 3.0.0 and successfully managed to create a pipeline (attached) to analyze sample data (also attached). Everything runs fine. Now, I copied over my cppipe to Ubuntu and ran the following command after moving the data to a tmp folder in the same folder my pipeline file was located in.

cellprofiler -p yeast_pipeline.cppipe -c -i tmp -o tmp

and now get the following error even though the images are all present.

CP-JAVA 00:11:31.645 [Thread-0] WARN o.c.imageset.ChannelFilter - Empty image set list: no images passed the filtering criteria.

Failed to prepare run for module ExportToSpreadsheet

Traceback (most recent call last):

File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/cellprofiler/pipeline.py”, line 2459, in prepare_run

not module.prepare_run(workspace)

File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/cellprofiler/modules/exporttospreadsheet.py”, line 700, in prepare_run

return self.check_overwrite(workspace)

File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/cellprofiler/modules/exporttospreadsheet.py”, line 958, in check_overwrite

image_number = metadata_group.image_numbers[0]

IndexError: index 0 is out of bounds for axis 0 with size 0

All of the images, pipeline file and environment file are in the attached zip file. If you can please help me sort this problem out, it will be a huge step in integrating CellProfiler with Calico’s analysis pipelines. Please advise.

Thanks,

Jagadish

sample_data_and_pipeline.zip (437 KB)

@bcimini There seems to have been an issue with the os.walk routine in lib/python3.7/site-packages/cellprofiler/main.py on line 822. I commented out the os.walk routine and replaced it with

pathnames =
for (dirpath, dirnames, filenames) in os.walk(options.image_directory):
pathnames += [os.path.join(dirpath, file) for file in filenames]

Now, I am able to get the full list of images I want to process, but then run into the error below. I tried setting JAVA_HOME to

/usr/lib/jvm/java-1.8.0-openjdk-amd64

as well as

/usr/lib/jvm/java-8-openjdk-amd64

but get the same erorr as below. The pipeline file and data are the same as attached in my earlier reply.

Exception in thread “Thread-0” java.lang.IllegalArgumentException: URI is not hierarchical
at java.io.File.(File.java:418)
at org.cellprofiler.imageset.ImageFile.getFileName(ImageFile.java:72)
at org.cellprofiler.imageset.filter.ExtensionPredicate.eval(ExtensionPredicate.java:21)
at org.cellprofiler.imageset.filter.ExtensionPredicate.eval(ExtensionPredicate.java:13)
at org.cellprofiler.imageset.filter.LogicPredicate.eval(LogicPredicate.java:48)
at org.cellprofiler.imageset.filter.Filter.eval(Filter.java:224)
at org.cellprofiler.imageset.filter.Filter.filterURLs(Filter.java:138)
Failed to prepare run for module Images
Traceback (most recent call last):
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/cellprofiler/pipeline.py”, line 2459, in prepare_run
not module.prepare_run(workspace)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/cellprofiler/modules/images.py”, line 366, in prepare_run
iffilter, “filterURLs”, “([Ljava/lang/String;)[Z”, file_array
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/javabridge/jutil.py”, line 887, in call
result = fn(*nice_args)
File “/home/jagadish/yeast_web/anaconda3/envs/cellprofiler_v2/lib/python3.7/site-packages/javabridge/jutil.py”, line 854, in fn
raise JavaException(x)
javabridge.jutil.JavaException: URI is not hierarchical

Ah! Before, you were showing a python2.7 environment; all the CellProfiler-3.X builds are python 2.7, so it won’t work in your 3.7 conda environment.

4.0, which we hope to be releasing very soon, will be Python3 based.

Great to hear. I do have a python 2.7 conda environment also with Cellprofiler, but run into other errors there like some I pasted above. It has CellProfiler v3.0.0rc3. I am ok with getting either environment to work, but currently neither does.

Look forward to the release!

Can you put a release version (not 3.0.0rc3, but 3.0.0, 3.1.5, 3.1.8, or 3.1.9 in your python 2.7 environment?

I switched it out to 3.0.0 and it works!! Thanks @bcimini for your patient help!

Once I am able to parse out the results etc., I am willing to create a pipeline and document my workflow as an example if it will help folks in the future. Please let me know the appropriate way to capture it.

1 Like