Encountered error while processing?

Hey there,

I’m running CellProfiler 2.2.0 (rev 9969f42) on a Mac Pro El Capitan (2 x 3.06 GHz 6-Core Intel Xeon. 48 GB 1333 MHz DDR3 ECC).
I’m running the attached pipeline to analyze 2400 image sets and I SYSTEMATICALLY end up with the following error towards the very end (2360 image sets analyzed over the 2400…which is a bit frustrating…):

Traceback (most recent call last):
File “wx/_core.pyc”, line 16766, in
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/gui/pipelinecontroller.py”, line 2697, in on_stop_analysis
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 364, in getitem
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 960, in get_measurement
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 367, in getitem
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 644, in get_dataset
File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/h5py/_hl/group.py”, line 164, in getitem
File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
File “h5py/h5o.pyx”, line 190, in h5py.h5o.open (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5o.c:3507)
KeyError: “Unable to open object (Can’t insert duplicate key)”

Any idea?
Thanks!

Matteo
SCREEN PIPELINE210616.cpproj (5.6 MB)

A couple- it looks like what’s failing is h5 file writing. Do you have the h5 file enabled in “Default Output Settings”? Do you need the h5 file given that you’re writing to database?

If you don’t have it set it may be generating anyway, which unfortunately is a known error.. It’s slightly more annoying to be sure, but are you able to successfully complete the analysis if you run your plate in two batches?

I do not have the h5 option enabled, I’m just saving everything on the database.

I thought it was a problem of giving cellprofiler too many processors (I gave it 10 out of the 12 I have) so I tried to run multiple CP windows each with 3 processors, to analyse 3 plates at the same time…still the same error on the 3 windows.
I did not try running it as a batch (never tried this actually…), my problem is that I have over 20 plates to analyse and if I start splitting them I’m afraid I will end up in a big mess!
The pipeline USED to work fine, is that something to do with the new CP release? If I go back to a previous version will it be better?

Thanks

Well if it used to work in an older version, I see no reason why it won’t again. A later build may work also. I’m sorry, this would indeed be very frustrating!

When I meant running it in a batch, I phrased that poorly- I just actually meant only give it half a plate of images to analyze not any sort of batch mode. Based on your and other people’s experiences I expect that with a smaller data set it would at least finish; I agree that it’s annoying to have to run 40 half plates rather than 20 full plates but if it’s the difference between success and failure at the moment it may be the best short-term solution.

I’ve created a bug report on GitHub based on your information; we’re getting reinforcements on the software development side in just a couple weeks so hopefully we’ll have this bug squished soon!

Thanks for your help!

I found an alternative workaround that has solved my problem (not necessarily yours… :wink:): As I have Windows running on Parallels on my Mac I used the exact same pipeline on the CellProfiler version installed on the virtual machine (2.2.0rc2).
No issues there, I’ve been running analysis all week end without a single problem…I guess it’s something in the Mac version of CP then…

Matteo

That’s actually very helpful information to know, so thank you for the update! Glad you have a way forward. Happy analyzing!