Encountered error while processing

Usage & Issues
#1

Hi,
I am desperately trying to have my pipeline working but I cannot seem to be able to get to the end without an error (last time at 2 min to the end of of a 3h and a half process…).
I am running the attached pipeline on a Mac (El Capitan) using Cellprofiler 2.2.0
This is the error message I get (needless to say that I don’t understand a word of it…):

Traceback (most recent call last):
File “wx/_core.pyc”, line 16766, in
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/gui/pipelinecontroller.py”, line 2697, in on_stop_analysis
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 364, in getitem
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 960, in get_measurement
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 367, in getitem
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 644, in get_dataset
File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
File “/Applications/CellProfiler.app/Contents/Resources/lib/python2.7/h5py/_hl/group.py”, line 164, in getitem
File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
File “h5py/h5o.pyx”, line 190, in h5py.h5o.open (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5o.c:3507)
KeyError: “Unable to open object (Can’t insert duplicate key)”

Any suggestion?
Thanks in advance

MatteoSCREEN PIPELINE200416.cpproj (5.3 MB)

1 Like
#2

I have the same issue. It goes through the first 775 images, and all of the sudden it gives me this error:
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module SaveImages # 25: 0.27 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module CalculateMath # 26: 0.00 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module CalculateMath # 27: 0.01 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module CalculateMath # 28: 0.01 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module CalculateMath # 29: 0.00 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module CalculateMath # 30: 0.01 sec
Jul 25 21:04:27 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module ExportToSpreadsheet # 31: 0.00 sec
Jul 25 21:04:28 cellprofiler[14090] : Failed to write measurements to database
Jul 25 21:04:28 cellprofiler[14090] : Traceback (most recent call last):
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/modules/exporttodatabase.py”, line 2987, in write_data_to_db
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 1283, in compute_aggregate_measurements
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 960, in get_measurement
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 360, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 367, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 644, in get_dataset
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/h5py/_hl/group.py”, line 164, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/h5o.pyx”, line 190, in h5py.h5o.open (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5o.c:3507)
Jul 25 21:04:28 cellprofiler[14090] : KeyError: "Unable to open object (Can’t insert duplicate key)"
Jul 25 21:04:28 cellprofiler[14090] : Error detected during run of module ExportToDatabase
Jul 25 21:04:28 cellprofiler[14090] : Traceback (most recent call last):
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/pipeline.py”, line 1819, in run_with_yield
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/pipeline.py”, line 2067, in run_module
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/modules/exporttodatabase.py”, line 1781, in run
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/modules/exporttodatabase.py”, line 2987, in write_data_to_db
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 1283, in compute_aggregate_measurements
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/measurements.py”, line 960, in get_measurement
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 360, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 367, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/cellprofiler/utilities/hdf5_dict.py”, line 644, in get_dataset
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
Jul 25 21:04:28 cellprofiler[14090] : File “/Users/rojin/Desktop/CellProfiler.app/Contents/Resources/lib/python2.7/h5py/_hl/group.py”, line 164, in getitem
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2458)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/_objects.c:2415)
Jul 25 21:04:28 cellprofiler[14090] : File “h5py/h5o.pyx”, line 190, in h5py.h5o.open (/Users/travis/build/MacPython/h5py-wheels/h5py/h5py/h5o.c:3507)
Jul 25 21:04:28 cellprofiler[14090] : KeyError: "Unable to open object (Can’t insert duplicate key)"
Jul 25 21:04:28 cellprofiler[14090] : Mon Jul 25 21:04:27 2016: Image # 775, module ExportToDatabase # 32: 0.47 sec
Jul 25 21:04:29 cellprofiler[14090] : Failed to stop Ilastik
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored
Jul 25 21:04:29 cellprofiler[14090] : Exception TypeError: “‘NoneType’ object is not callable” in ‘_javabridge.JB_Object.dealloc’ ignored

Any help?

#3

See this thread and links within; tl,dr is that it’s a known bug that we’re working on fixing, in the meantime you can try disabling h5 (if you have it enabled) or running smaller sets. Sorry for the inconvenience!

#4

Hello there, I know this is an old post but I have recently encountered the similar problem. I am trying to run tens of thousands of FLEX images (containing DAPI and GFP channels), but the software appears to crash after retrieving the metadata of 700 images. I wonder whether there is a limit on the amount of images the programme can process in one go. Or perhaps there is a way that I can turn this into a more high-throughput analysis.

Thank you for your time!

#5

I wonder whether there is a limit on the amount of images the programme can process in one go.

There definitely isn’t a limit the program can handle; what the specs of your computer can handle may be very different. Can you give us any more information about your machine, its specs, what version of CP you’re running, what (if any) crash message you’re getting, your pipeline, etc?

Depending on the answers there, you may need to investigate getting your hands on a more powerful machine to run the analysis or even seeing if your university has a cluster setup you can use. We also have a tool for running CellProfiler in a cloud computing environment, but it takes a nonzero amount of setup to get started.

#6

Hi,

Thank you for your response. I figure it may be my computer which has a limited processing power.

I’m using a Windows 7 PC, with an Intel i5-6500 processor (3.20 GHz). The RAM is 16GB.

The pipeline I use is modified from the human cell basic pipeline, measuring nuclear sizes.
Nuclear Size Analyses Pipeline DHD1 Plate 01.cppipe (12.4 KB)

The crash message I get is “Encountered unrecoverable error in Metadata during startup: GC overhead limit exceeded”
Traceback (most recent call last):
File “cellprofiler\pipeline.pyc”, line 2094, in prepare_run
File “cellprofiler\modules\metadata.pyc”, line 732, in prepare_run
File “cellprofiler\utilities\jutil.pyc”, line 785, in call
File “cellprofiler\utilities\jutil.pyc”, line 762, in fn
JavaException: GC overhead limit exceeded

The image inputs has 720 images (2 channels in each image). I ran through NamesandTypes and Groups modules finely, but when I started the Analyze Images button, it then crashed.

#7

Yeah, it’s a memory error; you can try upping the amount of memory CP’s Java has access to under File->Preferences, but you may also have to possibly just run smaller batches if that doesn’t help. Sorry!

#8

That’s okay. Thanks for the clarification! :slight_smile: