CellProfiler 3 Windows 10 large dataset

cellprofiler

#1

Hi community,
I have recently installed CellProfiler 3 on a Windows 10 PC. When I use my pipeline that is working well on MacOS and run a small number of images it works fine, but when I use my full data set (16.000 images) I get immediately the attached errors in the terminal window. No further error messages are displayed and the CellProfiler GUI indicates that the processing is running, but no progress is made. I tried to reinstall both CP3 and Java, but this did not change anything. Is it possible that this is a Python problem?
Thanks fro your help
Alex

Console error message:
Starting workers on address tcp://127.0.0.1:59745
Progress Counter({u’Unprocessed’: 8064})
Worker 5: Failed to write all bytes for svml_dispmd.dll
Worker 17: Failed to write all bytes for wxmsw30u_core_vc90.dll
Worker 14: Failed to write all bytes for wxbase30u_vc90.dll
Worker 28: Failed to write all bytes for svml_dispmd.dll
Worker 11: Failed to write all bytes for bioformats\jars\loci_tools.jar
Worker 7: Failed to write all bytes for bioformats\jars\loci_tools.jar
Worker 13: Failed to write all bytes for bioformats\jars\loci_tools.jar
Worker 29: Failed to write all bytes for sklearn.svm.libsvm.pyd
Worker 27: Failed to write all bytes for sklearn.svm.libsvm_sparse.pyd
Worker 0: Failed to write all bytes for win32evtlog.pyd
Worker 15: Failed to write all bytes for cellprofiler\data\icons\CellProfiler-favicon-120.png
Worker 3: Failed to write all bytes for cellprofiler\data\icons\CellProfiler-favicon-120.png
Worker 10: Failed to write all bytes for cellprofiler\data\icons\CellProfiler-favicon-120.png
Worker 20: Failed to write all bytes for skimage.transform._seam_carving.pyd
Worker 18: Failed to write all bytes for wx._html.pyd
Worker 22: Failed to write all bytes for win32com.shell.shell.pyd
Worker 4: Failed to write all bytes for wx.windows.pyd
Worker 8: Failed to write all bytes for wx.core.pyd
Worker 23: Failed to write all bytes for tcl85.dll
Worker 9: Failed to write all bytes for wx.core.pyd
Worker 25: Failed to write all bytes for sklearn.utils._random.pyd
Worker 26: Failed to write all bytes for sklearn.metrics.pairwise_fast.pyd
Worker 21: Failed to write all bytes for sklearn.svm.libsvm.pyd
Worker 24: Failed to write all bytes for zmq.backend.cython.socket.pyd
Worker 16: Failed to write all bytes for bioformats\jars\loci_tools.jar
Worker 2: Failed to write all bytes for cellprofiler\data\help\legacy_load_modules.rst
Worker 1: cellprofiler\data\help\display_menu_bar.rst could not be extracted!
Worker 19: cellprofiler\data\help\display_menu_bar.rst could not be extracted!
Worker 12: cellprofiler\data\help\display_menu_bar.rst could not be extracted!
Worker 6: cellprofiler\data\help\display_menu_bar.rst could not be extracted!


#2

Hi,

If the pipeline works well on that PC with a small number of images but not with 16K images, my guess is that you’re running out of RAM, temp space, permanent space, or some combination of the above. How much RAM does the machine have? How much disk space? If you try decreasing the number of workers and/or decreasing your batch size (say, 2 or 4K rather than 16K), does it succeed then?


#3

Hi Beth,
thank you for the fast answer. You were right, the temporal space was limiting. Windows unexpectedly loaded 130 Gb of updates. After that the pipeline worked. I only received one more error, that I attach here. Capture

Also at the end of the run I was not able to start a new analysis or close the CP window. In the terminal I get the following error:

Traceback (most recent call last):
File “CellProfiler\cellprofiler\gui\cpframe.py”, line 483, in OnClose
File “CellProfiler\cellprofiler\gui\pipelinecontroller.py”, line 1333, in check_close
File “site-packages\wx-3.0-msw\wx_controls.py”, line 193, in init
wx._core.PyAssertionError: C++ assertion “id == wxID_ANY || (id >= 0 && id < 32767) || (id >= wxID_AUTO_LOWEST && id <= wxID_AUTO_HIGHEST)” failed at …\src\common\wincmn.cpp(372) in wxWindowBase::CreateBase(): invalid id value

Thanks again for your help.
Best wishes
Alex


#4

Did the error pop up pretty early in the process, middle of the process, right at the end? Did the error keep CP from processing or keep it from outputting the data correctly? I’ve seen errors like that before, and I have some guesses as to what it might be related to, but not sure the best way to troubleshoot it if it only occurs at the end of processing a 16K image dataset.


#5

Hi Beth,
the first type of error occurred the first time after approx. 1500 images. CP did not stop processing, I could close the error window by clicking “Continue Processing”, although CP did not stop anyhow. The next morning an other 4 of these windows were present, but CP had finished the dataset.

The error in the terminal window only appeared when I tried to continue with a second dataset after the first 16k had finished. The GUI did not respond at all.

I hope that helps.
Best wishes
Alex