CellProfiler(3.1.9) MemoryError during Run of Module MeasureObjectSizeShape

Hi, CellProfiler Team and everyone
I am running into an issue where my pipeline works fine when run for a small image sets, But when I run more image set (about 60 image sets), MemoryError occurs during run of module MeasureObjectSizeShape. My image size is 2048×2048, an image set has 2 images. My computer has an 8-core processor and 16GB RAM.
Here is the error log information:

1-Error message
When I reduce the worker to 2 and change the Java memory to 4096 MB, the pipeline can run well, but it runs very slowly. So is there any other better method that can improve the analysis speed and avoid MemoryError.
If you can provide me with any suggestions, I would really appreciate it!
Thank you very much!

Hi @Gavyn,

This is indeed related to hitting your system’s memory limit, and reducing the worker count was a good idea to work around that. Without seeing the images and pipeline it’s difficult to make comprehensive suggestions, but one thing I would watch for is whether some of your images are having excessive numbers of unwanted objects detected. This can happen if you have images of empty or negative samples being processed without any lower limits in the Threshold or Identify modules, or if you have no size criteria specified for the objects you’re trying to find.

Hope that helps

Hi @DStirling, Thank you so much for your suggestions. The purpose of my pipeline is to analyze the mitochondrial morphology. Each image can identify about 1000-2000 mitochondria. When 20-30 image sets are completed, MemoryError starts to occur in module 30(MeasureObjectSizeShape). The error log information indicates that it may be related to measuring Zernike features.
Here is my pipeline and example images:
analysis_1.cppipe (87.8 KB)
test_B02_s14_w2.TIF (8.0 MB)
test_B02_s14_w1.TIF (8.0 MB)

Hi @Gavyn,

Thanks for that, if the objects are genuine then there are a couple of things we could do to try to reduce memory usage. Looking at your pipeline, the simplest change might be to use ExportToDatabase instead of ExportToSpreadsheet. The database exporter writes data progressively as the images are progressed, whereas the spreadsheet writer stores all the data in memory until each group is completed. You may want to setup ExportToDatabase in ‘sqlite’ mode, since this will produce a local database file from which spreadsheets for each object type can easily be exported.

It might also be worth upgrading to CellProfiler 4. In the newer version many of the measurement modules have been improved. We’ve made it easier to analyse multiple images without needing additional copies of the modules, and MeasureTexture should also be much faster.

Hi @DStirling ,
Thanks again for your suggestion, I will try it.

Hi, @DStirling
After selecting the ExportToDatabase module to export data, MemoryError will still occur when about 50 image sets are completed, so I have to choose not to calculate Zernike features for any object.
Since I did not label the cell membrane or cytoplasm, I selected the adaptive threshold strategy in CellProfiler (3.1.9) to segment cells. The identified cell object is only used to determine which cell the mitochondria belong to. But when I upgraded the pipeline to CellProfiler 4, no matter how I adjusted the pipeline, the identified cells will discard some small fragments of mitochondria, , which may be important.
Here are the cell segmentation results in different versions of CellProfiler:
CellProfiler 3.1.9

CellProfiler 4.0.6

By the way, when I was running the pipeline, ValueError happened by accident. Here are the details:

I don’t know what caused this error.
Thank you very much!

Hi @Gavyn,

We’re planning to release version 4.0.7 in the coming days, which will add a new option to more closely replicate thresholding results with CP3’s 3-class Otsu method.

Is the new error you captured happening on CP3 or CP4? It may help if you could upload the specific image set which is triggering this error, too. Thanks!

Hi @DStirling,
The new error happened on CP3. But when I processed the image set that triggered the error again, the error did not occur again. Here are images:
test_C02_s16_w1D8AD35C3-2119-4A1E-A31D-D90FC0E1F6A4.tif (8.0 MB) test_C02_s16_w25FE3C022-B5A6-40A1-91EA-8AD0DD589940.tif (8.0 MB)
When the new version of CellProfiler is released, I will try it and adjust my pipeline. Thanks!