Hi,
It seems CellProfiler 4.04 freezes when processing a pipeline that CP 3 completed successfully.
Screen shots and the debug log are here: link.
How to recreate the error:
- In CP4 create a pipeline with measurement steps followed by an “ExportToSpreadsheet” step (see screenshot of pipeline in link above)
- Add about 30 image sets, each with 2 large images.
- Run the pipepline; wait for about 20 minutes to get near the end of the image sets.
- CP 4 will freeze (see screenshot at above link). Here “freeze” means the CP output log stops generating new messages. The last two message it will show before freezing will look like this, or some variation on this with different numbers:
Progress Counter({‘Done’: 56, ‘InProcess’: 4, ‘FinishedWaitingMeasurements’: 1})
Progress Counter({‘Done’: 57, ‘InProcess’: 4}) - Wait an hour; nothing happens; CP stays frozen.
- Click “Stop Analysis”. The CP output log starts generating messages again (see log snapshot after clicking cancel at above link) while it cancels things, but it doesn’t export the spreadsheet.
One of the symptoms is that in resource manager, the amount of RAM being used keeps growing by about 1-2 GB with each successive image set processed, even though CP is completing groups. Why can’t it save to the spreadsheet at the end of processing each group, to free up RAM?
Another symptom is that the last two lines before it freezes are always some variation of this:
Progress Counter({‘Done’: 56, ‘InProcess’: 4, ‘FinishedWaitingMeasurements’: 1})
Progress Counter({‘Done’: 57, ‘InProcess’: 4})
Another symptom is that it only freezes when it’s processing the last few image sets in a set of a list of images (e.g. it will freeze at image set 28 out of 31; in this case each image set has two images).
Another symptom is that it freezes while there is still >10 GB out of 48 GB free. It only increases the RAM use by about 1-2 GB per image set, so it’s strange that it freezes with relatively so much RAM left free
Another symptom is that after it freezes, if you click “Stop Analysis”, then it is able to unfreeze, but it doesn’t save the spreadsheet. The files here contain snapshots of the CP output log before and after clicking “Stop Analysis”.
Maybe workers are running out of their own memory buffers, but why do they have these limits, if there’s lots of RAM left? In CP3 one could assign the amount of RAM available to Java. Does CP4 have this feature, and could it help here?
Is there a way to make CP save to the spreadsheet after each group completes, rather than waiting for all of the groups to complete, so it does not need to keep all of that data in memory until the end of the entire batch?
Things I’ve tried:
- using fewer workers (4 instead of 16, it still freezes when it gets to the last few image sets)
- different image set; it still froze at the last few image sets
- cropping the images to about 1/50th of their original size: this worked; it made it all the way through the image sets; however, CP3 was able to run the pipeline with the full-sized images
What am I doing wrong?
System:
Windows 10
48 GB RAM
Thank you.
Best regards,
John