Cellprofiler memory errors when processing large images

Hi, I’m trying to use cellprofiler to analyze whole slide mouse brain images. There are 3 channels in each image and a single tif channel by itself is around 100 Mb. I’m trying to analyze the whole image and not crop it into pieces. When I use the CP on a batch of images it will reliably fail with memory errors. One or few images are ok, more causes the error. I’ve tried this on a computer with 128 Gb ram. More images get processed, but it will still fail when the memory fills up. Unfortunately if CP fails during the run the results collected before failure are lost when using spreadsheet export (would database export write results for each as it goes along? answer: yes, but coping with failures and restarting is a pain).

…I’m editing this original post. The solution to this problem is to run one image then shut down and restart a new instance of CP for the next image. To script CP in this fashion it needs to be run headless. To do that refer to this post: CellProfiler install from source and run headless to solve batch processing memory errors

Hi John,
I’ve been running into the same issue! I actually just increased my RAM up from 16 to 64gb, as my previous configuration seemed to be saturating when the error messages showed up, but I still have this issue despite the RAM being only 50% used during the tasks…
Many thanks to anyone who could help us solve this issue!
Cheers,
Ben

Hi Guys

I have had this issue couple of times, the ways I have found around it are
1- cellprofiler preferences - reduce the detectable cores
2- increase the java default memory (fixed almost all my memory errors)
3- on one occasion the local temp directory was full, this is because CellProfiler saves all measurements and then only outputs what you ask at the end, sometimes this temp directory is never deleted.

I hope you guys fix it

Best
Lee