Memory leak?

[mbarish]

CP2.1.2 seems to have a memory leak. There are 20 or more sets of 3 16-bit gray scale images each 6000x6000 pixels organized by subdirectory. The pipeline has a number of intermediate images, and I run ConserveMemory on all of these images after each subdirectory iteration. But Task Manager shows memory accumulating with each iteration even though I limit CP to 1 worker and set Java memory to 52 GB, and CP eventually crashes even with 56 GB RAM.

A possibly related issue is that the pipeline writes images as it iterates through the subdirectories, but only seems to output the spreadsheets after processing all of the subdirectories. Thus if it does crash, the images remain but all of the measurements are lost.

Any thoughts?

Thanks.

[David_Logan]

Hi,

Thanks for this report, and I have forwarded it to our developers for comment. In the meantime, can you tell us what platform you are using? (Windows 7, OSX Yosemite, etc)
And you are using one of the trunk builds presumably? How recent?

Thanks,
David

[LeeKamentsky]

Also, could you send us the pipeline and, if possible, a representative set of images? The pipeline will have the version if you saved it with the cellprofiler version that you are running. I’ll file an issue when I get that information. You might see memory use grow as you step along a pipeline, but from the second image set on, there should not be a significant change in memory usage.

CellProfiler 2.1.2’s measurements are stored on disk and should consume no memory. We have done some work to reduce the memory footprint of images as well.

[David_Logan]

@mbarish,
As an answer to your last question, Lee also said:

The second problem relates to how ExportToSpreadsheet works. It writes entire spreadsheets at once, rather than appending, so it writes them after the analysis (in post_run if I remember right). [This] experience is what we’d expect.

David