CellProfiler 4 freezes near end of list of image sets

Usage & Issues
#1

Hi,

It seems CellProfiler 4.04 freezes when processing a pipeline that CP 3 completed successfully.

Screen shots and the debug log are here: link.

How to recreate the error:

  1. In CP4 create a pipeline with measurement steps followed by an “ExportToSpreadsheet” step (see screenshot of pipeline in link above)
  2. Add about 30 image sets, each with 2 large images.
  3. Run the pipepline; wait for about 20 minutes to get near the end of the image sets.
  4. CP 4 will freeze (see screenshot at above link). Here “freeze” means the CP output log stops generating new messages. The last two message it will show before freezing will look like this, or some variation on this with different numbers:
    Progress Counter({‘Done’: 56, ‘InProcess’: 4, ‘FinishedWaitingMeasurements’: 1})
    Progress Counter({‘Done’: 57, ‘InProcess’: 4})
  5. Wait an hour; nothing happens; CP stays frozen.
  6. Click “Stop Analysis”. The CP output log starts generating messages again (see log snapshot after clicking cancel at above link) while it cancels things, but it doesn’t export the spreadsheet.

One of the symptoms is that in resource manager, the amount of RAM being used keeps growing by about 1-2 GB with each successive image set processed, even though CP is completing groups. Why can’t it save to the spreadsheet at the end of processing each group, to free up RAM?

Another symptom is that the last two lines before it freezes are always some variation of this:
Progress Counter({‘Done’: 56, ‘InProcess’: 4, ‘FinishedWaitingMeasurements’: 1})
Progress Counter({‘Done’: 57, ‘InProcess’: 4})

Another symptom is that it only freezes when it’s processing the last few image sets in a set of a list of images (e.g. it will freeze at image set 28 out of 31; in this case each image set has two images).

Another symptom is that it freezes while there is still >10 GB out of 48 GB free. It only increases the RAM use by about 1-2 GB per image set, so it’s strange that it freezes with relatively so much RAM left free

Another symptom is that after it freezes, if you click “Stop Analysis”, then it is able to unfreeze, but it doesn’t save the spreadsheet. The files here contain snapshots of the CP output log before and after clicking “Stop Analysis”.

Maybe workers are running out of their own memory buffers, but why do they have these limits, if there’s lots of RAM left? In CP3 one could assign the amount of RAM available to Java. Does CP4 have this feature, and could it help here?

Is there a way to make CP save to the spreadsheet after each group completes, rather than waiting for all of the groups to complete, so it does not need to keep all of that data in memory until the end of the entire batch?

Things I’ve tried:

  • using fewer workers (4 instead of 16, it still freezes when it gets to the last few image sets)
  • different image set; it still froze at the last few image sets
  • cropping the images to about 1/50th of their original size: this worked; it made it all the way through the image sets; however, CP3 was able to run the pipeline with the full-sized images

What am I doing wrong?

System:
Windows 10
48 GB RAM

Thank you.

Best regards,
John

#2

Hmmm, that’s very unusual, and we’re sorry to hear it! We’ll try to reproduce with some image sets on our side; if we can’t, we may need to touch base with you to get your image sets.

#3

Hi @johnramunas,

It looks like the files you uploaded aren’t public, so I can’t view them. However, you’re right that ExportToSpreadsheet tries to write data after the run, rather than after each group or image set.

A possible solution might be to use the ExportToDatabase module instead. This module writes as the run progresses and so shouldn’t hit memory issues.

However, the program shouldn’t be freezing instead of completing. Pressing “Stop analysis” should indeed cancel the writing process entirely so that’s expected. We’ll take a look into this.

#4

Hi @bcimini,

I’ve recently started running into a similar issue as well with CP 4.07, with pretty much the exact symptoms that John described in his initial post. The pipeline processes all images until the very last image, and then just hangs. The Progress Counter just shows the second to last image InProcess. What’s curious is that this seems to happen ~90% of the time, but every once in awhile it’ll be able to finish up an image set. The same pipeline causes this issue across 3 different computers. Switching to Export to Database as David suggested above led to the pipeline eventually slowing down about halfway through, and then just extending in length indefinitely.

Re-creating the pipeline in CP 3 worked fine, even with 5000 images writing to spreadsheets at the end.

Did you guys ever get a chance to follow up on John’s initial issue? Happy to provide my pipeline/images if that help.

Thank you.

#5

Do you have a sense of the number of images (and what size, and ballpark how many objects per image- 10, 100, 1000, etc) that are necessary to trigger this issue in your hands- >1000 images, 2000, etc?

I ask because we haven’t been able to verify it in our hands (and typically folks can’t/don’t want to share thousands of images to help us verify - if you have a minimal image set that causes this issue on your machine, please do share it!), which makes it hard to debug and fix in the absence of an error message pointing to a specific issue. Typically we would switch to running on a cluster after more than a couple of hundred images. I can attempt to let this run on a single local machine though if you have a sense of how much data is needed to cause the issue.

It would also be great to have a look at your pipeline, to see if there’s anything unusual there compared to what we would typically run in our hands that might be why we haven’t seen it ourselves.