Cellprofiler halt without error messages

Hello,

I have been using Cellprofiler wormtoolbox pipelines to analyze the C. elegans fat content in the past few years, and I kept running into the issue that Cellprofiler somehow pause analyzing without any error messages or notifications, this often happens to the last few images, and the issue persists in every version of cellprofiler from 2.2.0 to 4.1.3, and I tried running on many different PCs with OS of windows 7 and windows 10, and all of them ran into this same issue. I am wondering if there is a solution to solve this problem. The only way to make it finish the pipeline is to remove the last few images and rerun the pipeline.

Here I am providing an example:


This is a screensnap of a current pipeline I am running and it stuck at the image 33/34 for 3h already. The typical time for a successful run of 40 images usually takes only ~1h.


And it shows no error messages or any sign of the issue here.


This is the screensnap of the pipelines.

Hope anyone could help with this issue.

Thanks,

Wenfan

Hi @WKK ,

We’re aware of this issue but have struggled to replicate it consistently. Seeing as you’ve had the same problem multiple times with this pipeline, is there any chance you could upload the pipeline and image sets for us to test this with?

Thanks

Hi David, this is the pipeline I am using:

UntangleWormsExtractMeasurements.cpproj (142.9 KB)

You may need to re-link the model in a few steps of the pipeline before you use it:
MyWormModel20210315.xml (19.5 KB)

You need to run the whole set of the images to have that problem, I tested when I just picked the last few images to run and that issue did not come up. The whole package of the images is too large to upload here, do you have other ways for me to send it to you?

Thanks again!

Wenfan

Thanks, there are a bunch of large file download services available. Something like Dropbox, Google Drive or WeTransfer should do just fine.

I uploaded the images to google drive, you can find them using this link:

Thanks for that, I’m taking a look now.

One quick question - you appear to be running 24 workers, does your analysis machine have 24 physical CPU cores? Using more workers than cores can negatively impact analysis performance.

Hi David,

Sorry for the delayed response. No, our machine do not have 24 physical CPU cores… I did not realize this because I just used the default setting on this. Which step in the pipeline I should change to make it using fewer cores?

Thanks,

Wenfan

You can configure core count in the File–>Preferences menu, under the “Maximum number of workers” setting.

We’ve also done some investigating on our end, and in the case of this error it looks like the problem usually occurs if there’s a corrupted image somewhere that crashed a worker thread. If you’re using ExportToDatabase then all your data should have still been saved correctly.