CellProfiler using processing power in short bursts Mac Pro Late 2013

Hello, I’ve recently acquired a 12 cores Mac Pro to help with my processing times with Cell Profiler. However, for some reason Cell Profiler will open all workers as usual, than proceed to drop CPU usage to 0%, then back up to optimal levels, than back down. This perpetuates indefinitely until processing is complete. As you would assume this is not efficient and ends up running at the exact same speed as my drastically less powerful machines. Any idea on how to resolve this? Yes I’ve CPU tested already and this problem only occurs with Cell Profiler. I’ve attached a screenshot to show this pattern.

How does the periodicity of the spikes compare with a)the overall time it takes to process an image b) when individual modules finish? Is it simply that there’s a compute-heavy portion(s) of your pipeline and then usage goes back down until you reach the next one? Do you see this spiking behavior if you limit CP to using only one or two workers, and if you do how does that affect the periodicity?

Thank you for your quick reply, the imaging ceases to advance for example 270 out of 600 until the processing increases once again then it proceeds to 271,272 etc. Once it drops back down, it ceases to progress through the set until it spikes once more. I have not tried limiting to a couple workers, I will test this and reply as soon as I see the outcome. What’s puzzling is that I have several other computers that are vastly inferior to this Mac Pro yet using the exact same pipeline with less images in some cases. I’ve never noticed a trend of the more computationally heavy portions of the pipeline causes issues especially on this computer where the highest % load it has reached is 90% with the rest being idle.

Between peaks it analyzes exactly 16 images during this “peak”. (I do have 16 workers running in total) I tried running on one worker and the spiking behavior still occurs. % CPU load ranging from 6-60% for that one worker. It will complete one, drop back down, then spike again for the next image. Is there any explanation as to why it is “taking a break” between sets? These breaks can border on 10 seconds each which fro our thousand image sets can almost double our processing time. Again, thanks for your assistance.

Can I ask what you mean by the processing “ceases to advance/progress”? Are you talking about the little counter at the bottom with the progress bar (which will only increment as images are finished, but does not mean analysis isn’t running on more images), or are you looking at the individual calls on the output log (which would look like what I’m posting below) and saying that nothing comes up there for a while at a time? If it’s the latter, that’s concerning; if it’s the former, it just sounds like your 12 images at a time are processing relatively synchronously, so you go through “spikes” of CPU intensive modules (sounds like maybe the beginning and/or end modules if it lines up well with the counter of images finishing incrementing) with some lower-CPU/memory-bound steps in the middle of your pipeline (certain measurement modules, for example, tend to consume lots of memory but less CPU in my understanding).

Mon Jul 17 19:54:36 2017: Image # 3446, module LoadData # 1: 16.58 sec
Mon Jul 17 19:54:55 2017: Image # 3446, module MeasureImageQuality # 2: 106.01 sec
Mon Jul 17 19:56:43 2017: Image # 3446, module MeasureImageQuality # 3: 4.24 sec
Mon Jul 17 19:56:47 2017: Image # 3446, module FlagImage # 4: 0.21 sec
Mon Jul 17 19:56:48 2017: Image # 3446, module CorrectIlluminationApply # 5: 0.78 sec

I doubt it’s “taking a break”, much more likely it’s just executing some module that doesn’t take tons of CPU- have you looked at memory usage and seen if it “spikes” while your CPU troughs? If it’s literally doing it between images, it could just be the image loading modules, which are quite complicated under the hood (by necessity of handling so many different kinds of files, metadata, etc) and often require a few seconds to pull up the next image set- I’ve never tried to profile CPU usage of the image loading modules but it’s possible that while it’s doing some of the handoffs CPU usage might be low.

Without looking at logs of individual steps like I put in my last post, it’s hard to say which if any modules are taking longer than they “should”, but 10 rogue seconds doesn’t sound like something we’d be able to necessarily be able to even track down- we routinely run image sets with many thousands of images using pipelines that take 1-2 hours to complete per image, I’m definitely impressed if your pipeline only takes 10 seconds per image not counting the loading! I’ve heard anecdotally that using the LoadData module rather than dragging and dropping image sets may improve the loading time somewhat, but if your pipeline is truly that fast you’d honestly probably spend almost as much time doing the CSV generation as you’d gain in processing speed, and it’d be “active” time rather than “walk away and get a coffee” time, so I’m not sure you’d come out ahead in the long run.

I see what you mean from your perspective. We only began questioning the process since we’ve run pipelines extensively on other computers and this is the first time we’ve seen this situation among 4+ different mac setups which is why we’re skeptical. I actually have never called on the specific logs before but will definitely begin doing so for further diagnostics . Also a slightly different issue we run into periodically, have you been able to pinpoint what is the issue when a single worker fails while the others run? In the end the run is stuck at 99/100 images or 450/451 images forever. The only thing we can do is restart the pipeline. Thanks!

Also a slightly different issue we run into periodically, have you been able to pinpoint what is the issue when a single worker fails while the others run? In the end the run is stuck at 99/100 images or 450/451 images forever. The only thing we can do is restart the pipeline.

I’ve run across that issue only once that I can remember, and I don’t think I was ever able to trace out a reason for it. If you’re finding it coming up often enough that it’s worth asking about, some food for thought- I’m not sure if you use ExportToDatabase or ExportToSpreadsheet, but at least with the former you’d have the database with the measurements from every image that DID finish (aka all but one, which depending on your use case might be enough OR you could just rerun that one missing image), since that module writes as it goes, but ExportToSpreadsheet only writes at the very end. If you’re able to get an error message or trace down a pattern of when it happens though, we’ll certainly look into it!

That is extremely helpful, we’ve always been asking if there was a method for accomplishing this because we are more than willing to sacrifice the last image. Thank you so much you were a great help!