CPP2.1 - computer stuck in analysis, not in test mode

Dear developers,
Id like to do the analysis of 18 sets/pair (each set is composed from nuclei and RNA dots image).

Pipeline works for me in test mode, but after running analysis whole computer stuck in 2 min. Even moving with mouse is impossible.
Some collegues came back to v.2.0 because 2.1 made their computer fall down.

I have Windows 7, 64-bit, Intel i3-2105 CPU @ 3.1 GHz with 4GB RAM.

Could it be a problem in memory and releasing.

Is the memory released every cycle?
Should I do that by ConserveMemory?

Thank you very much for being helpful.
Josef

PS: Is there any possibility to export pipeline from 2.1 and import that to 2.0? Thank you.

[quote=“josefjar”]
Pipeline works for me in test mode, but after running analysis whole computer stuck in 2 min. Even moving with mouse is impossible.
Some collegues came back to v.2.0 because 2.1 made their computer fall down.[/quote]

A couple of suggestions:

  • Each module window takes a certain amount of time and memory to produce. A single window which produces multiple panels (such as IdentifyPrimaryObjects), each of which containing a very large image, will probably increase the performance overhead until CellProfiler becomes unmanagable. Use Test mode to preview the results but once you are ready for the full analysis run, select Window > Close all open windows to remove the windows that already exist, and then Window > Hide all windows on run to prevent window creation.
  • We have found that for some mid-range computers, CellProfiler attempting to use all the available processors doesn’t play well with other activities. That is, CP takes up all the resources and leaves none for other user interactions. You may be ab;le to remedy this by decreasing the number of analysis workers (i.e, number of processors used) under File > Preferences… > Maximum number of workers
    . This number is set to the number of processors by default; change it to a lower number and you should see some performance improvement.

Since CP 2.1 involves such significant changes from 2.0, it is not feasible to export to a prior version. Your best bet is to have both versions open at the same time and mirror the modules/settings manually.

Regards,
-Mark

Dear Mark,

I have similar problem as Josef with stucking computers. I tried 4 different computers (1. 12-cores Win7 64bit 24GB RAM RAID5, 2. 2-cores 2.59GHz Win7 32-bit 4GB RAM, 3. 16-cores linux 32GB RAM, 4. 8-cores Win7 64bit 32GB RAM), only the 4th computer (8-cores) was able to process my pipeline. I have used same pipeline and set of images (197 images png ca 2MB each image) for analysis on all computers (module windows closed when processing). Computer 1. and 2. completely stuck I was not able even to move with the mouse (hard restart was the only option, no error message). I also tried your recommendation to decrease the number of workers -> no effect. I have tested CP versions 2.1.0 a 2.1.1, both versions behave the same way. 3rd computer (linux) gave me the error message (printscreen included in the attachment, I also reported to CP by sendind the report yesterday). Bellow I also attach the pipeline. If you have any suggestions what I can do more, please let me know. I’m starting to be desperate as the 4th computer (the only which passed the analysis) is 15 km away from my building and my project holds on this problem.

Thank you in advance,

Radek
140703_Colonies_output_phalloidin.cpproj (132 KB)


Hi Radek,
Could you also post some of your images, so I can try the pipeline myself? You can zip them up and post them to this thread.
-Mark

[quote=“mbray”]Hi Radek,
Could you also post some of your images, so I can try the pipeline myself? You can zip them up and post them to this thread.
-Mark[/quote]

Hi Mark,

thank you for your reply. Bellow I’m sending required sample (images are bigger 12MB each so if you need more for testing, I can send it via some storage site)… Looking forward to read your suggestions and thank you for your help.

Radek
Phalloidin_resized_sharpened_grey.zip (9.76 MB)

Just a note here, I’ll look at this in more depth. CellProfiler’s multiprocessing will, by default, in 2.1, try to run an analysis on each of your processor’s cores. If you’re working with large images, you will be memory-bound rather than CPU-bound - you don’t have enough memory to run CellProfiler simultaneously on all your CPUs. You can reduce the number of CPUs you use in the preferences (File/Preferences) by changing the maximum number of workers (try a single worker to see if this is why analysis mode doesn’t work).

I ran the pipeline and image sets on my computer (8 cores / 8gb memory / windows 7 x64 / 8 workers) and experienced no problems. You can try the Windows 64 build from the trunk builds page (cellprofiler.org/cgi-bin/trunk_build.cgi) - it is pretty close to what I am running.

Finally - we have Sophos anti-virus in house and, although 2.1.0 is signed by the Broad Institute and although I contacted Sophos, asking them to please put our program on their whitelist, they refused and Sophos just cripples your computer unless you make an exception for CellProfiler. You’d see that in test mode I think, but perhaps it’s only happening to the workers.

Dear developers,

I experienced the same problem with CP 2.1 that my pipeline works in test mode but not in analysis mode. I’m using a workstation with 16 CPU cores, 108 GB ram and Win7 64bit. Additionally I have a problem with a Java memory error which I tried to get rid off but with no luck. Let me explain what I’m doing: I have quite big images (15000*15000 pixel) with GFP-expressing cells. The file size of one image is more than 100MB, so I uploaded it here if you want to have a look:

file-upload.net/download-922 … .tiff.html

Please find attached the pipeline I’m using. The goal is to identify all cluster of cells (colonies) that consist of at least 50 cells, that’s why I need high resolution. When I now enter test mode everything works fine. But when I hit the analysis button I’m always getting an error message during the module “RunImageJ” (I use it for background substraction), please find it also attached (“java_error”). This always appears at the same array position. I already tried to start CP with the command line by the command “CellProfiler.exe --jvm-heap-size=50g” but the problem is still present. So I’ve done the background substraction manuelly with ImageJ and loaded this image into CP and started the pipeline after the RunImageJ module… this time I got an error message during saving the image (attached “save_error”)… by the way: it took 3 hrs to get it analyzed. At least I could export the results in a spreadsheet, but it would be cool to have the image as a control what was actually counted. I’m fully aware that the images are quite big but if I would separate them I would for sure lose some precision so I would like to give it a try with the whole image if possible. Even if java would not be able to handle such big files why is it working in test mode?

Thank you very much for your help!

All the best,
Martin




image_GFP.cpproj (226 KB)

I’m looking at this now (file downloaded to /imaging/analysis/People/Lee/topic3737 for anyone inside the Broad firewall…). Martin, have you tried setting the number of cpus to 1?

–Lee

Just finished rebooting… running the pipeline froze my machine while running in test mode. Coming up with ways to process images such as these by loading and processing tiles is on our roadmap, but for now, an image this large is going to cause problems. You’ll get much better performance if you can tile your image, then process the individual tiles, for instance using one of the techniques mentioned here: http://imagej.1557.x6.nabble.com/Creating-individual-tiles-from-one-large-image-td5000374.html
(I would choose bfconvert).

Unfortunately, we discovered a problem after the 2.1.0 release that prevented the workers from getting the correct JVM heap size argument. The problem is corrected in 2.1.1 which is nearing release - the release candidate on the trunk build page (cellprofiler.org/cgi_bin/trunk_build.cgi) is very stable and an rc2 with minor changes to it should be released within the next two weeks. Having said that, no matter how much memory you reserve, the JVM cannot allocate individual chunks larger than 2G and since we pass images to ImageJ using the “double” type, this is close to or beyond the size of your image and that is probably the cause of the first exception.

Hi Lee,

many thanks for your suggestions and your time. I tried with setting the number of CPUs to 1 but the problem was still present. If I understood you right (please correct me if I’m wrong) I would need to use the bioformats tool and the command bfconvert outside of CP… if possible if would like to do everything in the pipeline. Currently I’m playing around with the release candidate 2.1.1 (really cool that you can now set the Java space in the preferences!) and I also tried to use the resize command before processing the image… I still need to figure out what the maximum downsampling rate is I can use but for now I avoided the first Java error by a resizing factor of 0.5. However, I’m still not able to save the image at the end (java heap space error)… I guess that’s due to the fact that I want to have a color image with outlining all the counted colonies. If there would be a solution for saving the image this would be awesome.

Thanks again for your help,

Best,
Martin