Is there a parameter flag to set the number of workers in headless mode such as -worker=8? I would like to use the multiprocessing batch mode with headless execution in the CP2.1 trunk version. Thanks in advance for any suggestions on how to use this option.
I’m going to quote the relevant snippet from this post: viewtopic.php?f=18&t=3256&p=9078&hilit=workers#p9078
You can read the rest for more details. It might be useful to include such as flag (which doesn’t currently exist) but I’ll have to check.
I think the best way to look at the multiprocessing mechanism is to think of the CellProfiler UI as the interface for running CellProfiler headless. It starts up a number of workers appropriate for the CPU configuration and then sends them work to do. Eventually, our plan is to let users run multiple CellProfiler workers on multiple machines and have a broker that connects CellProfiler UI’s to available workers. That would give you both a headless behavior and an infrastructure for controlling it.
We’ve added a new command-line switch, “–get-batch-commands”. This is meant to be used with the CreateBatchFiles module and a pipeline that uses grouping to group image sets into reasonably-sized chunks. The output is a series of CellProfiler invocations and each invocation processes one group of images. We put this in to let people create their own informal batch processing system - you could turn the output into a script that started multiple copies of CellProfiler, possibly partitioning the work into batch scripts to be run on separate computers. For windows, adding "start " before each line will run all of the processes in parallel.
This flag is now available in CellProfiler 2.1 and later, which can now be downloaded from http://cellprofiler.org.
Hi thanks for the work on --get-batch-commands
This should be advertized heavily !
this option is just great ! it saves an amazing amount of time !