Analysis_worker mode on linux

cellprofiler

#1

I’m trying to find some infomation on how to run an “analysis_worker” job on linux-flavored cellprofiler. I’ve seen an analysis_worker.exe binary on windows, but can not seem to find the equalivent binary when compiled from source on linux (via CellProfiler/docker git repo). Is there some undocumented trick to running analysis_worker mode from the normal cellprofiler binary?

Thanks.


#2

So I’m not a CP-linux guru, but there’s a lot more information on how CP makes and calls workers in the git pages on Workers and Analysis .

Can I ask more generally what you’re trying to do? If you want workers because you want to spread things out across a cluster, we have ways to do that.


#3

Sorry, let me be more specific on what my needs are.

We have a distributed computing environment (PBS/Torque cluster). We would like to be able to submit a CellProfiler job to one of compute nodes, and assign other compute nodes as worker machines. As far as I am aware, this is possible on Windows by running CellProfiler.exe and then, on another machine, run analysis_worker.exe and pass in the IP of the machine running the main CellProfiler job, which then queries the main machine for jobs.

My problem is that it is not clear how to replicate this functionality on a collection of non-Windows machines.


#4

Hi gregjohnso,

I think what you would like to do is called “Batch Processing” in the CellProfiler manual. Check the corresponding section in the manual (pdf and html versions at http://cellprofiler.org/releases/). In short, its done a bit differently on Linux, in my eyes its actually simpler. You need to create a batch file from your pipeline, and then you can invoke CellProfiler independently on subsets of the input images, You do that by specifying the first and last index of images to be processed by each job. Afterwards you collect all the outputs and merge them. Does that help?

Cheers, Mario


#5

I spoke with Alan Goodman, and he mentioned that there is an undocumented flag on the linux version that produces the behavior that I’m looking for. To assign a job to a worker note, one must include the following flag to the command-line call on the worker machine.

–work-announce=tcp://[ip]:[port]