Using GNU parallel to process multiple files in headless cellprofiler

I have a pipeline where I just output the labels corresponding to the nuclei.
@bcimini suggested I install GNU parallel on my Ubuntu 16.04 server to batch process files in headless mode.

I wanted to do a test run where I only save the labels of the first 10 images of the first 4 movies.

I got the cellprofiler --get-batch-commands to spit out the first and last

dolivierhub@ficoides:~$ cellprofiler --get-batch-commands /home/dolivierhub/data/outputlabel/Batch_data.h5
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 1 -l 148
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 149 -l 296
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 297 -l 444
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 445 -l 592
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 593 -l 740
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 741 -l 887
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 888 -l 1034
CellProfiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -f 1035 -l 1181

the output directory has sub-folders named OutputDir_1, OutputDir_2, etc

then I put in the following code, to have 2 processes run in parallel and save the output in specified output directories.

parallel -j 2 cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o {1} -f {2} -l {3} --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP ::: /home/dolivierhub/data/outputlabel/OutputDir_1 /home/dolivierhub/data/outputlabel/OutputDir_2 /home/dolivierhub/data/outputlabel/OutputDir_3 /home/dolivierhub/data/outputlabel/OutputDir_4 ::: 1 149 297 445 ::: 10 159 307 455

The good news is: 2 processes are running the pipeline (i cna see that on the system monitor

the bad news is:

  • the 2nd process re-analyses the same images and overwrites them,
  • it processed until image 30 when I quit terminal

What did I do wrong??

I have lots and lots of movies that need to be analysed and starting it by hand one by one is going to drive me crazy
plus i’d really like to learn how to!

thanks in advance

Debbi

The first thing to always check if a parallel command isn’t giving you the output you think it should be is to put --dryrun in front of it, to see exactly what output you are getting! I ran this on a system and got below.

parallel --dryrun -j 2 cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o {1} -f {2} -l {3} --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP ::: /home/dolivierhub/data/outputlabel/OutputDir_1 /home/dolivierhub/data/outputlabel/OutputDir_2 /home/dolivierhub/data/outputlabel/OutputDir_3 /home/dolivierhub/data/outputlabel/OutputDir_4 ::: 1 149 297 445 ::: 10 159 307 455

cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 1 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 1 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 1 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 1 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 149 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 149 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 149 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 149 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 297 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 297 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 297 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 297 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 445 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 445 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 445 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 445 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 1 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 1 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 1 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 1 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 149 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 149 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 149 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 149 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 297 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 297 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 297 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 297 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 445 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 445 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 445 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 445 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 1 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 1 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 1 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 1 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 149 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 149 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 149 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 149 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 297 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 297 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 297 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 297 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 445 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 445 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 445 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 445 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 1 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 1 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 1 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 1 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 149 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 149 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 149 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 149 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 297 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 297 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 297 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 297 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 445 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 445 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 445 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 445 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP

If you notice, it’s taking all of your commands and doing all the permutations of all of them!

Going back to the parallel documentation, you’ll see there’s a flag to not do that- depending on what version of parallel you are running, it’s either --xapply or --link .

parallel --dryrun --xapply -j 2 cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o {1} -f {2} -l {3} --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP ::: /home/dolivierhub/data/outputlabel/OutputDir_1 /home/dolivierhub/data/outputlabel/OutputDir_2 /home/dolivierhub/data/outputlabel/OutputDir_3 /home/dolivierhub/data/outputlabel/OutputDir_4 ::: 1 149 297 445 ::: 10 159 307 455

cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_1 -f 1 -l 10 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_2 -f 149 -l 159 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_3 -f 297 -l 307 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP
cellprofiler -c -r -p /home/dolivierhub/data/outputlabel/Batch_data.h5 -o /home/dolivierhub/data/outputlabel/OutputDir_4 -f 445 -l 455 --plugins-directory /home/dolivierhub/CellProfiler-plugins --measurements -t /home/dolivierhub/data/tmpCP

We now have the expected output!

Try it on your system with --dryrun to see if you need --xapply or --link, then remove the --dryrun to actually run it.

oh thanks so much! I had gone to the parallel doc but didn’t see that!
It works now actually with both xapply and --link (surprisingly)

I am just curious about memory usage - am only running the UNET and saving labelled images (after a filtering step to get rid of low intensity nuclei) and it is using 50G of RAM and not releasing it. there is no tracking there, just classify unet and label export (but maybe that is another topic on its own).
Thank you a ton @bcimini (2 tacos from me)