Missing measurement file after job completed on CPCluster

Usage & Issues
#1

Hello,

I have been using CellProfiler for almost a year on a workstation, and I recently moved to using it on a cluster. I am not using batch processing yet, and I noticed that when I was testing my pipeline, the measurement file .mat would not produced at all (but it works on a computer). What could be the reason for this?
Would that be because the .mat files are not produced when running on a cluster (here I am just trying to test the cluster, so I am only requesting a single core) and therefore forcing us to use batch processing anyways?

All the other output files (the output database, the CPA properties file etc…)

Here is the end of the PBS script I am using:

End PBS preamble

Show list of CPUs you ran on, if you’re running under PBS

if -n “$PBS_NODEFILE” ]; then cat $PBS_NODEFILE; fi

Change to the directory you submitted from

if -n “$PBS_O_WORKDIR” ]; then cd $PBS_O_WORKDIR; fi

Put your job commands here:

Create a subdirectory for CellProfiler’s output files:

mkdir output-{PBS_JOBID} echo "Putting output files into the subdirectory output-{PBS_JOBID}"

Run CellProfiler:

CellProfiler.py --do-not-build --do-not-fetch --run-headless --run
–image-directory="./" --output-directory="./output-${PBS_JOBID}"
–pipeline=“pipeline.cp”

Thanks a lot for your help!

Remy

#2

Hi Remy,

If I understand correctly, it looks like you simply don’t have the optional “output file” in your command. This is the help from “CellProfiler.py”

[quote]$ python CellProfiler.py -h
Usage: CellProfiler.py [options] ])
where is the optional filename for the output file of measurements
when running headless[/quote]

If you include an “<OUTPUT_FILE>.mat” file after your command then there will be a Matlab handles structure output.
NOTE THOUGH that unless you are using Matlab as your post hoc analysis tool, this step is unnecessary! The same information is encompassed by the pipeline and the database or CSV files.

[quote]All the other output files (the output database, the CPA properties file etc…)
[/quote]

I think your post was truncated. Did you mean that they are all output OK?

Hope that helps,
David

#3

Hi David,

Indeed, all the other files are OK, and therefore I wrongly assumed that the option for the .mat file was already included in the pipeline. I added the option at the end of the command line, and it works very well. Since I am currently more familiar with MATLAB, it is easier for me to look at the .mat files to test the jobs submitted on the cluster.

I really appreciate your help,

Thanks,

Remy

#4

Great! Glad to help.
David