CP2 batch jobs export CSV file suffix

Hi folks,

We’re running CP2 on a cluster, analyzing a large set of images in small batch jobs. What I can’t figure out is how to get CP to write the output for each individual job to a CSV file with a suffix indicative of the range of images in the batch. Otherwise each of the CSV files output from parallel jobs overwrite each other :/.

Our version of CP2 is 11086.

For example
I wrote a batch file submission script to submit jobs to the cluster:

[code]import os, re
from optparse import OptionParser

Check that options are present, else print help msg

parser = OptionParser()
parser.add_option("-f", “–first”, dest=“first_img”, help=“start at first image number”, metavar=“first”)
parser.add_option("-l", “–last”, dest=“last_img”, help=“end at last image number”)
parser.add_option("-b", “–batch-file”, dest=“batch_file”, help=“use this batch file”)
(options, args) = parser.parse_args()

cmd = “submitjob”,“36”,"-m",“4”,
“python”, “/usr/cp2/CellProfiler/CellProfiler.py”
" -p",str(options.batch_file),
"-c", “-r”, “-b”,
"-f", str(options.first_img),
"-l", str(options.last_img)]

cmd = ’ '.join(cmd)
p=os.popen(cmd,‘r’)
output=p.read()
exit_code=p.close()

if output:
match = re.search(r’([0-9]+).’,output)
outfile = open(“cp2_batch_output.out”,‘w’)
if len(match.groups()) > 0:

[/code]

I’ll submit a job like this: $ python cp2_batch_runner.py -b <batch file> -f 1 -l 1000
Which will analyze images 1 to 1000. It will generate output files like this: DefaultOut_cell.csv, DefaultOut_Image.csv, DefaultOut_Experiment.csv
What I’d like is for that job to generate output files like: DefaultOut_1_1000_cell.csv, DefaultOut_1_1000_Image.csv, DefaultOut_1_1000_Experiment.csv

I suspect that this can somehow be set up in ExportToSpreadsheet (or ExportToDatabase) modules, but I can’t figure out where to specify this. I read something about metadata tokens in the CP2 manual, but am not sure how I could apply them in this case.

Here’s the outline of our pipeline:

LoadImages IdentifyPrimaryObjects OverlayOutlines SaveImages Crop IdentifyPrimaryObjects OverlayOutlines SaveImages RelateObjects MeasureImageIntensity MeasureImageSizeShape MeasureTexture MeasureImageQuality ExportToSpreadsheet

Let me know if you want me to upload the .mat file, if required.

Thanks,

Lee.

Hi Lee,

Unfortunately, ExportToSpreadsheet is not set up to insert tokens in this way. Fortunately, ExportToDatabase is. :smiley:

If you select the “MYSQL/CSV” option in ExportToDatabase, it will output .csv’s with the first and last image number as part of the file name. The downside is that the column headers are not part of the .csv’s themselves but included as the accompanying .sql file that is also output. The premise is that you would run the .sql file and it will create database tables with the specified columns and then upload the data. However, if you are willing to parse the .sql file, perhaps you can extract the headers and re-attach them to the .csv’s.

Regards,
-Mark

Perfect, thanks Mark.

Lee.

[quote=“mbray”]Hi Lee,

Unfortunately, ExportToSpreadsheet is not set up to insert tokens in this way. Fortunately, ExportToDatabase is. :smiley:

If you select the “MYSQL/CSV” option in ExportToDatabase, it will output .csv’s with the first and last image number as part of the file name. The downside is that the column headers are not part of the .csv’s themselves but included as the accompanying .sql file that is also output. The premise is that you would run the .sql file and it will create database tables with the specified columns and then upload the data. However, if you are willing to parse the .sql file, perhaps you can extract the headers and re-attach them to the .csv’s.

Regards,
-Mark[/quote]