Optimizing analysis and cluster submission

I’ve attached the pipeline and some sample images that I have been trying to run the pipeline on. I think I have the pipeline pretty well setup, but I’m sure there’s always room for improvement.

One of my main concerns with the pipeline itself is that due to the images being 4x and there being only so fine an adjustment I can make on primary object size, it seems that I cannot find the correct settings for CP to distinguish between real nuclei and debris. Additionally, I tried to count the numbers of nuclei that are in one cell type or another (neurons or other, basically) without much success.

On the cluster I’m attempting to run headless and submit the job via a .lsf script. The problem I run into there is that if I submit the job as one run, it does not get split up to take advantage of the parallel processing, and CP errors out at image number 168. I would like to use the -f -l flags to group the images I’m submitting, but then the output filenames overwrite each other. I tried outputting to a database, but I got a “expected exactly 4 inputs and only received 3” error for that, so I ended up giving up. Is there a way I can break the jobs up by row and break that out in my metadata to generate unique filenames for each row of each plate?

There seem to be more flags available when submitting a job command line than I can seem to find documented. I’m not sure if it’s a lack of documentation or if I’m just doing a terrible job searching.
WT.zip (6.18 MB)
FXS.zip (6.26 MB)
WntAssayPipe-08192011.cp (17.7 KB)

I’m piecing this together from your description and your pipeline, but it seems that you want to use grouping to get things to work.

You have defined ‘Plate’ metadata and are writing spreadsheets with metadata substitution based on the ‘Plate’ metadata tag. I infer that you want each spreadsheet to contain measurements from each plate? If so, you need to enable grouping in LoadImages by checking the “Group images by metadata?” setting and selecting ‘Plate’ in “Specify metadata tags to group by?” By doing this, you can then use the “-g GROUPS” flag to submit each batch, where GROUPS is the name of each plate metadata tag. CP will then divide up the image sets accordingly so that each node receives a plate.

If you want to create plate-row spreadsheets, you need to create a Row metadata tag from the filename, add the row tag to the output filename in ExportToSpreadsheet, select both Plate and Row in metadata to group by (shift-click allows you to select multiple items), and specify both Plate and Row with the -g flag.

Hope this helps!
-Mark