Script help for submitting multiple image sets to SGE cluster

Hi all,

Anyone had experience submitting multiple image sets on a SGE cluster? I’m a newbie and struggling to get this automated. I have all the batch commands to split up the image sets, but I’m not sure how to get them into a script that will submit them as separate processes to the cluster.

Cheers,

Ed

Got a fair bit of experience submitting cellprofiler jobs to a SGE cluster at my old group, it’s not exactly simple and a lot of it will be specific to your setup.

The general idea was to generate a csv file per task that was suitable for the LoadData module, and then create a cellprofiler command per task in a plain text file, then loop through these lines as an SGE array job with the $SGE_TASK_ID variable.
The other approach is to have one big csv file suitable for LoadData, and use the cellprofiler commands with the -f and -l flags, although I found Cellprofiler isn’t a fan of huge files for LoadData.

There’s python program here to automate the process on our setup, which might give you some ideas.

1 Like

That’s amazing thank you so much! I’ll give it a try.

Cheers!

Ed

1 Like