Unable to generate batch file

Hi,

I’m trying to generate a batch file with the attached pipeline, but when I click on “Analyze Images”, CP seems stuck in the following state (see attached photo). I’ve tried turning on debugging at the command line (no extra output), also seeding in pdb breakpoints in relevant modules (loadimages, createbatchfiles), but none are triggered. Could you try to send me email, or give me an idea about the execution path for creating batch files so I can try to debug this? Or is it a problem with the pipeline?

Thanks,

Lee.



2011_08_18_10415_IPANuclei_IPANucleoli_ISACells_RelateNucleoliToNuclei_SaveImages=WORKS_WithMeasurementsCondensed_ExportToDB_CreateBF.cp (19.1 KB)

I made generic batch scripts that will split the image batches equally between multiple cores in the post listed below. Download and execute the batch script in the images folder that you want to analyze.

Hi derek,

Thanks for this, but my problem is that I can’t get the pipeline to output the .h5 batch file. Until I can, I’ll have to wait to try your scripts.

Thanks,

Lee.

In the LoadImages module, you’re asking to process files in all subdirectories. Is it possible that it’s a deep directory hierarchy? I don’t do anything (I think) to prevent traversal of links, so there’s even a possibility that it could be repetitively processing the same directory.

You might try processing some of the subdirectories (In LoadImages, change “Analyze all subfolders within the selected folder?” to “Some”, then press the button below and select just a few directories).

–Lee

Hi Lee,

It’s a shallow but broad hierarchy: there are 58 subfolders under the default input, but each is a leaf node in the directory tree. There aren’t any links in the directory that could be messing things up. I’ll try re-running the same pipeline with just a few subfolders selected to see if it makes a difference, thanks.

Lee.

[quote=“LeeKamentsky”]In the LoadImages module, you’re asking to process files in all subdirectories. Is it possible that it’s a deep directory hierarchy? I don’t do anything (I think) to prevent traversal of links, so there’s even a possibility that it could be repetitively processing the same directory.

You might try processing some of the subdirectories (In LoadImages, change “Analyze all subfolders within the selected folder?” to “Some”, then press the button below and select just a few directories).

–Lee[/quote]

Hi Lee,

I tried generating a batch with 9 of 58 subdirectories, and it seems to have worked! It takes a lot longer to generate than I expected, but I can interrogate the resulting batch file just fine (according to the code you posted in response to my other question http://forum.image.sc/t/how-to-extract-the-number-of-image-sets-from-h5-batch-file/16879/1). Maybe what I thought was a hung process was in fact the amount of time required to generate this enormous batch. I’ll look into this more closely.

Thanks,

Lee.

[quote=“LeeKamentsky”]In the LoadImages module, you’re asking to process files in all subdirectories. Is it possible that it’s a deep directory hierarchy? I don’t do anything (I think) to prevent traversal of links, so there’s even a possibility that it could be repetitively processing the same directory.

You might try processing some of the subdirectories (In LoadImages, change “Analyze all subfolders within the selected folder?” to “Some”, then press the button below and select just a few directories).

–Lee[/quote]

I generated a batch file for a larger subset of the data, 867 flex files totalling 13872 images, and the results are as follows:

Wall Clock Time: begun at 12:43 ended no later than 3:07.
According to the /usr/bin/time: real: 146m, user: 65m, sys: 14m

Is this roughly in line with what you folks achieve on your cluster?

Thanks,

Lee.

Hi Lee,

We also find inordinately long times with our batch file creation with the new HDF5 scheme and are currently working on this issue.
-Mark

Ok, thanks for letting me know Mark.

Lee.

[quote=“mbray”]Hi Lee,

We also find inordinately long times with our batch file creation with the new HDF5 scheme and are currently working on this issue.
-Mark[/quote]