Strange CPCluster Problem

Hi there

I am completly new to Cellprofiler and CPCluster.

I have a strange problem running the CP on our cluster. I created the batchfile on a windows xp machine, with sucess.
When I try to run python2.5 CPCluster.py<path to mat file>/Batch_data.mat 1 2 <outputpath> Batch_ y 1000 I get the following error message:

Analyzing set 1. Pipeline: module 1 - LoadImages module 2 - ImageMath module 3 - ImageMath module 4 - ApplyThreshold module 5 - IdentifyPrimAutomatic module 6 - MaskImage module 7 - MaskImage module 8 - ApplyThreshold module 9 - IdentifyPrimAutomatic module 10 - ExpandOrShrink module 11 - MaskImage module 12 - ApplyThreshold module 13 - IdentifyPrimAutomatic module 14 - MeasureObjectIntensity module 15 - MeasureObjectAreaShape module 16 - MeasureObjectIntensity module 17 - ConvertToImage module 18 - ConvertToImage module 19 - ConvertToImage module 20 - SaveImages module 21 - SaveImages module 22 - SaveImages module 23 - CreateBatchFiles executing module 1 - LoadImages Batch Error: LoadImages: Error using ==> LoadImages at 365 Image processing was canceled in the LoadImages module because the directory "No" does not exist. Be sure that no spaces or unusual characters exist in your typed entry and that the pathname of the directory begins with /. (file: /tmp/CompiledCPCluster/CPCluster_mcr/Modules/LoadImages.m function: LoadImages line: 365)

the “No” comes from the load images module setting . If I set this value to yes the errormessage is the same, but the folder not found is “Yes”

I guess there is something wron creating the batch datafile

Any Ideas what can be the problem?

Hi Matze,

Glad to hear you’re trying CPCluster.
Have you tried the supplied BatchRunner.py script? It is our preferred way to run CPCluster.

Other questions:
Are you using an LSF cluster? If so, why are you not using “bsub”?
Which version of CPCluster are you using? 5811Bugfix? Compiled or Developer?

Cheers,
David

Hi David,

I am running the cluster with SGE. (so I would have to change the batchrunner script anyway)

The Problem is that with our setup (python 2.54 + latest scipy) the batchrunner script is not working . (but this is another story)
So I tried to use the CPCluster.py script as a first test.

I run the compiled version of CPCluster extracted from 5811bugfix.zip

many thanks for your help.
Cheers Matthias

Hi Matthias,

OK good, I see. Are you using the same version of CP as CPCluster, i.e. 5811Bugfix? This version is pretty old now, and we could send you a newer compiled version if necessary, or you can compile your own with BuildCellProfiler.m to match your CP version.

But in any case, the error does not sound familiar. It is my inclination to think that the Batch_data.mat is corrupt, so I would re-run CP to generate it afresh, and ensure that there are no errors or warnings that occur in the CP command window. Another local test is to remove CreateBatchFiles and let CP run for at least two cycles to make sure it would run locally without error.

I have not used SGE, nor do we have much experience with it (if at all) but it does not appear to be the problem here.

Let us know how it goes.
Dave

as i said this is just a testbatch I created.

it was running locally without any problems. The CP Version is also the same version. I also tried to do the same thing with the flyimage example from the Website (of course removed the export to excel stuff).
There I get the same error.

maybe I can send you the batch mat file as mail or pm ?

many thanks again

Sure, send either way.
David

Hi Matthias,

I loaded the Batch_data.mat in a compiled 5811Bugfix and it gave many warnings upon loading the pipeline. The I tried the most updated CP release (so-called 7522) and your pipeline loaded without error. In CP, click on Help->Getting Started and what version shows at the top? Something near 7522, or 6025 (which is 5811Bugfix).

Unfortunately, the released version of CPCluster is out of sync with the CP release version. We will rectify this as soon as we can, but let me know what you have.

Dave

so the CP version the mat file created is 1.07522 Compiled

and the CPCluster version is 5811Bugfix

Hi Matthias,

I updated the CPCluster download page to version 7522. http://www.cellprofiler.org/downloadClusterCP.htm
Give that CPCluster version a try with the Batch_data.mat file you sent me.

Dave

Hi ,

many thanks that worked for me… now the cpcluster is working for me.

The next problem I have is the Batchrunner.py script .

Am am using the python 2.5.4 and the latest version of scipy on the cluster system.

If I try to start the batchrunner.py script I get the following Error message:
File “BatchRunner.py”, line 4, in
from scipy.io.mio import loadmat
ImportError: No module named mio

so I changed the import statement to from scipy.io import loadmat, which seems to work.
Bu then I get the Error:
/usr/local/apps/python/lib/python2.5/site-packages/scipy/io/matlab/mio.py:84: FutureWarning: Using struct_as_record default value (False) This will change to True in future versions
return MatFile5Reader(byte_stream, **kwargs)
Traceback (most recent call last):
File “BatchRunner_new.py”, line 21, in
num_sets = batch_info’handles’].Current.NumberOfImageSets
AttributeError: ‘numpy.ndarray’ object has no attribute ‘Current’

any Ideas howto run this script with the latest Scipy version?

I solved my Problem now

it seems that the behavior of the scipy is different in newer Versions…

i changed the following lines and it works for me now


from scipy.io.matlab import loadmat
...

batch_info = loadmat("%(datadir)s/Batch_data.mat"% (locals()),struct_as_record=True)
val = batch_info'handles']
val1 = val[0,0]
val2 = val1'Current']
num_sets = val2'NumberOfImageSets']

Many thanks again to the developers for the fast Bugfixing

Great, glad you got it solved. We currently use python 2.6.1 on our cluster, and scipy 0.7.1.
So let me fully understand - your version of scipy is < 0.7.1? And leaving scipy alone, you changed these lines and it solved your problem?

Thanks,
David

I use on this cluster Python 2.5.4 with scipy 0.7.1 and numpy 1.30

so I changed the lines I mentioned in my previous Post and now its working for me.
I also had to change a few other things in the print statement to get it working with SGE

OK, thanks for posting this.
David