Problem loading batch_data.mat

cellprofiler

#1

Hi,

I have some trouble reading batch_data.mat into Python.

running

> python testBatchRunner.py /g/screening/tischer/data/CPClusterInOut hallo 100 yes 100

gives the following complaints:

**/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio.py:84: FutureWarning: Using struct_as_record default value (False) This will change to True in future versions
return MatFile5Reader(byte_stream, **kwargs)
Traceback (most recent call last):
File “testBatchRunner.py”, line 20, in
batch_info = loadmat("%(datadir)s/Batch_data.mat"%(locals()))
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio.py”, line 111, in loadmat
matfile_dict = MR.get_variables()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 354, in get_variables
res = getter.get_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 395, in get_array
arr = self.get_raw_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 481, in get_raw_array
item.dict[name] = self.read_element()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 348, in read_element
return self.array_reader.read_element(*args, **kwargs)
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 241, in read_element
return self.current_getter(byte_count).get_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 395, in get_array
arr = self.get_raw_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 481, in get_raw_array
item.dict[name] = self.read_element()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 348, in read_element
return self.array_reader.read_element(*args, *kwargs)
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 241, in read_element
return self.current_getter(byte_count).get_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 395, in get_array
arr = self.get_raw_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 449, in get_raw_array
result
= self.get_item()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 453, in get_item
return self.read_element()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 348, in read_element
return self.array_reader.read_element(*args, *kwargs)
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/mio5.py”, line 241, in read_element
return self.current_getter(byte_count).get_array()
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 396, in get_array
return self.array_reader.processor_func(arr, self)
File “/g/screening/tischer/software/lib/python2.6/site-packages/scipy/io/matlab/miobase.py”, line 310, in func
arr = np.empty(n_dims, dtype=‘U%d’ % dims-1])
TypeError: data type not understood

i tried to attached my Batch_data.mat file (not sure whether it worked, because i can’t see it anywhere);
in case you manage to read it into Python, could you maybe please sent me the files in the scipy/io/mio folder that did the job?! i guess i could just replace my files with yours and it should be fine…

best,
Christian.**


#2

I sent you instructions on how to ftp it to us. You can upload .mat files, but it may have exceeded the size limit.

One possibility is that we are using python 2.5.2, not 2.6. You can try ‘use -q python-2.5.2’ if your system has the ‘use’ utility.
I used the Superpack python installer for scipy and a few other python modules (macinscience.org/?page_id=6). That installs scipy 0.7.

Let me know if you upload it and I can test Batch_data.mat here.
David


#3

thanks; I uploaded the file.

And entering “use” at our system seems not to work (“command not found”).

best, Christian.


#4

Your Batch_data.mat loaded fine here (I don’t have your images, of course, so it errored when LoadImages ran, but it loaded fine). I checked the version of scipy on our cluster and it is actually 0.6.0. The newest version of scipy that you installed, 0.7.0 is beta and we are not using it here yet. I’d suggest uninstalling 0.7 and try 0.6.

Let us know how it works.
David


#5

thanks a lot for testing it!

I actually think it may be the easiest (for me at least) when you would sent me the files in your “mio” directory within the “scipy” installation.
I think I could simply replace mine with those ones withouyt having to un- and re-install scipy.

otherwise: as i am not at all experienced with Linux/Unix, could you maybe give me a hint as to how to uninstall “scipy” and maybe also “python” in a clean way?!

sorry for my ignorance :wink:

best, Christian.


#6

The ‘mio’ directory has a different directory structure between the two versions, (e.g. no ‘matlab’ directory in 0.6), so I suspect there are a lot of changes between them. So I highly suggest that you cleanly uninstall 0.7 and install 0.6 (and I can’t guarantee python2.6 will work either). Your local sysadmin, or our good friend Google, will have to help you with that!

Good luck,
David


#7

Hi!

using Python2.5 and scipy0.6 did the job!

thanks!


#8

Ah, good. We will add these software requirements to clarify our help files.

Best,
David