Grouping a lot of images problem

Hi,

I’m trying to group my images by plate id.
I have 8 folders containing 384 images each, when i try to group 2 folders it still works but i try the whole 8 folders
the group tab doesn’t show the preview lists(i see blank) and when i try to run the protocol it fails with error of the grouping model.

Am i doing something wrong or it’s a known problem?

Hi,
Can you do a copy/paste of the error that CellProfiler is producing when you try to run the pipeline? Also, if you could post your project file, that might be helpful too.
Regards,
-Mark

Hi Mark,

The error i get is this:
“the pipeline cannot be started because of a configuration problem in the Groups module:
the groups module is misconfigured. “Plate” was chosen as one of the metadata tags, but that metadata tag is not defined in the Metadata module.”

I’ve attached the project to.

thanks for the help,
Rotem
problem_in grouping.cpproj (542 KB)

OK, it appears that the problem is that channels are missing in some cases, and NamesAndTypes is balking (similar issue reported here: github.com/CellProfiler/CellProfiler/issues/714). If using “Order” for matching in NamesAndTypes, it doesn’t quite know what to do in this case.

The workaround (which I would recommend regardless) is to use “Metadata” to match in NamesAndTypes. For the 4 channels that appears, select “Plate” for the drop-downs for all four, then click the “+” button to add another row, then select “Well” for all four drop-downs. Click the “Update” button below the divider to confirm the image sets are created properly (which it should be). At this point, the Groups should now work, and you will see that a few plates have 383 rather than 384 wells available for grouping.

One additional issue: For your pipeline, I don’t think you’ll actually need grouping to output results by Plate, if that’s what you’re trying to do.

Regards,
-Mark

Thank you Mark, it worked!
I use grouping for the ‘-get-batch-commands’ flag, cause i’m running in parallel.
I doing that this way because the -f and -l flags don’t work.

Thank you,
Rotem

What error are you getting when you use -f and -l? We use it quite regularly here, so we’d like to know if it’s a bug…
-Mark

Hi Mark,

when i create the batch file it doesn’t generates for me a .mat file only .h5 so i use it in the command line(mayebe this is the problem).
anyway the command i run is: 'cellprofiler -p output/Batch_data1_3.h5 -c -r -f 1 -l 3’
and the error i get is :
‘Failed during initial processing of /tmp/Cpmeasurementsr_eVAt.hdf5
Traceback (most recent call last):
File “/usr/cellprofiler/src/CellProfiler/cellprofiler/utilities/hdf5_dict.py”, line 293, in init
maxshape = (None, ))
File “/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/group.py”, line 94, in create_dataset
dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
File “/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/dataset.py”, line 76, in make_new_dset
tid = h5t.py_create(dtype, logical=1)
File “h5t.pyx”, line 1379, in h5py.h5t.py_create (h5py/h5t.c:12683)
File “h5t.pyx”, line 1451, in h5py.h5t.py_create (h5py/h5t.c:12533)
TypeError: Object dtype dtype(‘object’) has no native HDF5 equivalent
Error loading HDF5 output/Batch_data1_3.h5
Traceback (most recent call last):
File “/usr/cellprofiler/src/CellProfiler/cellprofiler/measurements.py”, line 1703, in load_measurements
image_numbers = image_numbers)
File “/usr/cellprofiler/src/CellProfiler/cellprofiler/measurements.py”, line 269, in init
image_numbers=image_numbers)
File “/usr/cellprofiler/src/CellProfiler/cellprofiler/utilities/hdf5_dict.py”, line 293, in init
maxshape = (None, ))
File “/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/group.py”, line 94, in create_dataset
dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
File “/usr/cellprofiler/lib/python2.7/site-packages/h5py/_hl/dataset.py”, line 76, in make_new_dset
tid = h5t.py_create(dtype, logical=1)
File “h5t.pyx”, line 1379, in h5py.h5t.py_create (h5py/h5t.c:12683)
File “h5t.pyx”, line 1451, in h5py.h5t.py_create (h5py/h5t.c:12533)
TypeError: Object dtype dtype(‘object’) has no native HDF5 equivalent
So sorry. CellProfiler failed to remove the temporary file, /tmp/Cpmeasurementsr_eVAt.hdf5 and there it sits on your disk now.
stopping worker thread 0
stopping worker thread 1
stopping worker thread 2
stopping worker thread 3
stopping worker thread 4
stopping worker thread 5
stopping worker thread 6
stopping worker thread 7
stopping worker thread 8
stopping worker thread 9
stopping worker thread 10
stopping worker thread 11
stopping worker thread 12
Exiting the JVM monitor thread’

there a problem in the h5 module or something…

Thanks,
Rotem

Have you tried using the flags since I posted the workaround?
-Mark

I don’t understand which workaround you are refering to.
I’m using the version of cellprofiler available on your website, i’m not pulling the source from github or something like that

I’m referring to using metadata matching in NamesAndTypes re: my prior post:Grouping a lot of images problem.
-Mark