Load images & load data module help

Hi, I wondered how best to load treatment / dose values with ‘load data’ if Im using load images to load my images, and using regex’s to extract the plate, well, field metadata from the filenames… Seems you can use either ‘load images’ or ‘load data’, but not easily both… Do I need to add the plate info to the compound dose table and only use load data?

best regards,

Keith

Hi, a precision, I am trying to load my csv table with the Data_Dose & Data_Control columns and merge with the images from the LoadImages step.
First module in pipeline is LoadImages, second LoadData.

I have thus added a column in my dose csv file for Metadata_Well, this is the column that should match to the metadata being extracted in the first LoadImages step.

However, I get a warning in the interface (Name of file is highlighted yellow), and warning text pops up ‘Your pipeline has a LoadImages and a LoadData module. Best practice is to have only a single LoadImages or LoadData module.’… Then when I click on view CSV file contents, I get a crash:

Previously displayed uncaught exception:
Traceback (most recent call last):
File “cellprofiler\gui\moduleview.pyc”, line 919, in callback
File “cellprofiler\gui\moduleview.pyc”, line 1637, in __on_do_something
File “cellprofiler\settings.pyc”, line 1594, in on_event_fired
File “cellprofiler\modules\loaddata.pyc”, line 647, in browse_csv
File “encodings\utf_8.pyc”, line 16, in decode
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe9 in position 84: unexpect
ed end of data

Also, I am using the bleeding edge version to get the imageJ2 plugin possibility…

regards, Keith

Hi Keith,

LoadImages should not be used in conjunction with LoadData, but rather you should use only one or the other; having two potential sources of image data can lead to conflicts that are not easy to resolve.

As a workaround, you should use LoadData to load a csv that contains the dose, control and image location and path information (similar to the SBS BioImage data from our Examples page). One approach to generating the file/path information is to run LoadImages followed by ExportToSpreadsheet on the image data to produce a csv.

[quote=“KeithN”]Then when I click on view CSV file contents, I get a crash:
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe9 in position 84: unexpect
ed end of data[/quote]

You may want to check whatever editor you are using to create/modify the csv, in order to confirm what type of encoding is being applied. Saving the file as ASCII is the safest way to ensure the compatibility of the csv.

Regards,
-Mark

Hi Mark, I fixed the problem with the .csv file, it was excel’s fault…

However, I’m still a bit confused, in the SBS example, you have a LoadImages module just before a LoadData module which is what I want to do (read the images and parse the filenames to get plate metadata information, and read a csv file with rows per well, containing the compound and dose information)…

However, you state above that one should not have a LoadImages module & LoadData module in same workflow, so why does the SBS example do it this way?

And for my workflow, unlike the SBS example that works, when I add the LoadData step to my workflow, I get downstream modules that go to the red exclamation mark error condition - for example, the ‘MeasureImageQuality’ module, has the first parameter in red ‘calculate metrics for which images’: and the error is reported as ‘exception in cpmodule.test_valid list index out of range’…
Any ideas on why I get this error?

best regards,

Keith

Hi Keith,

Good point; you got us there. :smile: Originally, this was done for consistency with the CP 1.0 example pipeline, but at this point, the example should probably be revised.

[quote=“KeithN”]And for my workflow, unlike the SBS example that works, when I add the LoadData step to my workflow, I get downstream modules that go to the red exclamation mark error condition - for example, the ‘MeasureImageQuality’ module, has the first parameter in red ‘calculate metrics for which images’: and the error is reported as ‘exception in cpmodule.test_valid list index out of range’…
Any ideas on why I get this error?[/quote]

Not sure about this one; I may have to see the pipeline itself. Also, in MeasureImageQuality what happens if you change “All loaded imgaes” to “Select…”? What does it show, and can you select your images explicitly?

Cheers,
-Mark

Hi Mark, I tried changing to ‘select’ in ‘MeasureImageQuality’, and the ‘select the images to measure’ text is in red with nothing to select…
can I send you the workflow & plate layout datafile?

regards, Keith

Yes, please post them as an attachment to this forum thread.
-Mark

Hi Mark, I attach the pipeline and the data file I try to load with ‘LoadData’. Again, several modules downstream go red once I configure ‘LoadData’ to point to this CSV file. The CSV file looks fine when I open it with the ‘View’ button in the LoadData cellprofiler module.

best regards,

Keith

[attachment=1]doses_image_mapping.csv[/attachment]
[attachment=0]mito_pipe_load_data_02_08_12.cp[/attachment]
mito_pipe_load_data_02_08_12.cp (81.2 KB)
doses_image_mapping.csv (728 Bytes)