Exception in CellProfiler core processing error - .dv files

Hi,

I’m attempting to input image stacks in .dv file format into CellProfiler. The ability of CP to recognize these files as image stacks and extract metadata from the image file headers is inconsistent. I’m able to extract metadata from .dv files from one directory, but if I move the files to another directory, I get an exception in CP core processing error and the files are not recognized as image stacks. I’ve attached a screenshot showing this error output when I attempt to update metadata. You’ll notice that the same file is listed on lines 2 and 3 in the bottom box - it’s simply in different directories. I have uploaded my pipeline too, but have had no luck uploading a sample .dv file - possibly due to file size, each file is ~8 MB. I haven’t been able to troubleshoot this particular problem myself, so any suggestions would be appreciated.

Thanks,
Christina



LiveDeadBacteriaAnalysis.cpproj (860 KB)

If the forum doesn’t accept the extension, you can rename it to .tif and then try again, I believe. Alternately, you can zip them up and post the zip file.
-Mark

Hi Mark,

Thanks for the quick reply, here’s the zip file with an example image.

Christina
MOI1NoG250k02_R3D_D3D.zip (4.17 MB)

Try the following:

  • NamesAndTypes
    : Under “Image set matching method”, select “Metadata”. For the drop-downs that then appear, select “FileLocation” for all 4 channels. - Groups
    : Enable grouping, and select “FileLocation” as the metadata.
    Let us know if this works.
    -Mark

Hi Mark,

No luck, I’m afraid. I was able to follow your instructions exactly, but I get the “Sorry, your pipeline doesn’t produce any valid image sets as currently configured” error when I click the update button in NamesAndTypes after making those changes.

Any other thoughts?

Thanks,
Christina

Are you able to load and press the Update button in NamesAndTypes for the attached project file? All I did was make the suggested modifications in your project file, keeping everything else the same (including the files in the Images module). Pressing the button works OK for me.

Also, occasionally, information gets cached incorrectly in the Input modules, and sometimes it helps to step through each Input module in turn from the beginning. I.e, make sure Images is showing the right files, press the “Update metadata” button followed by the “Update” button in Metadata and then “Update” in NamesAndTypes.
-Mark
4014_04_07.cpproj (157 KB)

I can load and update NamesAndTypes in the project file you attached as long as I don’t change the input directory. As soon as I change the directory (same files, just moved to different folder), I get the following error again:
Traceback (most recent call last):
File “cellprofiler\gui\moduleview.pyc”, line 1102, in callback
File “cellprofiler\gui\moduleview.pyc”, line 1976, in __on_do_something
File “cellprofiler\settings.pyc”, line 1923, in on_event_fired
File “cellprofiler\modules\metadata.pyc”, line 484, in
File “cellprofiler\modules\metadata.pyc”, line 824, in do_update_metadata
File “cellprofiler\utilities\hdf5_dict.pyc”, line 1304, in add_metadata
TypeError: ‘NoneType’ object is not iterable

The program also generates this error with the original directory, the one that has been working, if I remove the directory from the Images File List and then add it back to the files list. So, I’m guessing that it’s some sort of caching issue as you mentioned. Previously, I was splitting the image stacks in ImageJ prior to my CellProfiler runs rather that using the .dv files directly. I can go back to that approach, if necessary.

I hate to admit it, but I’m at a bit of a loss at this point. I’ve filed a bug report on it here: github.com/CellProfiler/CellPro … ssues/1102. I will post back in this thread when we have a solution. In the meantime, it may be best for separate the image stacks in ImageJ and proceed from there.
-Mark

A few comments:

  • Going back to the screenshot in the 1st post, I notice that you have the following files listed in the 1st 3 rows of the metadata panel:

C:\Users\Christina\Desktop\cpinput\Day3MOI1NoTetNoG02_R3D_D3D.dv C:\Users\Christina\Desktop\cpinput\MOI1NoG250k02_R3D_D3D.dv C:\Users\Christina\Desktop\cpinput\MOI1NoG250k_R3D_D3D.dv
all of which have “None” for the associated information. Are these files listed in Images file list?
The reason I ask is that the normal format of the table is the file itself as a row, followed by the individual channels after, and so on for each file, like this:


But I don’t see this for the 1st three files. What happened to them?

  • If you move the images around on your file system, but don’t update the file list in Images accordingly, it makes sense that updating the metadata will fail (since the files aren’t there). Can you confirm that all files in Images are also listed in Metadata?
  • Lastly, what happens if you just start a new project and start over fresh?

-Mark

Hi Mark,

First I wanted to say how much I appreciate your help with this problem. CellProfiler has been invaluable for my research and I’m using it extensively for a number of different image analysis projects in my lab. This is the first problem I’ve run into that I haven’t been able to solve myself after 1.5 years of frequent CP use, so thanks!

To answer the questions raised on your last post:

  1. Yes, those files are listed in the input file list. I consistently get that “None” result for .dv files where the metadata extraction doesn’t work. If I create a directory with a combination of .dv files that work for metadata extraction and .dv files that don’t work, then update the images file list and attempt to extract the metadata, I see the same result.

  2. I started over with a new project. I dragged and dropped two example .dv files to the Images File List. Moving to Metadata, to selected “Extract from image file headers” and then clicked Update metadata. This error message results:
    Traceback (most recent call last):
    File “cellprofiler\gui\moduleview.pyc”, line 1102, in callback
    File “cellprofiler\gui\moduleview.pyc”, line 1976, in __on_do_something
    File “cellprofiler\settings.pyc”, line 1923, in on_event_fired
    File “cellprofiler\modules\metadata.pyc”, line 484, in
    File “cellprofiler\modules\metadata.pyc”, line 823, in do_update_metadata
    File “bioformats\formatreader.pyc”, line 794, in get_omexml_metadata
    File “cellprofiler\utilities\jutil.pyc”, line 436, in run_script
    JavaException

However, if I follow the same steps with some .dv files that have worked in the past, I get no error and the metadata extracts perfectly.

I’m wondering if this problem has to do with the way the files were transferred to my hard drive. The original .dv files with extractable metadata were written to my hard disk from an external drive formatted as ext3. The files that don’t work were emailed to me or downloaded from Dropbox, maybe something about the file transfer changed the image file headers? I’ll try to track this down this afternoon and will post an update if the file transfer does appear to be the issue.

Thanks again,
Christina

Yes, please let us know if this turns out to be the problem.
-Mark

Hi Christina,
Just checking in: Did you ever find out what the problem was here?
-Mark