Export to database does not work

Hi,

I have analyzed a screen on a cluster which does not allow me to write to a database directly. I wanted to export the data afterwards to a database using the export to database functionality, but it does not work. The tables are created, but they are empty and contain no data. Furthermore, the property file is also exported to the database instead of being written to the specified location.
Bye,
Marc

Hi Marc,

By your description, I take it that you mean using ExportToDatabase as a data tool (as opposed to as a module) doesn’t work?
-Mark

Hi Mark,

yes indeed. As a module it works just fine, it’s the data tool that is misbehaving.
Marc

Hi Marc,
Is the behavior similar to that posted here? Mergeoutputfiles does not work. I.e., works in Windows but not on the Mac?
-Mark

Hi Mark,

Happy New Year.
Not quite similar. The export to db appears to work under Windows, but the tables created are empty. The Mac version doesn’t do anything.
Bye,
Marc

We are not able to replicate this error on our Macs here, at least thus far. Are there any errors messages that are output?
-Mark

Hi Mark,

Things are getting stranger. When I tried exporting once more, CellProfiler quits which it didn’t do in the past. I have uploaded the error message at the link below. It’s most likely a Mac problem and not a CP problem. I have also uploaded the data I’d like to import into my MySQL db. It is big, so maybe that is an issue. It would be great if you could try importing that to see if that works with your Macs too.
idisk-srv1.mpi-cbg.de/~bickle/

Thank you,
Marc

Hi Marc,

Would mind producing the CellProfiler output from your console, as well? If you’re running compiled CP on the Mac, you should:

Open Console.app
Select “All Messages” in the left pane
Type “CellProfiler” into the upper-right “String Matching” field
Run CellProfiler
Try exporting
Save the output in Console.app and send post it here.

If it’s failing or crashing, this may give us some insight as to why.

Regards,
-Mark

Hi Mark,

sorry for not replying earlier, I had a lot of fish to fry.
I tested the export function on the new release and it failed again (see console message below). I get the new table, but it contains no data. I also add a screenshot of the module setting, just to check I don’t do anything stupid. I hope the resolution is sufficient.
I will be posting also other issues with the new release, so talk to you later. Sorry.
Marc

07/03/2012 16:48:54.016 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: wx._core.PyAssertionError: C++ assertion “wxAssertFailure” failed at /BUILD/wxPython-src-2.8.12.0/src/generic/listctrl.cpp(1876) in OnPaint(): unknown list item format
07/03/2012 16:50:13.775 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: /Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/gui/moduleview.py:1584: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
07/03/2012 16:51:52.528 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: Traceback (most recent call last):
07/03/2012 16:51:52.528 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/gui/datatoolframe.py”, line 202, in on_run
07/03/2012 16:51:52.551 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: self.module.run_as_data_tool(workspace)
07/03/2012 16:51:52.551 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 966, in run_as_data_tool
07/03/2012 16:51:52.560 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: self.run(workspace)
07/03/2012 16:51:52.560 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 983, in run
07/03/2012 16:51:52.560 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: pixels = image_set.get_image(name).pixel_data
07/03/2012 16:51:52.560 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/cpimage.py”, line 524, in get_image
07/03/2012 16:51:52.566 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: image = self.get_image_provider(name).provide_image(self)
07/03/2012 16:51:52.566 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfiler2.0.app/Contents/Resources/lib/python2.6/cellprofiler/cpimage.py”, line 570, in get_image_provider
07/03/2012 16:51:52.566 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: return providers[0]
07/03/2012 16:51:52.566 [0x0-0x2b92b9].org.pythonmac.unspecified.CellProfiler2.0: IndexError: list index out of range


Hi Mark,
I’m still not managing to export to a database. I am merging the output from our cluster into a single result file and then exporting to a local mysql db, but without any success. On the following link you can find the output from our cluster and the merged result file. Below is the output I got from the console.
It would be great if you could figure out what is going on.
Thanks,
Marc

idisk-srv1.mpi-cbg.de/~bickle/

15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: Failed to write measurements to database
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: Traceback (most recent call last):
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfilerTrunk.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 2076, in write_data_to_db
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: TypeError: object of type ‘NoneType’ has no len()
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: Previously displayed uncaught exception:
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: Traceback (most recent call last):
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfilerTrunk.app/Contents/Resources/lib/python2.6/cellprofiler/gui/datatoolframe.py”, line 206, in on_run
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfilerTrunk.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 1306, in run_as_data_tool
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfilerTrunk.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 1357, in run
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: File “/Applications/CellProfilerTrunk.app/Contents/Resources/lib/python2.6/cellprofiler/modules/exporttodatabase.py”, line 2076, in write_data_to_db
15/05/2012 17:56:13.398 [0x0-0x1a01a].org.pythonmac.unspecified.CellProfiler2.0: TypeError: object of type ‘NoneType’ has no len()

Hi Marc,
If you use the latest CP build, is this issue still present?
-Mark

Hi Mark,

I am back onto this project, our user has generated more data which we analyzed on our cluster. I have now used the bleeding edge version cp 2.0.0(rev 30f8dca) to merge the output files and to try to import into a mysql database. The error I get during the import is

Traceback (most recent call last):
File “cellprofiler\gui\datatoolframe.pyc”, line 206, in on_run
File “cellprofiler\modules\exporttodatabase.pyc”, line 1317, in run_as_data_tool
File “cellprofiler\modules\exporttodatabase.pyc”, line 1368, in run
File “cellprofiler\modules\exporttodatabase.pyc”, line 2042, in write_data_to_db
KeyError: Mean_Nuclei_AreaShape_Center_X

Can you please help?
Thanks,
Marc

Hi Marc,

I managed to replicate this issue myself. I believe the cause is that the data in the output file is from a previous version of CP, prior to the version that you’re using to upload to the database. On per-image aggregation, I think it’s looking for a measurement that exists in the bleeding edge version but not in the output file itself.

If I’m right, the workaround is to uncheck the box for per-image aggregation (mean, median, std). I gave this a shot and it seemed to export successfully.

In the meantime, I’ve filed it as a bug issue on GitHub.

Regards,
-Mark

Hi Mark,

thanks for looking into this issue. I have now tried to export to the db without the mean per image and I receive a new error message:

Traceback (most recent call last):
File “cellprofiler\gui\datatoolframe.pyc”, line 206, in on_run
File “cellprofiler\modules\exporttodatabase.pyc”, line 1317, in run_as_data_tool
File “cellprofiler\modules\exporttodatabase.pyc”, line 1368, in run
File “cellprofiler\modules\exporttodatabase.pyc”, line 2085, in write_data_to_db
File “cellprofiler\measurements.pyc”, line 556, in get_measurement
File “cellprofiler\utilities\hdf5_dict.pyc”, line 269, in getitem
AssertionError

I hope this helps?
Marc

Hmmm, I’ve just gotten this error as well. I’m adding it to the bug report.
-Mark

Hi Marc,
We’ve recently taken a crack to solving this issue. Can you download our latest trunk build and try again?
Regards,
-Mark

This issue has been resolved in CellProfiler 2.1 and later, new releases of which can now be downloaded from http://cellprofiler.org.

This issue has been resolved in CellProfiler 2.1 and later, new releases of which can now be downloaded from http://cellprofiler.org.