Failed to complete post_run processing for module ExportToDatabase

Hi,

I recently set up a test MySQL database (v5.6.41) on AWS so that in the future I can use multiple copies of CellProfiler on our cluster. I performed a run of 308 images, and when I check the database via MySQL workbench it looks fine. However, right after finishing processing the last image in this run, I receive this error:

Failed to complete post_run processing for module ExportToDatabase.
Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 2194, in post_run
module.post_run(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 2376, in post_run
self.write_properties_file(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 3635, in write_properties_file
all_properties = self.get_property_file_text(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 4081, in get_property_file_text
“”" % (locals())
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xff in position 254: ordinal not in range(128)

This results in no workspace or properties files. When I tested it on my PC with a single image, it worked just fine, though I’ve not tried it with 308 images. However, my colleague had the same error when he tried to run a similar pipeline on his PC, so I’m wondering if it might be something to do with the way the MySQL is configured.

I’ve attached the pipeline in case that might be the issue. Does anyone have any ideas?

Cheers,

Ed

Hi Edward

Could you try to rename the experiment name so it does not start with a number? I don’t know if this is the issue though.
I have no problem exporting the analysis of thousands of images to a MySQL database in CellProfiler 2.1.1, but I have not attempted in 3.0.0.

Regards,
Esben

Hi Esben,

Thanks for your suggestion! Unfortunately it did not work and we still have the same issue, though I can of course create the properties file myself.

Thanks,

Ed

I observe the exact same error in 3.1.8, using ExportToDatabase (running mysql locally).

properties = self.get_property_file_text(workspace)
  File "/home/username/anaconda3/envs/cp/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py", line 4080, in get_property_file_text
    """ % (locals())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 254: ordinal not in range(128)

The tables appear to have been created perfectly fine in MySQL Workbench. (Ubuntu 16.04). I’ve tried multiple different latin, utf and ascii encodings. I’ve also tried to make it work with an empty database on a GCP instance with no luck.

Does anyone have an answer to this? It’s still causing me problems.

Best,

Ed

Hmmm, I’m guessing this is related to some other Unicode bugs we’ve been squashing lately; I will make an issue.

Would it be possible for you to check (just with one or two image sets) if this works correctly if you just try to make an SQLite rather than MySQL database? That will help narrow it down some.

Hi Beth,

It does indeed work fine if I export to an SQLite database. I think I am now going to set up my own MySQL server locally, so hopefully this will no longer be an issue.

Cheers!

Ed

In the meantime I’ve opened an issue to look into it, but as long as the data is written then hopefully in the meantime you can perhaps reuse/rework some existing properties files to get up and running on CPA. LMK if you need assistance with that.

1 Like

Hi,

Just wondering if there’s been any update on this issue? I have set up my own MySQL server and I’m still running into the same problem.

Cheers,

Ed

Just to say I have got it to export something to MySQL in CellProfiler 2.2 it seems, however not CP 3.1.8. If this could be fixed for CP 3.1.8 that would be great as I need the improved functionality.

Cheers,

Ed

I have now tested it in several versions: it works fine in 2.2 and 3.0.0, but not 3.1.5 or 3.1.8, so the error must be creeping in at that point.

Ed

Have there been any updates with this? I’m still running into the same issues. If this cannot be fixed, can anyone point me to where the 2.2 version is held in github because I can’t see it. I’ll try to get that set up on the cluster instead.

Thanks,

Ed