Loading objects into CPA

Hello,

I am trying to load objects identified via an IdentifyPrimaryObjects module into CPA. I am using the ExportToDatabase module to create a properties file and CSV files containing image and object measurement information. However, when I load the generated properties file into CPA and enter the Classifier, I don’t see any of the objects that I am trying to load. I think the problem may have to do with the fact that my CSV files don’t contain any column headers, and I am not sure how to fix this problem. I have attached the Cellprofiler pipeline, the image, and the two CSV files generated by Cellprofiler. I wasn’t able to upload the properties file, but running the pipeline should generate something identical to what I am using.

Thanks in advance for any help you can provide.
SQL_41_1_Object.CSV (1.31 MB)
SQL_41_1_Image.CSV (856 Bytes)
Corless_CPA4.cpproj (430 KB)


Hi,

Please refer to the CPA manual here: cellprofiler.org/linked_files/Do … manual.pdf
and search for this section, titled: "To access multiple comma-separated value (.csv) files produced by CellProfiler: "

Essentially, you need to have CPA access the data as a sqlite database. (Let us know if you have trouble - I honestly haven’t tried this method for years! :smile: Another way to do this would be to create this sqlite database directly from ExportToDatabase, instead of the CSVs.

Hope that helps!
David

[quote=“davidlogan”]Hi,

Please refer to the CPA manual here: cellprofiler.org/linked_files/Do … manual.pdf
and search for this section, titled: "To access multiple comma-separated value (.csv) files produced by CellProfiler: "

Essentially, you need to have CPA access the data as a sqlite database. (Let us know if you have trouble - I honestly haven’t tried this method for years! :smile: Another way to do this would be to create this sqlite database directly from ExportToDatabase, instead of the CSVs.

Hope that helps!
David[/quote]

I decided to create the sqlite database and it worked! Thanks!

Great!
David

One more question if I may – when I try to “score all” in CPA, I get the following error:

An error occurred in the program:
IndexError: list index out of range

Traceback (most recent call last):
File “classifier.pyc”, line 1043, in ScoreAll
File “fastgentleboosting.pyc”, line 152, in PerImageCounts
File “multiclasssql.pyc”, line 214, in PerImageCounts
File “multiclasssql.pyc”, line 181, in do_by_steps
File “multiclasssql.pyc”, line 164, in where_clauses

And then the “Calculating cell counts” window hangs on 0% and the program freezes. Any ideas why this might be happening?

This looks like the same issue: CPA classifier error
Can you follow the link there and take a look? This is likely not an easy debug from where I sit, but you really should inspect your data tables. You can download free sqlite browsing tools like sqlitebrowser.app on Mac, and google search for others on Windows.

Do you have any NULL values in your tables? Empty? Did you save your per-object data as a single table (preferred)?

David

[quote=“davidlogan”]This looks like the same issue: CPA classifier error
Can you follow the link there and take a look? This is likely not an easy debug from where I sit, but you really should inspect your data tables. You can download free sqlite browsing tools like sqlitebrowser.app on Mac, and google search for others on Windows.

Do you have any NULL values in your tables? Empty? Did you save your per-object data as a single table (preferred)?

David[/quote]

David,

I did save my per-object data as a single table…I also went through the table (using CPA’s table viewer option) and couldn’t find any null values. As per the post you linked me to, I am using only one image for classification, and the image has 2000+ objects.

Edit: I also don’t see any scientific notation in the tables.

Thanks

Hello,

I am just following up to see if there is any more information I can provide to help find a solution to this error.

I really appreciate all the help.

Thanks

Hi,

I would hazard a guess (totally unconfirmed!) and say that scoring a database with a single image might not work, i.e. might have a bug.

You can try any or all of these:
(1) Post any text that might be available in the black main CPA gui? The error is in an SQL WHERE clause and I’d like to see it if possible.
(2) You could try a newer trunk build version, found here (though be patient for the page to load): cellprofiler.org/cgi-bin/trunk_build_cpa.cgi
(3) You could try hacking your database to have more than one image, by simply duplicating the Per_Image line.

David

Hi David,

Here is all of the text that is shown in the log from startup up until it freezes:

Logging level: DEBUG
Initialized New Fast Gentle Boosting Classifier
Loading training set from: C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\CPA_files\MyTrainingSet1.txt
[MainThread] Connecting to the database…
[MainThread] SQLite file: C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\CPA_files\DefaultDB1.db
[MainThread] SELECT ImageNumber FROM MyExpt_Per_Image GROUP BY ImageNumber
[MainThread] Connected to database: C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\CPA_files\DefaultDB1.db
[MainThread] SELECT * FROM MyExpt_Per_Object LIMIT 1
[MainThread] PRAGMA table_info(MyExpt_Per_Object)
Ignoring columns: ‘ImageNumber’, ‘ObjectNumber’]
[MainThread] SELECT * FROM MyExpt_Per_Object LIMIT 1
Training set loaded.
[TileLoader_Thread-3] Connecting to the database…
[TileLoader_Thread-3] SQLite file: C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\CPA_files\DefaultDB1.db
[TileLoader_Thread-3] SELECT ImageNumber FROM MyExpt_Per_Image GROUP BY ImageNumber
[TileLoader_Thread-3] Connected to database: C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\CPA_files\DefaultDB1.db
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=255)
ImageReader will use load_using_bioformats from CellProfiler.
[TileLoader_Thread-3] SELECT Image_PathName_One, Image_FileName_One FROM MyExpt_Per_Image WHERE ImageNumber IN (1)
Loading image from “C:\Users\vazimi\Pictures\corless_images\High-quality\High-Quality_Large\1.png-(Colour_1).png”
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=293)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=490)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=147)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=57)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=249)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=129)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=440)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=526)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=375)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=75)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=282)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=42)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=333)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=69)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=96)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=12)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=21)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=309)
[TileLoader_Thread-3] SELECT Nuclei_Cancer_Location_Center_X, Nuclei_Cancer_Location_Center_Y FROM MyExpt_Per_Object WHERE (ImageNumber=1 AND ObjectNumber=252)
Training set updated.
Classifier trained in 0.1s.
Exception fetching new version information from cellprofiler.org/CPAupdate.html: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Skipping table checking step for sqlite
[MainThread] SELECT ImageNumber FROM MyExpt_Per_Image GROUP BY ImageNumber
[MainThread] SELECT MyExpt_Per_Object.ImageNumber, COUNT(MyExpt_Per_Object.ObjectNumber) FROM MyExpt_Per_Object GROUP BY MyExpt_Per_Object.ImageNumber
[MainThread] SELECT MyExpt_Per_Image.ImageNumber FROM MyExpt_Per_Image

I will try out your other suggestions as well. Thanks again for your help on this issue!

-Vahid

I downloaded the bleeding-edge version and it seems to be working. Thanks again for your help!

Good to hear!

Cheers,
David

Thanks for posting the pdf link david :smiley:

Cheers,
Maria