Error using ScatterPlot, Histogram, DensityPlot

Hi guys,

I’m using CPA 2.0 r8634M on Windows, and I’m having a bit of trouble visualising data generated by CP 2.0.

I think I’ve been able to populate my mysql database with the proper data from the .CSV files generated by CP 2.0. I had to enter the Windows path to my .CSV files into the properties file, as per http://www.cellprofiler.org/forum/viewtopic.php?f=12&t=464

I can use the classifier and image viewer properly in CPA 2.0, but clicking on ScatterPlot, Histogram, DensityPlot results in an error (console dump below) - ValueError: list.index(x): x not in list

Any idea how I can get these functions to work?

I have tried using the SQLite database as well, but CPA 2.0 still seems to try to connect to mysql, even when this is commented out in my properties file. In any case, don’t worry about that for now.

Thanks for your help - I’m otherwise really impressed with what the software will be able to do for me.

C:\Drew\CPAnalyst>cd /d C:\Drew\CPAnalyst\\dist

C:\Drew\CPAnalyst\dist>cpa.exe
WARNING:Properties:PROPERTIES WARNING (group_SQL_YourGroupName): Undefined group

WARNING:Properties:PROPERTIES WARNING (filter_SQL_YourFilterName): Undefined fil
ter
WARNING:Properties:PROPERTIES WARNING (classifier_ignore_substrings): This field
 name is deprecated, use classifier_ignore_columns.
INFO:DBConnect:[MainThread] Connecting to the database...
DEBUG:DBConnect:[MainThread] Connected to database: DREW as drew@localhost
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Image
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Object
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image WHERE 1 IS NULL O
R 1=""
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image WHERE 1 IS NULL O
R 1=""
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image GROUP BY ImageNum
ber
DEBUG:DBConnect:[MainThread] SELECT Per_Object.ImageNumber, COUNT(Per_Object.Obj
ectNumber) FROM Per_Object GROUP BY Per_Object.ImageNumber
DEBUG:DBConnect:[MainThread] SELECT Per_Image.ImageNumber FROM Per_Image
DEBUG:DBConnect:[MainThread] SHOW TABLES
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 182, in launch_scatter_plot
  File "Scatter.pyc", line 250, in __init__
  File "Scatter.pyc", line 39, in __init__
ValueError: list.index(x): x not in list
DEBUG:DBConnect:[MainThread] SHOW TABLES
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 190, in launch_histogram_plot
  File "Histogram.pyc", line 172, in __init__
  File "Histogram.pyc", line 38, in __init__
ValueError: list.index(x): x not in list
DEBUG:DBConnect:[MainThread] SHOW TABLES
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 194, in launch_density_plot
  File "Density.pyc", line 273, in __init__
  File "Density.pyc", line 40, in __init__
ValueError: list.index(x): x not in list

Hiya!

This sounds like it’s simply a problem in your properties file. You might want to read the properties section of the CPA 2.0 manual (linked here: cellprofiler.org/linked_files/CP … Manual.pdf). I’ve also attached the properties README file, a file you can use as a starting point for filling in your own properties data.

The main misconception I think you’re having is in the database section. If you want to use a MySQL database, use the following:

db_type = mysql db_port = 3306 db_host = <your_host_name> db_name = <your_database_name> db_user = <your_user_name> db_passwd = <your_password>

If you want to have CPA set up an SQLite DB from your existing data (exported from CP ExportToDatabase), then use:

db_type = sqlite db_sql_file = </path/to/setup.sql>

If you want to use a pre-existing SQLite database then use the following:

db_type = sqlite db_sqlite_file = </path/to/sqlite_database.db>

I hope this helps… if not, send me your props and I’ll take a closer look.
Properties_README.zip (3.88 KB)

Hi,

Thanks for your reply Adam. I noticed the properties file I was using says “Classifier 2.0 properties file”. This is the output I get from CellProfiler 1.0.7522, so I just assumed it would work with CPA 2.0. Looking through, I can’t see any major difference with the Properties_README that is bundled with CPA 2.0.

I am using mysql, the database connection properties are:

# ======== Database Info ========
# Classifier needs to know how to access your database.
db_type      = mysql
db_port      = 3306
db_host      = localhost
db_name      = DREW
db_user      = drew
db_passwd    = drew

mysql is running locally, and I can see my data in the database “DREW”. I can view any image with the ImageViewer, so CPA 2.0 can “see” the images…
So it just seems strange that the ScatterPlot, Histogram, DensityPlot aren’t working…

Sorry, but I have no other bright ideas at the moment. I might try to set it up with an SQLite db instead, but so far I have run into errors that way…

Does anything jump out at you from the properties?

My properties file:

#Mon December 14 12:24:18 2009
# ==============================================
#
# Classifier 2.0 properties file
#
# ==============================================
# =============================================================================
#
#                 Properties file README for Classifier 2.1
#
# NOTE: Classifier 2 will not read old CPA properties files, nor will CPA read
#       this properties file format.  The two formats can, however be easily
#       converted by hand.
#
# This file is an example properties file to help users of Classifier 2 to
# setup your own properties file.  The syntax is simple.  Lines that begin
# with the "#" sign are comments which are ignored by Classifier.  All other
# lines must be in one of one of the following 2 forms:
#
#    property_name  =  value
#    property_list  =  value1, value2
#
# Optional fields may be left blank:
#
#    optional_property  =
#
# Below, many properties are filled in with example values surrounded by angled
# brackets like <this>.  These MUST BE REPLACED.  Values not flanked by angled
# bracked are suggested guesses.  These values may work as-is, but do read the
# description for each section so you know whether it applies to you.
#
# =============================================================================

# ======== Database Info ========
# Classifier needs to know how to access your database.
db_type      = mysql
db_port      = 3306
db_host      = localhost
db_name      = DREW
db_user      = drew
db_passwd    = drew

# ======== CSV File Info ========
# You may use CSV files for your per-image and per-object tables in place of a
# MySQL database. Classifier will use them to create a SQLite database.
# In this case you must COMMENT OUT THE FIELDS in the Database Info section and
# uncomment the fields below.

#db_type          =   sqlite
#image_csv_file   =  C:\Drew\Work\SQL_1_2_image.CSV
#object_csv_file  =  C:\Drew\Work\SQL_1_2_object.CSV

# ==== Database Tables ====
image_table   = Per_Image
object_table  = Per_Object

# ==== Database Columns ====
# Specify the database column names that contain unique IDs for images and
# objects (and optionally tables).
#
# table_id (OPTIONAL): This field lets Classifier handle multiple tables if
#          you merge them into one and add a table_number column as a foreign
#          key to your per-image and per-object tables.
# image_id: must be a foreign key column between your per-image and per-object
#           tables
# object_id: the object key column from your per-object table

#table_id    =  TableNumber
image_id      = ImageNumber
object_id     = ObjectNumber
plate_id      = 1
well_id     = 1

# Also specify the column names that contain X and Y coordinates for each
# object within an image.

cell_x_loc    = Cells_Location_Center_X
cell_y_loc    = Cells_Location_Center_Y

# ======== Image Path and Filename Columns ========
# Classifier needs to know where to find the images from your experiment.
# Specify the column names from your per-image table that contain the image
# paths and file names here.
#
# Individual image files are expected to be monochromatic and represent a single
# channel. However, any number of images may be combined by adding a new channel
# path and filename column to the per-image table of your database and then
# adding those column names here.
#
# NOTE: These lists must have equal length!

image_channel_paths = Image_PathName_Hoechst,Image_PathName_TG30,
image_channel_files = Image_FileName_Hoechst,Image_FileName_TG30,

# Give short names for each of the channels (respectively)...

image_channel_names = Hoechst,TG30,


# Specify a default color for each of the channels (respectively)
# Valid colors are: [red, green, blue, magenta, cyan, yellow, gray, none]

image_channel_colors = blue,green,


# ==== Image Accesss Info ====
# Specify for HTTP image access. This address will be prepended to the image
# path and filename pulled from the database columns listed above when loading
# an image.
#
# Example: If you set image_url_prepend to "http://yourserver.com/" and the
#   path and filename in the database for a given image are "yourpath" and
#   "file.png"
#   Classifier will try to open "http://yourserver.com/yourpath/file.png"
#
# Leave blank if images are stored locally.

image_url_prepend    = 

# ==== Dynamic Groups ====
# OPTIONAL
# Here you can define ways of grouping your image data, by linking column(s)
# that identify unique images (the image-key) to a unique group of columns the
# (group-key). Note that the group-key columns may come from other tables, so
# long as the tables have a common key.
#
# Example: With the "Well" group defined below, Classifier will allow you to
#   fetch cells from images from a particular well by providing you with a well
#   dropdown in the user interface. It will also allow you to group your data
#   by each unique well value when scoring.
#
# Example 2: Also note the "Plate+Well" group. This group will specify unique
#   pairs of plate and well values. Since well values such as "A01" are likely
#   to NOT be unique across multiple plates, this will provide a way to refer
#   to cells from, plate X, well A01, rather than just any well named "A01".
#
# FORMAT:
#   group_XXX  =  MySQL select statement that returns image-key columns followed by group-key columns. XXX will be the name of the group.
# EXAMPLE GROUPS:
#   group_SQL_Well        =  SELECT TableNumber, ImageNumber, well FROM Per_Image_Table
#   group_SQL_Plate+Well  =  SELECT Per_Image_Table.TableNumber, Per_Image_Table.ImageNumber, Well_ID_Table.Plate, Per_Image_Table.well FROM Per_Image_Table, WELL_ID_Table WHERE Per_Image_Table.well=Well_ID_Table.well
#   group_SQL_Treatment   =  SELECT Per_Image_Table.TableNumber, Per_Image_Table.ImageNumber, Well_ID_Table.treatment FROM Per_Image_Table, Well_ID_Table WHERE Per_Image_Table.well=Well_ID_Table.well

group_SQL_YourGroupName  =  

# ==== Image Filters ====
# OPTIONAL
# Here you can define image filters to let you fetch or score objects from a
# subset of the images in your experiment.
#
# Example: With the CDKs filter defined below, Classifier will provide an extra
#   option to fetch cells from CDKs... that is, images who's corresponding gene
#   entry starts with CDK.
#
# FORMAT:
#   filter_SQL_XXX  =  MySQL select statement that returns image-key columns for images you wish to filter out. XXX will be the name of the filter.
# EXAMPLE FILTERS:
#   filter_SQL_EMPTY  =  SELECT TableNumber, ImageNumber FROM CPA_per_image, Well_ID_Table WHERE CPA_per_image.well=Well_ID_Table.well AND Well_ID_Table.Gene="EMPTY"
#   filter_SQL_CDKs   =  SELECT TableNumber, ImageNumber FROM CPA_per_image, Well_ID_Table WHERE CPA_per_image.well=Well_ID_Table.well AND Well_ID_Table.Gene REGEXP 'CDK.*'

filter_SQL_YourFilterName  =  

# ==== Meta data ====
# What are your objects called? (e.g. cells, worms, etc.)
# This is used to provide the correct syntax for the GUI.
# FORMAT:  object_name  =  singular name, plural name
object_name  =  cell, cells,

# What size plates were used?  384 or 96?
plate_type  = 96

# ==== Excluded Columns ====
# OPTIONAL
# Classifier uses columns in your per_object table to find rules. It will
# automatically ignore ID columns defined in table_id, image_id, and object_id
# as well as any columns that contain non-numeric data.
#
# Here you may list other columns in your per_object table that you wish the
# classifier to ignore when finding rules.
#
# You may also use regular expressions here to match more general column names.
#
# Example: classifier_ignore_substrings = WellID, Meta_.*, .*_Position
#   This will ignore any column named "WellID", any columns that start with
#   "Meta_", and any columns that end in "_Position".
#
# A more restrictive example:
# classifier_ignore_substrings = ImageNumber, ObjectNumber, .*Parent.*, .*Children.*Count, .*_Location_Center_.*,.*_Metadata_.*
# A less restrictive example:
classifier_ignore_substrings = ImageNumber, ObjectNumber

# ==== Other ====
# Classifier will show you square thumbnails of objects cropped from their
# original images. Specify the thumbnail size here. The approximate maximum
# diameter of your objects (in pixels) is a good start.

image_tile_size   =  50

# ======== Auto Load Training Set ========
# OPTIONAL
# You may enter the full path to a training set that you would like Classifier
# to automatically load when started.

training_set  =

# ======== Area Based Scoring ========
# OPTIONAL
# You may specify a column in your per-object table which will be summed and
# reported in place of object-counts when scoring.  The typical use for this
# is to report the areas of objects on a per-image or per-group basis.

area_scoring_column =

# ==== Internal Cache ====
# It shouldn't be necessary to cache your images in the application, but the
# cache sizes can be set here.
#
# Example: image_buffer_size = 100
#   This will tell Classifier to keep up to 100 images stored in the program
#   for fast access.

image_buffer_size  =  1
tile_buffer_size   =  1

Thanks for your help.

Well, I’m having trouble connecting remotely to our version tracking system so I won’t be able to look into this much more until tomorrow. In the meantime I found a few problems with your props file:

These fields were set to 1 and should be set to their column names, (or left blank if you aren’t going to use plate viewer):

plate_id = <plate column name> well_id = <well column name>

These fields should either be commented out or removed altogether.

filter_SQL_YourFilterName = group_SQL_YourGroupName =

The latter fix maaay actually fix your problem. Let me know.

If not, you could try running the query “SHOW TABLES” when connected to your MySQL database… it appears to have returned nothing which is unexpected since you have your image and object tables in there.

Thanks again Adam for the suggestions.
I made those changes you suggested, still get the same error, here is the console dump:


C:\Drew\CPAnalyst>cd /d C:\Drew\CPAnalyst\\dist

C:\Drew\CPAnalyst\dist>cpa.exe
WARNING:Properties:PROPERTIES WARNING (classifier_ignore_substrings): This field
 name is deprecated, use classifier_ignore_columns.
WARNING:Properties:PROPERTIES WARNING (plate_id): Field is required for plate ma
p viewer.
WARNING:Properties:PROPERTIES WARNING (well_id): Field is required for plate map
 viewer.
INFO:DBConnect:[MainThread] Connecting to the database...
DEBUG:DBConnect:[MainThread] Connected to database: drew as drew@localhost
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Image
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Object
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image GROUP BY ImageNum
ber
DEBUG:DBConnect:[MainThread] SELECT Per_Object.ImageNumber, COUNT(Per_Object.Obj
ectNumber) FROM Per_Object GROUP BY Per_Object.ImageNumber
DEBUG:DBConnect:[MainThread] SELECT Per_Image.ImageNumber FROM Per_Image
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 182, in launch_scatter_plot
  File "Scatter.pyc", line 250, in __init__
  File "Scatter.pyc", line 39, in __init__
ValueError: list.index(x): x not in list

When I run SHOW TABLES on the database through phpMyAdmin, I get

[quote]Tables_in_drew
per_image
per_object[/quote]

which seems to be working…

I’ll keep trying, but let me know if you discover something! Cheers

Hrm… I’m digging and not coming up with anything very useful. The last query it made from your last post is “SELECT Per_Image.ImageNumber FROM Per_Image” – you might want to run that in MySQL and see that

also try adding this to your properties file and running again:

Sorry I haven’t been much help, this is a bit of a tough one. Typically these problems arise from some inconsistency in the database, but if your DB was created from ExportToDatabase then it’s hard to guess what might be wrong.

Another path you might take is trying SQLite. I looked at the lines commented out of your props file, and it looks like you were using the wrong SQLite mode:

#db_type = sqlite #image_csv_file = C:\Drew\Work\SQL_1_2_image.CSV #object_csv_file = C:\Drew\Work\SQL_1_2_object.CSV
image and object_csv_file expect there to be column headers in the files passed in, but the files that you are passing in appear to have come from ExportToDatabase. What you would want to do instead is this:

db_type = sqlite db_sql_file = C:\Drew\Work\setup.SQL
…assuming that’s where ExportToDatabase sent your CSVs and .SQL file.

Good luck, and let me know where these options take you.
-Adam

Hi,

Thanks for your suggestions. Here’s what I’ve been able to try so far:

It returned all the results, however in CPA, I still got:

DEBUG:DBConnect:[MainThread] SELECT Per_Image.ImageNumber FROM Per_Image None

Even with check_tables = true, or check_tables = yes, nothing different came up. I tried using a properties file written from the CPA example properties file, not the one automatically generated with CP ExportToDatabase, and still the same problem.

Next, I tried SQLite using the settings you recommended.

At first, I encountered and error, but forgot to save the console dump (Inspecting the SQLite DB tables, they were all empty.)

This all seemed quite strange to me, so I did a completely new install on a different machine (still Win32). Now, I was able to get the SQLite DB to populate properly and display ScatterPlot, Histogram, DensityPlots. I took this setup back to my original machine and it has since worked (console dump below).

C:\Drew\work\CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win>cd /d C:\Drew\Work\
CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win\\dist

C:\Drew\Work\CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win\dist>cpa.exe
INFO:Properties:PROPERTIES: Using default image_buffer_size=1
INFO:Properties:PROPERTIES: Using default tile_buffer_size=1
WARNING:Properties:PROPERTIES WARNING (plate_id): Field is required for plate ma
p viewer.
WARNING:Properties:PROPERTIES WARNING (well_id): Field is required for plate map
 viewer.
INFO:DBConnect:Skipping table verification step for sqlite
INFO:DBConnect:[MainThread] Connecting to the database...
INFO:DBConnect:[MainThread] SQLite file: C:\Documents and Settings\s4056811\CPA\
CPA_DB_6d7dabbf064c96ae10ebfd70b7f35f59.db
DEBUG:DBConnect:[MainThread] Created fast classifier function in sqlite
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image GROUP BY ImageNum
ber
INFO:DBConnect:[MainThread] Creating SQLite database at: C:\Documents and Settin
gs\s4056811\CPA\CPA_DB_6d7dabbf064c96ae10ebfd70b7f35f59.db.
DEBUG:DBConnect:[MainThread] CREATE TABLE Per_Image (ImageNumber INTEGER PRIMARY
 KEY,
Image_FileName_Hoechst VARCHAR(128),
Image_PathName_Hoechst VARCHAR(128),
Image_FileName_TG30 VARCHAR(128),
Image_PathName_TG30 VARCHAR(128),
Image_Threshold_OrigThreshold_Nuclei FLOAT NOT NULL,
Image_Threshold_FinalThreshold_Nuclei FLOAT NOT NULL,
Image_Threshold_WeightedVariance_Nuclei FLOAT NOT NULL,
Image_Threshold_SumOfEntropies_Nuclei FLOAT NOT NULL,
Image_Count_Nuclei FLOAT NOT NULL,
Image_Threshold_OrigThreshold_Cells FLOAT NOT NULL,
Image_Threshold_FinalThreshold_Cells FLOAT NOT NULL,
Image_Threshold_WeightedVariance_Cells FLOAT NOT NULL,
Image_Threshold_SumOfEntropies_Cells FLOAT NOT NULL,
Image_Count_Cells FLOAT NOT NULL,
Mean_Nuclei_Location_Center_X FLOAT NOT NULL,
Mean_Nuclei_Location_Center_Y FLOAT NOT NULL,
Mean_Nuclei_Children_Cells_Count FLOAT NOT NULL,
Mean_Nuclei_Intensity_IntegratedIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MeanIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_StdIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MinIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MaxIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_IntegratedIntensityEdge_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MeanIntensityEdge_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_StdIntensityEdge_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MinIntensityEdge_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MaxIntensityEdge_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MassDisplacement_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_LowerQuartileIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_MedianIntensity_Hoechst FLOAT NOT NULL,
Mean_Nuclei_Intensity_UpperQuartileIntensity_Hoechst FLOAT NOT NULL,
Mean_Cells_Parent_Nuclei FLOAT NOT NULL,
Mean_Cells_Location_Center_X FLOAT NOT NULL,
Mean_Cells_Location_Center_Y FLOAT NOT NULL,
Mean_Cells_Intensity_IntegratedIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MeanIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_StdIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MinIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MaxIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_IntegratedIntensityEdge_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MeanIntensityEdge_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_StdIntensityEdge_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MinIntensityEdge_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MaxIntensityEdge_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MassDisplacement_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_LowerQuartileIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_MedianIntensity_TG30 FLOAT NOT NULL,
Mean_Cells_Intensity_UpperQuartileIntensity_TG30 FLOAT NOT NULL,
StDev_Nuclei_Location_Center_X FLOAT NOT NULL,
StDev_Nuclei_Location_Center_Y FLOAT NOT NULL,
StDev_Nuclei_Children_Cells_Count FLOAT NOT NULL,
StDev_Nuclei_Intensity_IntegratedIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MeanIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_StdIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MinIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MaxIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_IntegratedIntensityEdge_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MeanIntensityEdge_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_StdIntensityEdge_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MinIntensityEdge_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MaxIntensityEdge_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MassDisplacement_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_LowerQuartileIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_MedianIntensity_Hoechst FLOAT NOT NULL,
StDev_Nuclei_Intensity_UpperQuartileIntensity_Hoechst FLOAT NOT NULL,
StDev_Cells_Parent_Nuclei FLOAT NOT NULL,
StDev_Cells_Location_Center_X FLOAT NOT NULL,
StDev_Cells_Location_Center_Y FLOAT NOT NULL,
StDev_Cells_Intensity_IntegratedIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MeanIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_StdIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MinIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MaxIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_IntegratedIntensityEdge_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MeanIntensityEdge_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_StdIntensityEdge_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MinIntensityEdge_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MaxIntensityEdge_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MassDisplacement_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_LowerQuartileIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_MedianIntensity_TG30 FLOAT NOT NULL,
StDev_Cells_Intensity_UpperQuartileIntensity_TG30 FLOAT NOT NULL);

DEBUG:DBConnect:[MainThread] CREATE TABLE Per_Object(ImageNumber INTEGER,ObjectN
umber INTEGER,
Nuclei_Location_Center_X FLOAT NOT NULL,
Nuclei_Location_Center_Y FLOAT NOT NULL,
Nuclei_Children_Cells_Count FLOAT NOT NULL,
Nuclei_Intensity_IntegratedIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MeanIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_StdIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MinIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MaxIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_IntegratedIntensityEdge_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MeanIntensityEdge_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_StdIntensityEdge_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MinIntensityEdge_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MaxIntensityEdge_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MassDisplacement_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_LowerQuartileIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_MedianIntensity_Hoechst FLOAT NOT NULL,
Nuclei_Intensity_UpperQuartileIntensity_Hoechst FLOAT NOT NULL,
Cells_Parent_Nuclei FLOAT NOT NULL,
Cells_Location_Center_X FLOAT NOT NULL,
Cells_Location_Center_Y FLOAT NOT NULL,
Cells_Intensity_IntegratedIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_MeanIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_StdIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_MinIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_MaxIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_IntegratedIntensityEdge_TG30 FLOAT NOT NULL,
Cells_Intensity_MeanIntensityEdge_TG30 FLOAT NOT NULL,
Cells_Intensity_StdIntensityEdge_TG30 FLOAT NOT NULL,
Cells_Intensity_MinIntensityEdge_TG30 FLOAT NOT NULL,
Cells_Intensity_MaxIntensityEdge_TG30 FLOAT NOT NULL,
Cells_Intensity_MassDisplacement_TG30 FLOAT NOT NULL,
Cells_Intensity_LowerQuartileIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_MedianIntensity_TG30 FLOAT NOT NULL,
Cells_Intensity_UpperQuartileIntensity_TG30 FLOAT NOT NULL,
PRIMARY KEY (ImageNumber, ObjectNumber));

INFO:DBConnect:Populating image table with data from SQL_1_27_image.CSV
INFO:DBConnect:Populating object table with data from SQL_1_27_object.CSV
DEBUG:DBConnect:[MainThread] Commit
DEBUG:DBConnect:[MainThread] Connected to database: C:\Documents and Settings\s4
056811\CPA\CPA_DB_6d7dabbf064c96ae10ebfd70b7f35f59.db
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image GROUP BY ImageNum
ber
DEBUG:DBConnect:[MainThread] SELECT Per_Object.ImageNumber, COUNT(Per_Object.Obj
ectNumber) FROM Per_Object GROUP BY Per_Object.ImageNumber
DEBUG:DBConnect:[MainThread] SELECT Per_Image.ImageNumber FROM Per_Image

HOWEVER… there is no menu bar on any of these plot windows and I cannot select certain dots, etc. The menus (File, Plot, Search, Show Info, Select Points) which are on the demo movie of CPA are not visible on the ScatterPlot window! I am really quite puzzled by this and wonder if my only option is to find a Mac to run CPA on. Has anyone reported this before? I am finding this problem on both machines I have used.

For interest, I tried to get this same dataset working with MySQL. When I import the Drew_Setup.SQL file (the reference to the .csv files at the end of this setup file must be changed to include the full Windows path, using forward slashes), there is an unspecified error:

[code]Error

SQL query:

LOAD DATA LOCAL INFILE ‘C:/Drew/Work/Out_Row5/SQL_1_27_object.CSV’ REPLACE INTO TABLE Per_Object
FIELDS TERMINATED BY ‘,’;

MySQL said: [/code]

Not sure what this means but all the data seems to be in the tables - they have the same number of rows as my SQLite tables. Now when I run CPA with the appropriate MySQL settings, I still get the original error that this thread started with. Console dump:

C:\Drew\work\CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win>cd /d C:\Drew\Work\
CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win\\dist

C:\Drew\Work\CPAnalyst_2.0_r8634_win\CPAnalyst_2.0_r8634_win\dist>cpa.exe
INFO:Properties:PROPERTIES: Using default image_buffer_size=1
INFO:Properties:PROPERTIES: Using default tile_buffer_size=1
WARNING:Properties:PROPERTIES WARNING (plate_id): Field is required for plate ma
p viewer.
WARNING:Properties:PROPERTIES WARNING (well_id): Field is required for plate map
 viewer.
INFO:DBConnect:[MainThread] Connecting to the database...
DEBUG:DBConnect:[MainThread] Connected to database: DREW as drew@localhost
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Image
DEBUG:DBConnect:[MainThread] SHOW INDEX FROM Per_Object
DEBUG:DBConnect:[MainThread] SELECT ImageNumber FROM Per_Image GROUP BY ImageNum
ber
DEBUG:DBConnect:[MainThread] SELECT Per_Object.ImageNumber, COUNT(Per_Object.Obj
ectNumber) FROM Per_Object GROUP BY Per_Object.ImageNumber
DEBUG:DBConnect:[MainThread] SELECT Per_Image.ImageNumber FROM Per_Image
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 182, in launch_scatter_plot
  File "Scatter.pyc", line 250, in __init__
  File "Scatter.pyc", line 39, in __init__
ValueError: list.index(x): x not in list
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 190, in launch_histogram_plot
  File "Histogram.pyc", line 172, in __init__
  File "Histogram.pyc", line 38, in __init__
ValueError: list.index(x): x not in list
None
Error in sys.excepthook:
Traceback (most recent call last):
  File "ClassifierGUI.pyc", line 1274, in show_exception_as_dialog
ValueError: list.index(x): x not in list

Original exception was:
Traceback (most recent call last):
  File "cpa.py", line 194, in launch_density_plot
  File "Density.pyc", line 273, in __init__
  File "Density.pyc", line 40, in __init__
ValueError: list.index(x): x not in list

So finally, I can get my dataset working with SQLite, but there are no menus displayed on the plot windows. MySQL is still returning the same errors. Sorry for the long post but hopefully it might help shed some light on what the problem is.

I think my next strategy, is to use an updated build of CPA, or try it on a Mac, because I need those menu options. Any other ideas?

Thanks for your help with this thread.

–The problem is that the demo movies you watched, show CPA 1.0 (Java based) in action, not CPA 2.0, the version you are using. I’m sorry if the website didn’t make this clear… we’re actually planning on gutting the whole thing relatively soon with the release of CP 2.0.

Anyway, I could try to respond to the other issues you mentioned, but I think we should step back and ask whether CPA 2.0 will be sufficient for your analysis. Since we are still developing 2.0, there are still a few things that 1.0 does, that 2.0 doesn’t.

Here are some major differences:
CPA 2.0

  • improved multi-class classifier
  • plate viewer tool
  • basic (non-interactive) plotting tools
  • improved image viewer and handling for more than 3 channels
  • under development… if you find any bugs, they will be fixed quickly and committed to SVN… new features are also being added daily (eg: the scatterplot now supports selection, and viewing images from a selection)
  • Supports SQLite

CPA 1.0

  • 2-class Classifier
  • More fully featured plotting/visualization tools
  • MySQL only
  • No longer under development.

…also note that both versions have different formats for properties files.

CPA 1.0 can be downloaded here (cellprofiler.org/downloadCPA.htm) if you’re curious.

Let me know either way, and I’ll help you get up and running.

-Adam

Okay, thanks for clearing that up Adam. I guess I was getting confused between Classifier 2.0 and CPA 2.0 or something… In any case, I might give CPA 1.0 a look-in, but I’ve also had some good results using Excel and MATLAB for the data visualisation, as my image set is not huge.

Looking forward to the new CellProfiler 2.0 and continued CPA 2.0 development.

Cheers!

I should mention that the plotting functionality in CPA v2 is already getting fairly close to what it was in CPA v1. Currently, you can plot points from any table on a histogram, scatterplot or density plot… axes can be changed to log space, as can the density plot color map. Selection and viewing images from scatterplot is available, but this isn’t built in the other plots yet. Zooming and panning on all plots also works.

Anyway, do let me know if you need any more help :mrgreen:

We just posted a new version of CPA to the website. You might give it another try.

If nothing else, we should make an effort to figure out what the actual cause of the errors you’re seeing is.

Looking at this again, I can see what appears to be causing the error: “list.index(x): x not in list”

The plot is trying to populate a list of table options to plot data from and it is trying to select your image table in the list but is not finding it. I can easily change the code to simply not select it if it can’t find the table, but I’d like to understand what is happening here as well.

The query that is getting the list of table names is:
SHOW TABLES; (for MySQL)
SELECT name FROM sqlite_master WHERE type=“table” ORDER BY name; (for SQLite)

…that query should be printed somewhere in your console output, but I don’t see it there, which is a bit confusing. Nevertheless, if you can make the appropriate query on your db, and verify that your image table is indeed listed (with letters in the correct case i.e.“Per_Image” not “per_image”), that will get us somewhere.

I’m committing a change to the plotting code so it won’t fail for this reason anymore, and I could build you a release for windows if you’d like, but do let me know what happens with the show tables query.

Hi guys,

Well, I tried the queries you suggested. It seems SQLite lists the tables with the correct case (Per_Image, Per_Object). But MySQL tables are listed in lowercease (per_image, per_object).

My SQL setup files generated by CellProfiler’s export to database have the correct case… so not really sure what happened here.

I got the new CPA too, will give it a go when I get a chance and let you know how it goes…

This could be a configuration difference in your MySQL server, though I can’t see a setting on this page (dev.mysql.com/doc/refman/5.0/en/ … ivity.html) that would result in the behavior you describe. Using the same case in the .SQL setup files and in the .properties file should always work.

We’ll add some code that checks for tables with different case, to try to do something reasonable.

In the meantime, could you query “SHOW VARIABLES” on your MySQL server and let us know the settings for lower_case_file_system and lower_case_table_names?

Hi

Yes, the setting seems to be:

lower_case_file_system ON
lower_case_table_names 1

For your info, I’m using EasyPHP (Apache/MySQL/PHP), with MySQL 5.1.37, so they may have changed the default settings as part of that install package.

Okay, I just realized what’s going on. It’s a bug on our side. I was misreading the behavior of MySQL as controlled by these variables. The fix on our side will be to do all table comparisons in lowercase.

As a workaround, you can just change the table names in the properties file to be all lowercase.