Overwrite Per-Image, Rescale, pipeline, and CPa errors

Hi again,
I have been further learning CP and (trying) CPa and I have some more questions about my pipeline. I attached some errors I am getting, to illustrate. Hope my questions are clear and I do appreciate any time invested in answering me. My aim is to use this for our analysis of all of our HCS data, as I found the ScanR program not working for us.

Summary of what I am trying to do.
Fix alignment, rescale and create metadata from the file names in LoadImages.
Find primary (nuclei by DRAQ5) (<-- Mark adjusted this in my original post)
Find secondary (Using nuclei as object, and then distance)
Find tertiary (subtraction to find cyt.)
Measurements
Calculate the ratio of YFP in the primary / tertiary objects (our positive control treatment induces creation of granules. Can be seen in Screeny#2. Weak but there - I need to identify these cells. In neg controls, there is NOTHING in the cyt.)
Export to database to be able to use CPa to classify etc.

Questions:

  1. What is “Export to database will overwrite Per-Image and Per_Object tables” (Screeny1). Even If I keep changing the output dir, it keeps giving me this. I am probably doing something wrong.

  2. I added x/y correction (Align module):
    a. I don’t see aligned images later in CPa, rather the images still show as the original, unaligned ones, for example when I open an image in CPa. Am I supposed to see them aligned? or is it just behind the scenes?
    b. The align module seems to take a lot of processing time for each cycle - is there anyway to simply tell it manually to adjust say y axis by say -7 for every single image set?
    c. When I choose “Crop” in the align module, I can see the result of the alignment in the result box but I immediately get error messages as pipeline continues, but when I choose “Keep”, I get no errors, yet the result images shows a yellow image (Screeny2). I realized the yellow is becasue of the Image contrast - when I choose Keep, I need to transfer to Log to be able to view the result of the align. Not sure why. So, which Crop mode is more correct for me, and why the error? You can see how the 7 px shift solves the misaligned channels.

  3. Image processing: In my first post (Exporting to MySQL db issue (solved)) Mark fixed my LoadImages to help better define the nuc. However, this did not seem to take into account the 12bit camera to 16bit save issue. So I added RescaleIntensity, and that module did seem to rescale the images, but then the IdentifyPrimaryObject that Mark adjusted did not work for all he images so I changed it.
    What is the difference between the RescaleIntensity module and the check-mark option “Rescale intensities” in LoadImages?
    I do need this, right?

  4. I am not managing to define a group. I want to call all column 1 controls, I don’t see where to do this. I thought it’s in “group fields”. In the pull down it has: “Image_Metadata_Well”, but I am not sure what to do with that, how to use that. Specifically, column 1 on my plates are all positive controls and columns 23, 24 are all negative. How to I define those so that I can group them later in CPa.

  5. Calculate statistics give me the following error: Screeny3. This is currently not in the pipline.

6. In CPa: Plate viewer, Scatter plot, Histogram, Density and BoxPlots all give me an error - Screeny4. I can open Classifier, load some cells and view them. I can also open “table viewer” and “open from database”. So it seems I am connecting to the db.

Sorry this is so long, but I am trying to get the hang of this :smile:.
3image sets.zip (7.61 MB)
Screenys scanr_CP_CPa_.zip (460 KB)
scanr_forums_align_rescale.cp (14.1 KB)

If you don’t specify a different table prefix, ExportToDatabase will default to naming the tables “Per_Image” and “Per_Object”. If they already exist (e.g, you’ve already done another run with the same settings), ExportToDatabase will re-write these tables, but will prompt you before doing so.

[quote=“rogueiam”]2. I added x/y correction (Align module):
a. I don’t see aligned images later in CPa, rather the images still show as the original, unaligned ones, for example when I open an image in CPa. Am I supposed to see them aligned? or is it just behind the scenes?[/quote]

The aligned images are used in the pipeline but are not saved to the hard drive so CPA doesn’t know about them. You can will need to use SaveImages to save the aligned images, noting that you should check the “Store file and path information to the saved image?” setting. This will store the filename/path information to the database, so CPA knows where it is.

Not, but there should be. I’ve added it as a requested feature to our bug tracking list here.

For the yellow image issue, see my answer to the next question.

Re: error message - I don’t seem to get this message with the images you choose. Am I guaranteed to get this message with the 3 images you provided? If not, I’ll need to the same images that generated the error.

[quote=“rogueiam”]3. Image processing: In my first post (Exporting to MySQL db issue (solved)) Mark fixed my LoadImages to help better define the nuc. However, this did not seem to take into account the 12bit camera to 16bit save issue. So I added RescaleIntensity, and that module did seem to rescale the images, but then the IdentifyPrimaryObject that Mark adjusted did not work for all he images so I changed it.
What is the difference between the RescaleIntensity module and the check-mark option “Rescale intensities” in LoadImages?
I do need this, right?[/quote]

The difference is the following:

  • The “Rescale intensities” setting in LoadImages essentially chooses whether to scale the image to 0-1 based on the maximum intensity value used in the image (in your case, 12-bits), or the value based on the image format (in your case, 16-bits). We realize that this setting is a little misleading in wording so we’re changing it for the next release.
  • The RescaleIntensity module grants more flexibility in how the image is later rescaled. However, keep in mind that the identification modules work perfectly fine with the native image bit-depth, whatever that may be.

After taking a look at your images, I see what you mean about “not taking into account the 12bit camera to 16bit save issue”. Usually, the 12-to16-bit images we see have the 12-bit image placed in the lower bits of the image, but in your case, the image is offset to the upper half of the bit-depth. So either checking or unchecking the “Rescale intensities” in LoadImages will actually not quite do what you want it to do.

My suggestion is use the two RescaleIntensity modules on the raw images, i.e, after LoadImages but before Align. Uncheck the “rescale” setting in LoadImages, then do the following in RescaleIntensity:

  • Use “Choose specific values to be reset to a custom range” as you have currently.
  • Intensity range for input image: 0.5, 0.5625
  • Intensity range for final image: 0,1
  • Rescaling pixels below lower limit: Set to zero
  • Rescaling pixels above upper limit: Set to one

This will correct for the 12-to-16-bit issue andthe extra offset. You should see that the Align module display is improved. This is because the shifted image is padded with 0’s for display, but since your image was squeezed into the 0.5+ range, everything appears white. When these images (one colored red, the other green) are merged in the display, it looks uniformly yellow.

As a side note: In LoadImages, “individual images” for the file type is sufficient. “tiff,flex,zvi movies” is not optimal since your images are not movies.

In CellProfiler, grouping is used to define across which subset of images the analysis run is to be taken as a self-contained set. Despite the wording, it has nothing to do with groups in CPA, which is an SQL statement which defines a collection of images. For your run, you do not need to define groups in LoadImages, which is why I unchecked it.

Based on your description, I think you want a “filter” rather than a “group”, since filtering is use to define subsets of images for which a certain operation is to be used in CPA (like enrichment scoring; see the CPA manual for more details).

I can’t comment on this unless I see the settings used.

In CPa: Plate viewer, Scatter plot, Histogram, Density and BoxPlots all give me an error - Screeny4. I can open Classifier, load some cells and view them. I can also open “table viewer” and “open from database”. So it seems I am connecting to the db. Unfortunately, I’m not sure about this one.

Regards,
-Mark

Thanks for the detailed reply.

[quote=“mbray”]

[quote=“rogueiam”]2. I added x/y correction (Align module):
a. I don’t see aligned images later in CPa, rather the images still show as the original, unaligned ones, for example when I open an image in CPa. Am I supposed to see them aligned? or is it just behind the scenes?[/quote]

The aligned images are used in the pipeline but are not saved to the hard drive so CPA doesn’t know about them. You can will need to use SaveImages to save the aligned images, noting that you should check the “Store file and path information to the saved image?” setting. This will store the filename/path information to the database, so CPA knows where it is. [/quote]

OK, so here is my next question. If I understand correctly, CP does the measurements, and CPa will analyze these measurements. So I don’t really need to save the aligned images for use by CPa right? When *would *I need to save the aligned images, other than for presentation or QA purposes?

[quote=“mbray”]

Not, but there should be. I’ve added it as a requested feature to our bug tracking list here. [/quote]

Excellent. Indeed it takes about 15-20x longer (for me at least, literally, according to the small test sets I am running) to do a run with the align module, than without. And I am pretty sure I need it becasue my Cyt signal is somewhat perinuclear and I need the nuc mask to be accurate. But for the moment I think I will just make do and leave it out.
Any chance this will come out tomorrow? :smiling_imp: (jk)

[quote=“mbray”]

[quote=“rogueiam”]3. Image processing: SNIP
What is the difference between the RescaleIntensity module and the check-mark option “Rescale intensities” in LoadImages?
I do need this, right?[/quote]

SNIP
You should see that the Align module display is improved. This is because the shifted image is padded with 0’s for display, but since your image was squeezed into the 0.5+ range, everything appears white. When these images (one colored red, the other green) are merged in the display, it looks uniformly yellow. [/quote]

OK did this, but still not clear on whether Keep or Crop is preferred in Align. I tested with Keep and got no errors. Crop still gives me the error with any of my images. But like I mentioned above, atm I had to remove Align because of the processing time, so I’ll push off working through that, especially if Keep is fine. (I just hate getting errors in anything I work with that I cannot figure out why…)

[quote=“mbray”]

In CellProfiler, grouping is used to define across which subset of images the analysis run is to be taken as a self-contained set. Despite the wording, it has nothing to do with groups in CPA, which is an SQL statement which defines a collection of images. For your run, you do not need to define groups in LoadImages, which is why I unchecked it.

Based on your description, I think you want a “filter” rather than a “group”, since filtering is use to define subsets of images for which a certain operation is to be used in CPA (like enrichment scoring; see the CPA manual for more details). [/quote]

Hmm. I’ll explain a bit better. I want to group the results from all of column 1 as controls. I see can do that in Classifier by dragging cells from control wells into the control bin to make rules.
But from what I understand, also from the CPa forum and manual, is that I can create groups in properties file. And I rather not do that manually.
I just cannot figure out how to use the Group fields that opens up when you check “create proprieties file”, that is in turn meant for CPA, not CP, in Export to Databases. I know I am bit confused as to the best way to do this, and in terms of how to use a group.

[quote=“mbray”]

I can’t comment on this unless I see the settings used. [/quote]

OK, I’ll probably re-ask this at a later point in time.

[quote=“mbray”]

In CPa: Plate viewer, Scatter plot, Histogram, Density and BoxPlots all give me an error - Screeny4. I can open Classifier, load some cells and view them. I can also open “table viewer” and “open from database”. So it seems I am connecting to the db. [/quote]

Unfortunately, I’m not sure about this one.

I solved this by simply using SQLite. So it was clearly still a MySQL db setup issue. Supposedly not as good for CPa as MySQL, but SQLite is *far *easier to set up. If I run into problems with “number of rules” I’ll go back to it and worry about it later.

Regards,
-Mark

Thanks,
Avi

Correct.

Other than those cases, you probably don’t need to save them at all.

It pretty rare that I need to use an option other than “Keep size” since displacements are usually small. But I can’t diagnose why you are getting an error with “Crop” since none of the images you posted produce it.

[quote=“rogueiam”]
Hmm. I’ll explain a bit better. I want to group the results from all of column 1 as controls. I see can do that in Classifier by dragging cells from control wells into the control bin to make rules.
But from what I understand, also from the CPa forum and manual, is that I can create groups in properties file. And I rather not do that manually.
I just cannot figure out how to use the Group fields that opens up when you check “create proprieties file”, that is in turn meant for CPA, not CP, in Export to Databases. I know I am bit confused as to the best way to do this, and in terms of how to use a group.[/quote]

To use grouping, the identity of the experimental conditions (including controls) must be located as a per-image column of data or combination of columns. The manual mentions this as the definition of a group: group_SQL_<XXX> = <MySQL SELECT statement that returns image-key columns followed by group-key columns>Let’s take an example grouping statement in the CPA properties file (from the manual):group_SQL_Well_Plate = SELECT ImageNumber, plate, well FROM per_imageThe image key is “ImageNumber” and the group keys are “plate” and “well”, each of which are a column of metadata collected with the other per-image measurements. So for the group name setting, you would enter “Well_Plate” and for the per-columns, you would enter “ImageNumber, plate, well” since these per-image columns specify the image key and uniquely specify the wells.

Unless you have metadata which specifies column 1 as controls (which you do not seem to have), you cannot use grouping for this purpose with your data as it stands now. You could satisfy this requirement by using LoadData to include the plate layout information, though, and then specify the metadata column here.

Regards,
-Mark