Different results CP3 vs CP4 - Focus on MeasureObjectSizeShape

Hello CellProfiler Team and all the community members,

Following the release of CP4, we considered to migrate from CP3 to CP4, especially to benefit of the performance improvement.

For this, we migrated our main pipeline to be CP4 compatible, and run a test on 2 different datasets.

To make it compatible, we had to remove the LoadImage module which is deprecated and use Image, Metadata, … instead.

We focus our verification on the existing CP3 metrics.

The tests have been done with the same images and the same input values.

As a result:

  • For the masks:

    • the main part was very similar or equal for CP3 / CP4 runs.

    • Some of them have significant difference (such as merging 2 cells into 1), but this may be potentially resolved by finetuning of thresholding settings (to be checked).

  • For the metrics:

    • Even by focusing on the “very similar or equal” images, we can see that results are similar… modulo a multiplying factor. For instance:
DataSet1 (6 images) Dataset2 (2 images)
Mutliplying Fact. Mutliplying Fact.
Compactness X15 X5
Perimeter X0.87 0.90
Circularity X1.21 X1.23
  • Looks like the impacted metrics are the ones linked to “MeasureObjectSizeShape”

  • Did someone observe same behavior and/or have any explanation about these multiplying factors?

  • For the compactness for dataset1, the values for dataset 1 were around 30 with CP3, and now around 2… Any idea about which values looks relevant?

Hi @Nicolas_F,

Generally speaking, we advise that measurements taken in CP3 are not directly compared to those from CP4 for analysis. Major version jumps represent ‘breaking’ changes, so it’s best to analyse all your data on the same version.

Regarding thresholding, a lot of the underlying thresholding functions have been updated with newer implementations whcih can be ever so slightly different. As of 4.0.7 we’ve also added a log transformation option which was previously hard-coded into some (but not all!) of the methods from CP2 and CP3. It should be possible to replicate results from previous versions, but you can expect to see some small differences.

As for metrics, MeasureObjectSizeShape has been overhauled to make use of updated metric implementations from scikit-image. This is somewhat faster and provides access to a lot of new metrics, but it does mean that some results are a little different. Perimeter in particular is now measured in whole pixels rather than edges, so in a perfect square each ‘corner’ pixel represents +1 instead of +2 if that makes sense. This also factors into the circularity measurement. Compactness is now calculated differently, so I’ll have to double-check that everything is working properly there. I would have thought that a compactness value of 30 is a bit extreme - are your objects very irregularly shaped?