CPA Edit Rules for Validation

Hello,

I have used CPA to train a classifier with a training set. However, I am having difficulties doing a validation on a unseen test set.

I have tried 3 methods to load a previously trained model, unfortunately, none have worked:

  1. Loading a saved model → Error loading model
  2. Copying and pasting rules into CPA → unable to paste
  3. Advanced > Edit rules → “unable to parse…”

Unfortunately, without being able to paste rules from a previously trained classifier, it cannot be validated against an unseen data set. It would be useful to be able to do so in CPA so the number of rules can be quickly edited, whole images classified instantly and confusion matrices obtained easily. It would be impossible to do some of these with the Filter module in cellprofiler.

It seems the only way to re-obtain a trained model is to open the training set in CPA and hit train classifier. But this is not possible when trying to validate the model on an unseen data set.

If anyone knows a solution, any advice would be much appreciated! Thanks!

Hi,

Thanks for your input, it’s always good to know what functionality people need!

As you say, you can apply rules from a previous run of training on your next experiment by importing them into CellProfiler in the FilterObjects module when you run the next set. You’ll then know how your old rules would have worked on those objects, based on the object’s Children_ClassN columns (each of which will be 0 or 1). In that sense, whole images WILL be already classified, and you can validate them by say, using OverlayOutlines to make images where all your objects are circled in different colors by class.

You would NOT be able to immediately generate confusion matrices for a new set in CPA even if the functionality you wanted existed, because confusion matrices only apply to objects in the training bins, so you’d need to spend time populating the bins to create a new training set for the new experiment anyway. Once you do that, when you train the rules are going to change anyway, so importing the old ones has limited value. If you DO take the time to create a training set and training for the new experiment, though, you can then go back and compare against those Children columns to try to do a validation of your new training vs your old one.

Another option would be to include your training set in all future runs so you could always regenerate it- there’s a recent post on it linked below. If there are other functionalities you think could be aided by creating such a functionality, please do let us know and we can consider it!

~Beth

Hi Beth,

Thanks very much! That’s very helpful.
I think being able to edit the rules or being able to copy and paste them would be quite helpful. It would allow us to quickly tweak the number of rules and then score specific images on CPA that we have selected for validation. One of the issues is that I have trained a model and just used the max rules for now but it may be over-fitted to the training set so I’m trying to figure out what would be a good number of rules to settle on. But having to make several separate .txt files each with different number of rules and then changing each FilterObjects module then re-running the pipeline would take quite a bit of time. Still possible but perhaps a bit more inconvenient.

Thanks very much again for the solutions and your prompt reply!

Jonas