Question on evaluate_multianimal_crossvalidate, importance of pck/rpck

Hi all, first off thank you for all your amazing work!
I have a question regarding model performance.

I have trained my model on ~300 frames with around 10 animals per frame.
I trained it for around 100k iterations (batch size 4, also changed the nr of iterations for the learning rate).
The model is visibly performing decently when creating a labeled video, but it does have some weird jumps and sometimes not detecting an animal when its clearly visible.
When using the evaluate_multianimal_crossvalidate function the RMSE i get is between 5-10 (test RMSE is higher than train) and the pck/rpck is quite low (up to 0.4).

I was wondering what was the importance of pck/rpck. Which target is better to optimize, rpck or rmse?
Also would a low rpck indicate undertraining? My test rmse is almost double of the train so i would assume there is overfitting instead.

Any tips on this would be very welcome! I considered to change from resnet 50 to 101 due to the multiple animals. Although not sure that would necessarily help.

Hi @pbugneac, glad to hear you’re liking DeepLabCut :slight_smile: We found that rPCK was a much better target in most scenarios, as it encourages the selection of parameters that give more complete animals (whereas RMSE may cause the assembly to discard all but the strongest bodypart connections).
There is a lot going on though between the moment bodyparts are detected and the tracking of complete animals over time. Have you tried to run deeplabcut.create_video_with_all_detections and deeplabcut.utils.make_labeled_video.create_video_from_pickled_tracks? This gives two different videos, helping you and us understand whether the culprit are the raw detections (unlikely) or the animal assembly :slight_smile: Hope that helps!

Hey @jeylau! Thanks for the quick reply.

Is it then possible to optimize both rpck and rmse? In your user guide you mention that we should aim for a low RMSE and high rpck. In my case i havent been able to keep rpck high no matter how I changed the training.
Indeed i tried both deeplabcut.create_video_with_all_detections and deeplabcut.utils.make_labeled_video.create_video_from_pickled_tracks. The detections (as expected) look good, however the tracking mixes up animals sometimes. Not sure how to correctly influence the tracking.