Ideal number of frames extracted for 5-6 minute videos

I am running DeepLabCut on multiple 5-6 minute videos, each one has 1 mouse in a fixed room from a top-down angle. How many frames should I extract, and how many iterations should I run, to ensure good tracking and a stable skeleton?

When using single animal DLC, skeleton is not used for training, just plotting. Length of the video doesn’t matter when considering number of frames to extract, what matters is to extract as much different frames with different poses as possible. Depending on the number of videos you have, extract from each video enough frames to have overall 200-250 frames. Train model for ~250-300k iterations. Then I’d analyze a couple of videos and extract outlier frames to add up to 300-350 frames and retrain the model. Note that when you retrain after adding outlier frames you start new iteration (model starts from scratch) so you have to train all the way again. I also like to manually inspect the analyzed video (using manual extraction method for outlier frames and going to frames where model had dips in accuracy) and hand-pick bad frames (usually it’s when animal is rearing or grooming and takes odd shape compared to walking or standing).

For my use case I had to label around 600 frames overall (including the extracted outliers) to get a really robust model that worked on different videos with insignificant number of bad tracking frames. This maybe less for you if you have consistent recordings - mine had differences in lighting/rat size/cage layout/resolution etc…

Can you clarify what you mean by adding up frames to get to 300-350 frames? The way I’ve been going about it is that every time I start a new video, I go through the process of relabeling each video (i.e. I have to relabel 20-40 frames each time I start a new video, then I do the next video and have to do another 20-40 frames). What you’re saying sounds like, if you have similar animals or videos, you carry over frames from previous videos when you start a new project to create a cumulative effect across multiple videos.

I’m not super familiar with all the different functions of DeepLabCut yet. I haven’t tried training multiple videos at a time, and I haven’t messed around with the different variables in the config file such as pcutoff, alphavalue, or batch_size since I didn’t know what they did. I’ve just been trying to create one video that was relatively stable.

I have multiple videos using mice in various tests: CPA box, restraint test, forced swim, marble test. If I analyze multiple videos at a time, as long as those videos are using the same test in the same setting, can I extract multiple frames from each one to achieve 300+ frames to increase accuracy and tracking?

Yes, for your use case, you should make 4 projects, one for each experimental condition. To each of them add appropriate videos, let’s say 10-15 videos for each project/experimental condition. Then you extract frames from videos (like 25 or so per video, adding up to ~300 frames in the project. Label them, create dataset and train your network. After that evaluate, analyze and retrain if needed. For each of those projects different labels might be of interest, so that’s mostly why you should make different projects.

Do I understand correctly, that you created new project for every single video? It’s not possible to train a good model that way.

If you are tracking the same things (body parts, other things in view) then you can track them in a single project, and it doesn’t matter how many videos you put in there.

If those body parts might look different in different scenarios (e.g. lighting differences), or if you are tracking something different in each scenario (e.g. different body parts or different non-animal objects), then those should be different projects.

Once you have a robust model (trained well on potentially one or two videos, maybe more depending on what happens in other videos) that model should be able to fully track other videos without retraining. If you try it on a novel video and get poor results, extract outliers, refine, retrain, and try it.

As you add these outliers to the ever-growing training set, the model will get “smarter” and you will have fewer things to refine.