Training data set when adding new data to trained network

Hi all!

I’m working on data sets with animals of the same species, but recorded in different locations and with varying camera angles, zoom levels, light conditions, and so on. Moving forward we will be recording in more new locations and again with varying conditions.

I am therefore in the process of testing how well a network trained in one location performs in a new location.

My question is this:
When using a network that I’ve already trained as a base (initial weight), and adding new data (from new location), should I then A) keep the data from the first location in the training data set, or B) train on the data from the new location only?

My concern is that A might lead to overfitting since the network will end up having trained for a very large number of iterations on the base data set or that B might lead to the network, sort of, drifting and “forgetting” what it learned while training on the base data set.

I hope my question makes sense. :slight_smile:



if you have multiple different videos (different locations/lighting/camera angles) use all those different videos to train the model and then train network based on snapshot but with new data only and for lower number of iteration (40-50k). That’s my approach when making a robust model. Maybe someone has a better or more thought out way. If you model was very robust from the beggining it should be okay. Plus if you feel it stops to work good on older videos, you can combine all the datasets and give the model a bigger retraining session.


Thank you so much for the reply. Very helpful and I’ll test it out today.

What’s been your approach to setting the multi_step values in pose_cfg.yaml when re-training?


I keep the multi-step default since it’s hard to overfit the model anyway (with those retraining sessions you won’t get to 1mil iters that quickly and even then it’s not overfitted if you have proper size dataset).

Hi, just to clarify - when you label new data, dlc.merge and set the snapshot index you wish to train from in the pose_cfg.yaml file, you are retraining your network from the set snapshot, but using all the data, not just the newly labeled one, correct? If so, how can you specify to retrain only using the new data?


You can comment out (# at the start of the lines) the data sets (videos) you don’t want to include in the training in the config.yaml file. If you do that, and then create a new training dataset (on a new shuffle), it will train only on the new data data.

(I don’t know for sure that this is the correct way of doing it but this is what I’m doing at this stage)