Training data set when adding new data to trained network

Hi all!

I’m working on data sets with animals of the same species, but recorded in different locations and with varying camera angles, zoom levels, light conditions, and so on. Moving forward we will be recording in more new locations and again with varying conditions.

I am therefore in the process of testing how well a network trained in one location performs in a new location.

My question is this:
When using a network that I’ve already trained as a base (initial weight), and adding new data (from new location), should I then A) keep the data from the first location in the training data set, or B) train on the data from the new location only?

My concern is that A might lead to overfitting since the network will end up having trained for a very large number of iterations on the base data set or that B might lead to the network, sort of, drifting and “forgetting” what it learned while training on the base data set.

I hope my question makes sense. :slight_smile:

Best,
Jonas

Hi,

if you have multiple different videos (different locations/lighting/camera angles) use all those different videos to train the model and then train network based on snapshot but with new data only and for lower number of iteration (40-50k). That’s my approach when making a robust model. Maybe someone has a better or more thought out way. If you model was very robust from the beggining it should be okay. Plus if you feel it stops to work good on older videos, you can combine all the datasets and give the model a bigger retraining session.

2 Likes

Thank you so much for the reply. Very helpful and I’ll test it out today.

What’s been your approach to setting the multi_step values in pose_cfg.yaml when re-training?

Hi,

I keep the multi-step default since it’s hard to overfit the model anyway (with those retraining sessions you won’t get to 1mil iters that quickly and even then it’s not overfitted if you have proper size dataset).