Resume training by specifying snapshot path

Usage & Issues
#1

Hi,

I am running DLC on Google Colab to take advantage of the free GPU. I uploaded my labelled data and started training until 267500 iterations before my storage ran out. I have cleared my trash and want to resume training from 267500 iterations. I changed the path to the latest snapshot in init_weights in the pose_cf.yaml config file and saved the updated file in Google Drive:

image
image1670×194 32.5 KB

I tried restarting the analysis from the beginning (installing dlc) but at the training step, it restarts from 0 instead of the most recent snapshot. However, the init_weights path is updated:

image
image1207×542 19.2 KB
.

Can you please advise how to resume training, specifically the step you need to start from? I saw similar posts that mentioned resuming training but there were no specifics besides changing the init_weights path in the pose_cfg file.

Thanks!

#2

Hi Eris,
From a similar issue reported on Github, it seems that the training picks up from the most recent snapshot, but the counter actually restarts from 0. :slight_smile:

https://github.com/DeepLabCut/DeepLabCut/issues/490#issuecomment-554639484 (have a look at the figure, under init_weights)