I am using Google Colab to train DLC. For training, we are using 6 videos (about 600 -1000 frames each) of guppies that were caught in different locations and have different prey types. The 6 videos were chosen based on if the video has a black background/white fish, white background/black fish, or grey background/grey-ish fish so that DLC can train on different looking videos. From the training, we would like to analyze a few hundred videos we have of the guppies to analyze feeding kinematics. When using Colab, we time out of the GPU around 100,000 iterations which is causing issues in the training and results in high body part detection error. In the paper it says to pick up where training left off follow
However, when changing the path of the posse_config file, Colab recognizes the new path but the new training is seemingly never saved. If we train the first time to 100K iterations and pick up training for another 89k iterations before timeout, all previous iterations and checkpoints are overwritten and the only saved iterations are those from the 89k.
Is this possibly due to an error in the change of code we are using?
We tried using
init_weights: /usr/local/lib/python3.6/dist-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_101.ckpt-snapshot-109800but DLC did not recognize this. We then went to the checkpoint file and used the path of the last saved checkpoints.
init_weights: /content/drive/My Drive/DLC/Examples/SixGuppyTrainingVideos-victoria-2019-07-22/dlc-models/iteration-0/SixGuppyTrainingVideosJul22-trainset60shuffle1/train/snapshot-109800
We were able to start training with the path above but with no increase in the quality of training and no way to tell if the new iteration are actually being added to previous training iterations.
Thanks in advance!
Guppy for good luck