My setup: Windows 10 64-bit, Conda 4.5.12, Python 3.6.8, Deeplabcut 22.214.171.124, without a GPU
I have a problem restarting training from the last snapshot. It just hangs on ‘Starting training…’ (I’ve left it for 12 hrs and nothing happens).
I’ve just had this problem resuming at snapshot-34000. I’ve tried 32000 and 30000 and so far it won’t continue training.
I’ve managed to resume training on two previous attempts by going back through the snapshots, until I find one that it works with. When resume does work, training starts fairly immediately (I’ve set display_iters: 50, so I get to see something happening at Command Prompt).
Any thoughts, anyone?