I am having an issue when during training (Win 10, DLC 2.0). After running deeplabcut.train_network(path_config_file), and when the snapshot files are going to be written (100K iterations), the program crashes with the following Error:
... Caused by op 'save_1/SaveV2', defined at: File "<stdin>", line 1, in <module> File "D:\DeepLabCut2.0\deeplabcut\pose_estimation_tensorflow\training.py", line 79, in train_network train(str(poseconfigfile),displayiters,saveiters,maxiters,max_to_keep=max_snapshots_to_keep) #pass on path and file name for pose_cfg.yaml! File "D:\DeepLabCut2.0\deeplabcut\pose_estimation_tensorflow\train.py", line 98, in train saver = tf.train.Saver(max_to_keep=max_to_keep) # selects how many snapshots are stored, see https://github.com/AlexEMG/DeepLabCut/issues/8#issuecomment-387404835 File "D:\dlc2.0\deeplabcut\lib\site-packages\tensorflow\python\training\saver.py", line 1102, in __init__ self.build() File "D:\dlc2.0\deeplabcut\lib\site-packages\tensorflow\python\training\saver.py", line 1114, in build self._build(self._filename, build_save=True, build_restore=True) ... FailedPreconditionError (see above for traceback): Failed to rename: D:\DeepLabCut2.0\file-29apr2019-Edgar-2019-05-01\dlc-models\iteration-1\file-29apr2019May1-trainset80shuffle1\train\snapshot-100000.data-00000-of-00001.tempstate1095526838806107042 to: D:\DeepLabCut2.0\file-29apr2019-Edgar-2019-05-01\dlc-models\iteration-1\file-29apr2019May1-trainset80shuffle1\train\snapshot-100000.data-00000-of-00001 : The process cannot access the file because it is being used by another process.
I seem to remember that this issue was posted before somewhere but I can not find it. Any idea why this is happening or how to solve it?