I faced with the cuDNN launch failure : input shape([8,3,709,1248]) filter shape([7,7,3,64]) when I train a network with DLC .
I set the allow_growth flag in train.py, training.py, and train_multianimal.py to True based on this comment cuDNN launch failure - don't think it's memory related?
I also reduced the batch size to 1 in config.yaml and pose_cfg.yaml and cropped our images.
I saw this page https://github.com/DeepLabCut/DeepLabCut/issues/1 as well. I am not sure downgrading cuDNN can help or not as I understood it is a memory issue?
My system has: Ubuntu 18.04
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
CUDA Version 10.0.130
Any hints greatly appreciated!