Could not create cudnn handle

For deeplabcut specifically, there are two main reasons you hit this error:

(1) wrong versions, see my first answer.

(2) too small of a GPU; to limit this, we do not allow_growth automatically, as it can crash. But, if you want to change this you can actually just pass this when you train (vs. setting env variable):

train_network(config, shuffle=1, trainingsetindex=0, max_snapshots_to_keep=5, displayiters=None, saveiters=None, maxiters=None, allow_growth=False, gputouse=None, autotune=False, keepdeconvweights=True, modelprefix='')

i.e., just pass allow_growth=True

I have hit the same error when running the video analysis. Not as critical as I can just analyze on the CPU, but though I’d mention it.

The memory seems to fill up. I had the same issue with training but solved it with allow_growth=True.

RTX 2060, Ubuntu 20.04, Cuda 10.0, tensorflow, 1.13.1, DeepLabCut 2.2b8

1 Like