Resource Exhausted Error: OOM when allocating tensor

I’ve been trying to run DeepLabCut on a laptop using the GPU (it is a Quadro P2000 4GB device). The train_network function runs for me for a few (~200-300) iterations, then bails with an error indicating a memory allocation problem. Error message looks like this:

ResourceExhaustedError: OOM when allocating tensor with shape[1,2048,64,114] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Any advice on a workaround? Does this machine just simply have insufficient video card memory?

Unfortunately it is likely your frames are too large for only 4GB of memory; you can downsize your videos before extraction and labeling the frames and not sacrifice DLC performance, see our preprint here: https://www.biorxiv.org/content/early/2018/10/30/457242

Reducing video size resolved the issue, thanks for the advice!

1 Like

no problem, glad it’s up and training!