Deeplabcut is default to take all the memory usage of GPUs

Usage & Issues
#1

every time I run training or analysis_videos in GPU, they take almost all the memory usage of all the GPUs I have, even only one of the GPU-Util is used, other of them can do nothing but occupied.gpu

I just want to run two or more task at the same time to fully use my GPUs, does any one know some idea about these?

1 Like
#2

here is a correction: only analysis_video will occupy all the memory but not training.

1 Like
#3

is this on Windows? A bit more context could be helpful, thanks!

#4

Check out the answer here:

#5

Thanks for your reply. It’s on linux(centos) with LSF, docker is not compatible here.

#6

Thanks a lot. I will try first, then I’ll tell you whether it work here.

#7

I have had a try
I define gputouse = 2 like this

image
image1118×50 4.37 KB

but it will still occupy all the memory when analyze videos like this:
image
is there somewhere i have mad a mistake or else?
thanks

#8

anothre correction: even when training where the memory is not all be taken up, the analyze_video can still not be perform. an error like " failed to create session" showed up.
image

image
image1128×91 6.98 KB

my OS is centos with LSF(IBM) in which docker is uncompatible.
I am wondering whether it’s because there is no docker?

1 Like
#9

Last correction today, It works if both tasks are training, but there could be and only be one of the analysis_video task. It’s strange. I am wandoring it’s a bug or an intentional setting…
Sorry for the intermittent corrections, I hope those won’t let you confused.

1 Like
#10

Thanks for all the details. I looked into the code again and noticed that the environment variable is set after the TF session is initialized (for the predict code). I just swapped the order and updated the github repo (not pypi so far). Could you please download 2.0.6.3 and check if it works now for you?

#11

Ok, solved: https://github.com/AlexEMG/DeepLabCut/issues/272

#12

Yes, it works. Thank you very much. You have done a great job.:100:

2 Likes
#13

It has been a long time that I use deeplabcut for tracking animals. it does help a lot, thanks. and nowadays, i am configuring deeplacut in another computer which gets UBUNTU18.04 LTS and 1070ti.
i got several questions since i have not upgrade my deeplabcut since deeplabcut 2.0.6.3 which could run in cuda 9. and I run the latest deepacut in cuda 10.0 in this computer, I found the occupation of memory of GPU have dramatically cut down to less than 100MB. does this thank to your optimization or just my wrong configuration?

#14

and suddenly it goes wrong when a choose a directory to deeplabcut.label_frames() in my labtop (just for label frames). I am sure I did not change anything since I used the deeplabcut last time.

image
image898×259 16 KB

I guess whether it is the wrong version of wxPython, but it’s still there whether it’s wxPython 4.0.3 or 4.0.6.

#15

TensorFlow version and DLC versions are almost completely independent. Thus, you could have sticked with cuda 9 and whatever TF you used, btw…

#16

Too me the error looks like there is no image?

#17

I restart the computer, it works. thanks for your reply

#18

it turns out to be that the video name which contains special characters will lead to the bug.

2 Likes
#19

Check out these https://github.com/AlexEMG/DeepLabCut/pull/458

#20

I came across this thread with a similar question but I have one GPU (NVIDIA Quadro RTX6000, 24GB). training and analyzing videos takes up the entire GPU when I doubt it needs it all. Is there a way to prevent the entire gpu from being engaged with DLC? It looks like something with allow_growth=true but I’m not sure where or how to add that in the GUI

1 Like