Deeplabcut is default to take all the memory usage of GPUs

every time I run training or analysis_videos in GPU, they take almost all the memory usage of all the GPUs I have, even only one of the GPU-Util is used, other of them can do nothing but occupied.gpu

I just want to run two or more task at the same time to fully use my GPUs, does any one know some idea about these?

1 Like

here is a correction: only analysis_video will occupy all the memory but not training.

1 Like

is this on Windows? A bit more context could be helpful, thanks!

Check out the answer here:

Thanks for your reply. It’s on linux(centos) with LSF, docker is not compatible here.

Thanks a lot. I will try first, then I’ll tell you whether it work here.

I have had a try
I define gputouse = 2 like this


but it will still occupy all the memory when analyze videos like this:
image
is there somewhere i have mad a mistake or else?
thanks

anothre correction: even when training where the memory is not all be taken up, the analyze_video can still not be perform. an error like " failed to create session" showed up.
image


my OS is centos with LSF(IBM) in which docker is uncompatible.
I am wondering whether it’s because there is no docker?

1 Like

Last correction today, It works if both tasks are training, but there could be and only be one of the analysis_video task. It’s strange. I am wandoring it’s a bug or an intentional setting…
Sorry for the intermittent corrections, I hope those won’t let you confused.

1 Like

Thanks for all the details. I looked into the code again and noticed that the environment variable is set after the TF session is initialized (for the predict code). I just swapped the order and updated the github repo (not pypi so far). Could you please download 2.0.6.3 and check if it works now for you?

Ok, solved: https://github.com/AlexEMG/DeepLabCut/issues/272

Yes, it works. Thank you very much. You have done a great job.:100:

2 Likes

It has been a long time that I use deeplabcut for tracking animals. it does help a lot, thanks. and nowadays, i am configuring deeplacut in another computer which gets UBUNTU18.04 LTS and 1070ti.
i got several questions since i have not upgrade my deeplabcut since deeplabcut 2.0.6.3 which could run in cuda 9. and I run the latest deepacut in cuda 10.0 in this computer, I found the occupation of memory of GPU have dramatically cut down to less than 100MB. does this thank to your optimization or just my wrong configuration?

and suddenly it goes wrong when a choose a directory to deeplabcut.label_frames() in my labtop (just for label frames). I am sure I did not change anything since I used the deeplabcut last time.


I guess whether it is the wrong version of wxPython, but it’s still there whether it’s wxPython 4.0.3 or 4.0.6.

TensorFlow version and DLC versions are almost completely independent. Thus, you could have sticked with cuda 9 and whatever TF you used, btw…

Too me the error looks like there is no image?

I restart the computer, it works. thanks for your reply

it turns out to be that the video name which contains special characters will lead to the bug.

2 Likes

Check out these https://github.com/AlexEMG/DeepLabCut/pull/458