Training network in Docker, but GPU not engaged

Hi all,

I am a PhD student, new to most things computer but I am learning fast and trying to use DLC for my project. I’ve followed all the steps and tutorials and everything seems to be working well so far, but now that I try to train my network in the Docker it is very, VERY slow. Checked the GPU with nvidia-smi, with this output:

Showing 0% usage of the GPU… How do I get the GPU to engage when running DLC? I was under the impression that with tensorflow-gpu (and all other packages installed with DLC) that this was supposed to happen automatically?

Any help and insight is appreciated!

Can you post the command you used to build the Docker? Also, I’m not sure a Quadro K600 is large enough for the network. Lastly, typically you cannot have multiple processes per GPU, and your nvidia-smi shows many :slight_smile:

I followed the instructions as outlined here:

and when starting the Docker with this:

GPU=1 bash ./dlc-docker run -d -p 2351:8888 -e USER_HOME=$HOME/DeepLabCut --name containername dlc_username/dlcdocker

I set the GPU to 0, per the info by nvidia-smi.

I agree about the Quadro K600, unfortunately. You recommend at least 8GB in your notes, and the Quadro only has 1GB, so yeah… We are trying to incorporate DLC in my lab now (with me being the one learning it), but first wanted to get acquainted with the program and see if the Quadro could handle it before jumping the gun and getting a better GPU. My thinking right now is that I will definitely get an upgrade :stuck_out_tongue:

Ah, ok… So, if I follow your youtube-tutorial closely, then I should terminate all processes except /usr/bin/python3, correct (with the kill -9 or kill -15 command to target a specific process, from what I’ve read other places)? I am running the program with two displays, where one display is used for the terminals (one for general use, one for running the DLC and one for the nvidia-smi) and the other is used mainly for reading up on everything regarding this process. Does that mean that I can’t have the Firefox-browser up and running while running DLC? Just asking due to the /usr/lib/firefox/firefox process in the screenshot.