Train_network error

Hi,

at the initiation of deeplabcut.train_network(config_path), I encountered error saying (failed to get convolution algorithm. This is probably because cuDNN failed to initialize.).
I have checked other forum, and found that this code can solve the problem.

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tfSession(config=config)

This code did solved the problem and training was started, but training was done only in CPU, not using GPU.
Is there any way that I can do to use GPU for training?

Hardware Information
Machine : AMD64
CPU Name : skylake
CPU count : 12
CPU Features :
64bit adx aes avx avx2 bmi bmi2 clflushopt cmov cx16 f16c fma fsgsbase invpcid
lzcnt mmx movbe pclmul popcnt prfchw rdrnd rdseed sahf sgx sse sse2 sse3 sse4.1
sse4.2 ssse3 xsave xsavec xsaveopt xsaves

OS Information
Platform : Windows-10-10.0.17763-SP0
Release : 10
System Name : Windows
Version : 10.0.17763
OS specific info : 1010.0.17763SP0Multiprocessor Free

Python Information
Python Compiler : MSC v.1915 64 bit (AMD64)
Python Implementation : CPython
Python Version : 3.7.3
Python Locale : ko_KR cp949

LLVM information
LLVM version : 8.0.0

CUDA Information
Found 1 CUDA devices
id 0 b’GeForce GTX 1660 Ti’ [SUPPORTED]
compute capability: 7.5
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported
CUDA driver version : 10010
CUDA libraries:
Finding cublas from
ERROR: can’t locate lib
Finding cusparse from
ERROR: can’t locate lib
Finding cufft from
ERROR: can’t locate lib
Finding curand from
ERROR: can’t locate lib
Finding nvvm from
ERROR: can’t locate lib
Finding libdevice from
searching for compute_20… ERROR: can’t open libdevice for compute_20
searching for compute_30… ERROR: can’t open libdevice for compute_30
searching for compute_35… ERROR: can’t open libdevice for compute_35
searching for compute_50… ERROR: can’t open libdevice for compute_50

ROC Information
ROC available : False
Error initialising ROC due to : No ROC toolchains found.
No HSA Agents found, encountered exception when searching:
Error at driver init:

HSA is not currently supported on this platform (win32).
:

SVML Information
SVML state, config.USING_SVML : True
SVML library found and loaded : True
llvmlite using SVML patched LLVM : True
SVML operational : True

Threading Layer Information
TBB Threading layer available : False
±-> Disabled due to : Unknown import problem.
OpenMP Threading layer available : True
Workqueue Threading layer available : True

Numba Environment Variable Information
None set.

Conda Information
conda_build_version : 3.18.8
conda_env_version : 4.7.10
platform : win-64
python_version : 3.7.3.final.0
root_writable : True

Current Conda Env
_tflow_select 2.1.0 gpu
absl-py 0.8.0 py36_0
astor 0.8.0 py36_0
attrs 19.1.0 py36_1
backcall 0.1.0 py36_0
blas 1.0 mkl
bleach 3.1.0 py36_0
blosc 1.16.3 h7bd577a_0
bzip2 1.0.8 he774522_0
ca-certificates 2019.8.28 0 anaconda
certifi 2019.9.11 py36_0
chardet 3.0.4 pypi_0 pypi
click 7.0 pypi_0 pypi
cloudpickle 1.2.2 pypi_0 pypi
colorama 0.4.1 py36_0
cudatoolkit 10.0.130 0 anaconda
cudnn 7.6.0 cuda10.0_0
cycler 0.10.0 pypi_0 pypi
decorator 4.4.0 py36_1
deeplabcut 2.0.9 pypi_0 pypi
defusedxml 0.6.0 py_0
easydict 1.9 pypi_0 pypi
entrypoints 0.3 py36_0
gast 0.3.2 py_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
idna 2.8 pypi_0 pypi
imageio 2.3.0 pypi_0 pypi
intel-openmp 2019.0 pypi_0 pypi
ipykernel 5.1.2 py36h39e3cac_0
ipython 6.0.0 pypi_0 pypi
ipython_genutils 0.2.0 py36h3c5d0ee_0
ipywidgets 7.5.1 py_0
jedi 0.15.1 py36_0
jinja2 2.10.1 py36_0
jpeg 9b hb83a4c4_2
jsonschema 3.0.2 py36_0
jupyter 1.0.0 py36_7
jupyter_client 5.3.3 py36_1
jupyter_console 6.0.0 py36_0
jupyter_core 4.5.0 py_0
keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 pypi_0 pypi
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.9.2 h7bd577a_0
libsodium 1.0.16 h9d3ae62_0
lz4-c 1.8.1.2 h2fa13f4_0
lzo 2.10 h6df0209_2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markdown 3.1.1 py36_0
markupsafe 1.1.1 py36he774522_0
matplotlib 3.0.3 pypi_0 pypi
mistune 0.8.4 py36he774522_0
mkl 2018.0.3 1
mkl_fft 1.0.6 py36hdbbee80_0
mkl_random 1.0.1 py36h77b88f5_1
mock 3.0.5 py36_0
moviepy 0.2.3.5 pypi_0 pypi
msgpack 0.6.2 pypi_0 pypi
msgpack-numpy 0.4.4.3 pypi_0 pypi
msys2-conda-epoch 20160418 1
nbconvert 5.6.0 py36_1
nbformat 4.4.0 py36h3a5bc1b_0
networkx 2.3 pypi_0 pypi
notebook 6.0.1 py36_0
numexpr 2.6.8 py36h9ef55f4_0
numpy 1.14.5 py36h9fa60d3_4
numpy-base 1.14.5 py36h5c71026_4
opencv-python 3.4.7.28 pypi_0 pypi
openssl 1.1.1 he774522_0 anaconda
pandas 0.25.1 pypi_0 pypi
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py36_1
parso 0.5.1 py_0
patsy 0.5.1 pypi_0 pypi
pickleshare 0.7.5 py36_0
pillow 6.1.0 pypi_0 pypi
pip 19.2.3 py36_0
prometheus_client 0.7.1 py_0
prompt-toolkit 1.0.16 pypi_0 pypi
prompt_toolkit 2.0.9 py36_0 anaconda
protobuf 3.9.2 py36h33f27b4_0
psutil 5.6.3 pypi_0 pypi
pygments 2.4.2 py_0
pyparsing 2.4.2 pypi_0 pypi
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
pyrsistent 0.15.4 py36he774522_0
pytables 3.4.4 py36h1da0976_0
python 3.6.9 h5500b2f_0
python-dateutil 2.7.5 pypi_0 pypi
pytz 2019.2 pypi_0 pypi
pywavelets 1.0.3 pypi_0 pypi
pywin32 223 py36hfa6e2cd_1
pywinpty 0.5.5 py36_1000
pyyaml 5.1.2 pypi_0 pypi
pyzmq 18.1.0 py36ha925a31_0
qt 5.9.7 vc14h73c81de_0
qtconsole 4.5.5 py_0
requests 2.22.0 pypi_0 pypi
ruamel-yaml 0.16.5 pypi_0 pypi
ruamel-yaml-clib 0.2.0 pypi_0 pypi
scikit-image 0.14.5 pypi_0 pypi
scikit-learn 0.19.2 pypi_0 pypi
scipy 1.1.0 pypi_0 pypi
send2trash 1.5.0 py36_0
setuptools 41.2.0 py36_0
simplegeneric 0.8.1 pypi_0 pypi
sip 4.19.8 py36h6538335_0
six 1.11.0 pypi_0 pypi
snappy 1.1.7 h777316e_3
sqlite 3.29.0 he774522_0
statsmodels 0.9.0 pypi_0 pypi
tabulate 0.8.5 pypi_0 pypi
tensorboard 1.13.1 py36h33f27b4_0
tensorflow 1.13.1 gpu_py36h9006a92_0
tensorflow-base 1.13.1 gpu_py36h871c8ca_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.13.1 h0d30ee6_0
tensorpack 0.9.7.1 pypi_0 pypi
termcolor 1.1.0 py36_1
terminado 0.8.2 py36_0
testpath 0.4.2 py36_0
tornado 6.0.3 py36he774522_0
tqdm 4.36.1 pypi_0 pypi
traitlets 4.3.2 py36h096827d_0
urllib3 1.25.6 pypi_0 pypi
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_0
wcwidth 0.1.7 py36h3d5aa90_0
webencodings 0.5.1 py36_1
werkzeug 0.16.0 py_0
wheel 0.31.1 pypi_0 pypi
widgetsnbextension 3.5.1 py36_0
wincertstore 0.2 py36h7fe50ca_0
winpty 0.4.3 4
wxpython 4.0.4 py36ha925a31_0
xz 5.2.4 h2fa13f4_4
zeromq 4.3.1 h33f27b4_3
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0

Sorry, I found that even though the Task Manager show very low GPU usage, when I checked with cmd-> nvidia-smi, GPU was actually working.

1 Like