Issue Evaluating Network and Analyzing Videos (maDLC2.2b5)

Hello everyone,

First time posting here and so glad to have multi-animal support for DLC! I am new to coding but I will try to include everything that seems relevant.

I am using Win10 Pro for workstations and a GeForce RTX 2080Ti. I have successfully used DLC 2.1 for single animals several times in the past.

The issue comes when I evaluate the network using the DLC ma2.2b5 GUI. I am able to extract and label frames, create the training data set, train the network (100K iterations), and plot the maps under the evaluation tab.

However I cannot seem to X-validate or analyze the videos. It appears that the relevant pickle file for the model was never created in the traininset95shuffle1 folder.

Thank you so much for any help you can provide,

Kyle

Here is the X-validation error:

Microsoft Windows [Version 10.0.18363.720]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\Fish_Behavior>conda activate dlc-WindowsGPU

(dlc-WindowsGPU) C:\Users\Fish_Behavior>ipython
Python 3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import deeplabcut

In [2]: deeplabcut.launch_dlc()
note to user: currently only ResNet50-v1 is available for maDLC
note to user: currently model comparision is not available in maDLC
C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\dlc-models\iteration-0\5LZF_MODELJun3-trainset95shuffle1\test\inference_cfg.yaml
optimizing parameters using rpck_train as a target...
C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\evaluation-results\iteration-0\5LZF_MODELJun3-trainset95shuffle1  already exists!
C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\evaluation-results\iteration-0\5LZF_MODELJun3-trainset95shuffle1\DLC_resnet50_5LZF_MODELJun3shuffle1_100000-snapshot-100000.h5
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\gui\evaluate_network.py in cross_validate(self, event)
    364             init_points=self.inpts.GetValue(),
    365             n_iter=self.n_iter.GetValue(),
--> 366             target=self.targettypes.GetValue(),
    367         )
    368

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\evaluate_multianimal.py in evaluate_multianimal_crossvalidate(config, Shuffles, trainingsetindex, pbounds, edgewisecondition, target, inferencecfg, init_points, n_iter, dcorr, leastbpts, printingintermediatevalues, modelprefix, plotting)
    565             dcorr=dcorr,
    566             leastbpts=leastbpts,
--> 567             modelprefix=modelprefix,
    568         )
    569

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\lib\crossvalutils.py in bayesian_search(config_path, inferencecfg, pbounds, edgewisecondition, shuffle, trainingsetindex, modelprefix, snapshotindex, target, maximize, init_points, n_iter, acq, log_file, dcorr, leastbpts, printingintermediatevalues)
    338     )
    339     predictionsfn = fns[snapshotindex]
--> 340     data, metadata = auxfun_multianimal.LoadFullMultiAnimalData(predictionsfn)
    341     params = set_up_evaluation(data)
    342     columns = ["train_iter", "train_frac", "shuffle"]

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\utils\auxfun_multianimal.py in LoadFullMultiAnimalData(dataname)
     88 def LoadFullMultiAnimalData(dataname):
     89     """ Save predicted data as h5 file and metadata as pickle file; created by predict_videos.py """
---> 90     with open(dataname.split(".h5")[0] + "_full.pickle", "rb") as handle:
     91         data = pickle.load(handle)
     92     with open(dataname.split(".h5")[0] + "_meta.pickle", "rb") as handle:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Fish_Behavior\\Desktop\\DLCwork\\5LZF_MODEL-Kyle-2020-06-03\\evaluation-results\\iteration-0\\5LZF_MODELJun3-trainset95shuffle1\\DLC_resnet50_5LZF_MODELJun3shuffle1_100000-snapshot-100000_full.pickle'

Here is the Analysis error:

Analyzing ...
Using snapshot-100000 for model C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\dlc-models\iteration-0\5LZF_MODELJun3-trainset95shuffle1
Initializing ResNet
Activating extracting of PAFs
2020-06-10 13:00:32.401256: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2020-06-10 13:00:32.555752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.03GiB
2020-06-10 13:00:32.561936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-06-10 13:00:32.964405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-10 13:00:32.968666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0
2020-06-10 13:00:32.971277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N
2020-06-10 13:00:32.973880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8697 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Starting to analyze %  C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\vids2analyze\20200528_AB_7dpf_5fish_CuSO4.mov
C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\vids2analyze  already exists!
Loading  C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\vids2analyze\20200528_AB_7dpf_5fish_CuSO4.mov
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\gui\analyze_videos.py in analyze_videos(self, event)
    475                 gputouse=None,
    476                 cropping=crop,
--> 477                 robust_nframes=robust,
    478             )
    479             if self.create_video_with_all_detections.GetStringSelection() == "Yes":

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\predict_videos.py in analyze_videos(config, videos, videotype, shuffle, trainingsetindex, gputouse, save_as_csv, destfolder, batchsize, cropping, get_nframesfrommetadata, TFGPUinference, dynamic, modelprefix, c_engine, robust_nframes)
    298                     destfolder,
    299                     c_engine=c_engine,
--> 300                     robust_nframes=robust_nframes,
    301                 )
    302         else:

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\predict_multianimal.py in AnalyzeMultiAnimalVideo(video, DLCscorer, trainFraction, cfg, dlc_cfg, sess, inputs, outputs, pdindex, save_as_csv, destfolder, c_engine, robust_nframes)
     55
     56         if robust_nframes:
---> 57             nframes = auxfun_videos.get_nframes_robust(video)
     58         else:
     59             nframes = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

~\.conda\envs\dlc-WindowsGPU\lib\site-packages\deeplabcut\utils\auxfun_videos.py in get_nframes_robust(videofile)
    291 def get_nframes_robust(videofile):
    292     cmd = f"ffprobe -i {videofile} -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1"
--> 293     output = subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
    294     return int(output)

~\.conda\envs\dlc-WindowsGPU\lib\subprocess.py in check_output(timeout, *popenargs, **kwargs)
    354
    355     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 356                **kwargs).stdout
    357
    358

~\.conda\envs\dlc-WindowsGPU\lib\subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    436         if check and retcode:
    437             raise CalledProcessError(retcode, process.args,
--> 438                                      output=stdout, stderr=stderr)
    439     return CompletedProcess(process.args, retcode, stdout, stderr)
    440

CalledProcessError: Command 'ffprobe -i C:\Users\Fish_Behavior\Desktop\DLCwork\5LZF_MODEL-Kyle-2020-06-03\vids2analyze\20200528_AB_7dpf_5fish_CuSO4.mov -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1' returned non-zero exit status 1.

the ffprobe error affected some users, but also it shows up if there is any typo in the file path to the video, so (1) update to 2.2b6 (as this is a beta release and not stable, expect changes :wink: and (2) double check the video.

see:

thanks for the reply

I upgraded to 2.2b6 and still got the same error. I am trying CoLab right now . . .

how’d colab go? :slight_smile: :slight_smile:

so far I’ve had more success analyzing with CoLab, so I know the videos are not corrupted. If I understand correctly, I must refine the tracklets locally, merge the data, and upload the corrections back into CoLab?