Problem with shuffle=3 in filterpredictions

If I select shuffle=3 in filterpredictions, I get the following error:

(deeplabcut)[jalal@scc-k11 openfield-Pranav-2019-12-04]$ python mouse.py 
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/evaluation-results/  already exists!
/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/evaluation-results/iteration-0/openfieldDec4-trainset95shuffle1  already exists!
This net has already been evaluated!
Running  DLC_resnet50_openfieldDec4shuffle1_426600  with # of trainingiterations: 426600
Plotting...(attention scale might be inconsistent in comparison to when data was analyzed; i.e. if you used rescale)
/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/evaluation-results/iteration-0/openfieldDec4-trainset95shuffle1/LabeledImages_DLC_resnet50_openfieldDec4shuffle1_426600_snapshot-426600  already exists!
100%|############################################################################################################################################################################| 116/116 [00:20<00:00,  5.81it/s]
The network is evaluated and the results are stored in the subdirectory 'evaluation_results'.
If it generalizes well, choose the best model for prediction and update the config file with the appropriate index for the 'snapshotindex'.
Use the function 'analyze_video' to make predictions on new videos.
Otherwise consider retraining the network (see DeepLabCut workflow Fig 2)
Using snapshot-426600 for model /projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/dlc-models/iteration-0/openfieldDec4-trainset95shuffle1
Initializing ResNet
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Starting to analyze %  /projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/videos/m3v1mp4.mp4
Video already analyzed! /projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/videos/m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600.h5
The videos are analyzed. Now your research can truly start! 
 You can create labeled videos with 'create_labeled_video'.
If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract any outlier frames!
Traceback (most recent call last):
  File "mouse.py", line 12, in <module>
    deeplabcut.filterpredictions(path_config_file, videofile_path, shuffle=3, videotype='.mp4', filtertype='arima', ARdegree=5, MAdegree=2)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/post_processing/filtering.py", line 95, in filterpredictions
    DLCscorer,DLCscorerlegacy=auxiliaryfunctions.GetScorerName(cfg,shuffle,trainFraction = cfg['TrainingFraction'][trainingsetindex])
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/utils/auxiliaryfunctions.py", line 331, in GetScorerName
    Snapshots = np.array([fn.split('.')[0]for fn in os.listdir(modelfolder) if "index" in fn])
FileNotFoundError: [Errno 2] No such file or directory: '/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/dlc-models/iteration-0/openfieldDec4-trainset95shuffle3/train'
(deeplabcut)[jalal@scc-k11 openfield-Pranav-2019-12-04]$ ls /projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/dlc-models/iteration-0/
total 2
drwxr-xr-x 3 jalal ivcgroup 512 Dec  4 22:09 ..
drwxr-xr-x 3 jalal ivcgroup 512 Dec  4 22:09 .
drwxr-xr-x 4 jalal ivcgroup 512 Dec  4 22:09 openfieldDec4-trainset95shuffle1

I have already trained and now I am just analyzing:

Here is the code:

import os
os.environ["DLClight"]="True"
import deeplabcut

path_config_file = "/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/config.yaml"
#deeplabcut.create_training_dataset(path_config_file, augmenter_type='imgaug')
#deeplabcut.train_network(path_config_file, shuffle=1, displayiters=10, saveiters=100, gputouse=1)
deeplabcut.evaluate_network(path_config_file, plotting=True)
videofile_path = ['/projectnb/ivcgroup/jalal/mouse10k/openfield-Pranav-2019-12-04/videos/m3v1mp4.mp4'] #Enter the list of videos to analyze.
deeplabcut.analyze_videos(path_config_file,videofile_path, videotype='.mp4', gputouse=0)
deeplabcut.filterpredictions(path_config_file, videofile_path, shuffle=3, videotype='.mp4', filtertype='arima', ARdegree=5, MAdegree=2)
deeplabcut.create_labeled_video(path_config_file, videofile_path, shuffle=3, filtered=True)
deeplabcut.plot_trajectories(path_config_file, videofile_path, shuffle=3, filtered=True)

Could you please help how to fix it?

I also created a new project and ran the training for 200 iterations and still get the same error:

(deeplabcut)[jalal@scc-k11 openfield-filtered]$ python mouse.py 
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Loaded, now creating training data...
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/labeled-data/m3v1mp4/CollectedData_Pranav.h5  not found (perhaps not annotated)
Annotation data was not found by splitting video paths (from config['video_sets']). An alternative route is taken...
The following folders were found: ['m4s1']
Downloading a ImageNet-pretrained model from http://download.tensorflow.org/models/resnet_v1_101_2016_08_28.tar.gz....
The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/labeled-data/m3v1mp4/CollectedData_Pranav.h5  not found (perhaps not annotated)
Annotation data was not found by splitting video paths (from config['video_sets']). An alternative route is taken...
The following folders were found: ['m4s1']
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//train  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//test  already exists!
The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
Config:
{'all_joints': [[0], [1], [2], [3]],
 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'],
 'batch_size': 1,
 'bottomheight': 400,
 'crop': True,
 'crop_pad': 0,
 'cropratio': 0.4,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/openfield-filtered_Pranav95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deterministic': False,
 'display_iters': 1000,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_101.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'leftwidth': 400,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 0.05,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'max_input_size': 1500,
 'mean_pixel': [123.68, 116.779, 103.939],
 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/Documentation_data-openfield-filtered_95shuffle1.pickle',
 'min_input_size': 64,
 'minsize': 100,
 'mirror': False,
 'multi_step': [[0.005, 10000],
                [0.02, 430000],
                [0.002, 730000],
                [0.001, 1030000]],
 'net_type': 'resnet_101',
 'num_joints': 4,
 'optimizer': 'sgd',
 'pos_dist_thresh': 17,
 'project_path': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered',
 'regularize': False,
 'rightwidth': 400,
 'save_iters': 50000,
 'scale_jitter_lo': 0.5,
 'scale_jitter_up': 1.25,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1/train/snapshot',
 'stride': 8.0,
 'topheight': 400,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
Starting with imgaug pose-dataset loader.
Batch Size is 1
Initializing ResNet
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Loading ImageNet-pretrained resnet_101
WARNING:tensorflow:From /share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Max_iters overwritten as 200
Display_iters overwritten as 10
Save_iters overwritten as 100
Training parameter:
{'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'imgaug', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'], 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/openfield-filtered_Pranav95shuffle1.mat', 'display_iters': 1000, 'init_weights': '/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_101.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/Documentation_data-openfield-filtered_95shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.005, 10000], [0.02, 430000], [0.002, 730000], [0.001, 1030000]], 'net_type': 'resnet_101', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered', 'save_iters': 50000, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2}
Starting training....
iteration: 10 loss: 0.2914 lr: 0.005
iteration: 20 loss: 0.0455 lr: 0.005
iteration: 30 loss: 0.0358 lr: 0.005
iteration: 40 loss: 0.0361 lr: 0.005
iteration: 50 loss: 0.0284 lr: 0.005
iteration: 60 loss: 0.0344 lr: 0.005
iteration: 70 loss: 0.0270 lr: 0.005
iteration: 80 loss: 0.0309 lr: 0.005
iteration: 90 loss: 0.0266 lr: 0.005
iteration: 100 loss: 0.0292 lr: 0.005
iteration: 110 loss: 0.0274 lr: 0.005
iteration: 120 loss: 0.0322 lr: 0.005
iteration: 130 loss: 0.0244 lr: 0.005
iteration: 140 loss: 0.0264 lr: 0.005
iteration: 150 loss: 0.0246 lr: 0.005
iteration: 160 loss: 0.0227 lr: 0.005
iteration: 170 loss: 0.0271 lr: 0.005
iteration: 180 loss: 0.0250 lr: 0.005
iteration: 190 loss: 0.0231 lr: 0.005
iteration: 200 loss: 0.0237 lr: 0.005
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[{{node fifo_queue_enqueue}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py", line 81, in load_and_enqueue
    sess.run(enqueue_op, feed_dict=food)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
	 [[node fifo_queue_enqueue (defined at /projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py:67) ]]

Caused by op 'fifo_queue_enqueue', defined at:
  File "mouse.py", line 8, in <module>
    deeplabcut.train_network(path_config_file, shuffle=1, displayiters=10, saveiters=100, gputouse=0, maxiters=200)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/training.py", line 132, in train_network
    train(str(poseconfigfile),displayiters,saveiters,maxiters,max_to_keep=max_snapshots_to_keep,keepdeconvweights=keepdeconvweights,allow_growth=allow_growth) #pass on path and file name for pose_cfg.yaml!
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py", line 118, in train
    batch, enqueue_op, placeholders = setup_preloading(batch_spec)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py", line 67, in setup_preloading
    enqueue_op = q.enqueue(placeholders_list)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/ops/data_flow_ops.py", line 345, in enqueue
    self._queue_ref, vals, name=scope)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4158, in queue_enqueue_v2
    timeout_ms=timeout_ms, name=name)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/share/pkg.7/tensorflow/1.13.1/install/lib/site-packages/../python3.6-gpu/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

CancelledError (see above for traceback): Enqueue operation was cancelled
	 [[node fifo_queue_enqueue (defined at /projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/train.py:67) ]]


The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network.
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/evaluation-results/  already exists!
Config:
{'all_joints': [[0], [1], [2], [3]],
 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'],
 'batch_size': 1,
 'bottomheight': 400,
 'crop': True,
 'crop_pad': 0,
 'cropratio': 0.4,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/openfield-filtered_Pranav95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deconvolutionstride': 2,
 'deterministic': False,
 'display_iters': 1000,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_101.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'leftwidth': 400,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 0.05,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'max_input_size': 1500,
 'mean_pixel': [123.68, 116.779, 103.939],
 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/Documentation_data-openfield-filtered_95shuffle1.pickle',
 'min_input_size': 64,
 'minsize': 100,
 'mirror': False,
 'multi_step': [[0.005, 10000],
                [0.02, 430000],
                [0.002, 730000],
                [0.001, 1030000]],
 'net_type': 'resnet_101',
 'num_joints': 4,
 'optimizer': 'sgd',
 'output_stride': 16,
 'pos_dist_thresh': 17,
 'project_path': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered',
 'regularize': False,
 'rightwidth': 400,
 'save_iters': 50000,
 'scale_jitter_lo': 0.5,
 'scale_jitter_up': 1.25,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1/test/snapshot',
 'stride': 8.0,
 'topheight': 400,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
Running  DLC_resnet101_openfield-filteredDec4shuffle1_200  with # of trainingiterations: 200
Initializing ResNet
Analyzing data...
116it [00:04, 28.10it/s]
Done and results stored for snapshot:  snapshot-200
Results for 200  training iterations: 95 1 train error: 37.58 pixels. Test error: 36.87  pixels.
With pcutoff of 0.1  train error: 37.58 pixels. Test error: 36.87 pixels
Thereby, the errors are given by the average distances between the labels by DLC and the scorer.
Plotting...
100%|############################################################################################################################################################################| 116/116 [00:19<00:00,  5.77it/s]
The network is evaluated and the results are stored in the subdirectory 'evaluation_results'.
If it generalizes well, choose the best model for prediction and update the config file with the appropriate index for the 'snapshotindex'.
Use the function 'analyze_video' to make predictions on new videos.
Otherwise consider retraining the network (see DeepLabCut workflow Fig 2)
Config:
{'all_joints': [[0], [1], [2], [3]],
 'all_joints_names': ['snout', 'leftear', 'rightear', 'tailbase'],
 'batch_size': 1,
 'bottomheight': 400,
 'crop': True,
 'crop_pad': 0,
 'cropratio': 0.4,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/openfield-filtered_Pranav95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deconvolutionstride': 2,
 'deterministic': False,
 'display_iters': 1000,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_101.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'leftwidth': 400,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 0.05,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'max_input_size': 1500,
 'mean_pixel': [123.68, 116.779, 103.939],
 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4/Documentation_data-openfield-filtered_95shuffle1.pickle',
 'min_input_size': 64,
 'minsize': 100,
 'mirror': False,
 'multi_step': [[0.005, 10000],
                [0.02, 430000],
                [0.002, 730000],
                [0.001, 1030000]],
 'net_type': 'resnet_101',
 'num_joints': 4,
 'optimizer': 'sgd',
 'output_stride': 16,
 'pos_dist_thresh': 17,
 'project_path': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered',
 'regularize': False,
 'rightwidth': 400,
 'save_iters': 50000,
 'scale_jitter_lo': 0.5,
 'scale_jitter_up': 1.25,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1/test/snapshot',
 'stride': 8.0,
 'topheight': 400,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
Using snapshot-200 for model /projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1
Initializing ResNet
Starting to analyze %  /projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/videos/m3v1mp4.mp4
Loading  /projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/videos/m3v1mp4.mp4
Duration of video [s]:  77.67 , recorded with  30.0 fps!
Overall # of frames:  2330  found with (before cropping) frame dimensions:  640 480
Starting to extract posture
2346it [00:35, 68.48it/s]                                                                                                                                                                                          Detected frames:  2330

Saving results in /projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/videos...
The videos are analyzed. Now your research can truly start! 
 You can create labeled videos with 'create_labeled_video'.
If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract any outlier frames!
Traceback (most recent call last):
  File "mouse.py", line 12, in <module>
    deeplabcut.filterpredictions(path_config_file, videofile_path, shuffle=3, videotype='.mp4', filtertype='arima', ARdegree=5, MAdegree=2)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/post_processing/filtering.py", line 95, in filterpredictions
    DLCscorer,DLCscorerlegacy=auxiliaryfunctions.GetScorerName(cfg,shuffle,trainFraction = cfg['TrainingFraction'][trainingsetindex])
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/utils/auxiliaryfunctions.py", line 331, in GetScorerName
    Snapshots = np.array([fn.split('.')[0]for fn in os.listdir(modelfolder) if "index" in fn])
FileNotFoundError: [Errno 2] No such file or directory: '/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle3/train'
(deeplabcut)[jalal@scc-k11 openfield-filtered]$

log of all files in this projects:

(deeplabcut)[jalal@scc-k11 openfield-filtered]$ tree .
.
|-- config.yaml
|-- dlc-models
|   `-- iteration-0
|       `-- openfield-filteredDec4-trainset95shuffle1
|           |-- test
|           |   `-- pose_cfg.yaml
|           `-- train
|               |-- checkpoint
|               |-- learning_stats.csv
|               |-- log
|               |   `-- events.out.tfevents.1575579029.scc-k11
|               |-- log.txt
|               |-- pose_cfg.yaml
|               |-- snapshot-100.data-00000-of-00001
|               |-- snapshot-100.index
|               |-- snapshot-100.meta
|               |-- snapshot-200.data-00000-of-00001
|               |-- snapshot-200.index
|               `-- snapshot-200.meta
|-- evaluation-results
|   `-- iteration-0
|       `-- openfield-filteredDec4-trainset95shuffle1
|           |-- DLC_resnet101_openfield-filteredDec4shuffle1_200-results.csv
|           |-- DLC_resnet101_openfield-filteredDec4shuffle1_200-results.h5
|           |-- DLC_resnet101_openfield-filteredDec4shuffle1_200-snapshot-200.h5
|           `-- LabeledImages_DLC_resnet101_openfield-filteredDec4shuffle1_200_snapshot-200
|               |-- Test-m4s1-img0044.png
|               |-- Test-m4s1-img0079.png
|               |-- Test-m4s1-img0088.png
|               |-- Test-m4s1-img0103.png
|               |-- Test-m4s1-img0104.png
|               |-- Test-m4s1-img0109.png
|               |-- Training-m4s1-img0000.png
|               |-- Training-m4s1-img0001.png
|               |-- Training-m4s1-img0002.png
|               |-- Training-m4s1-img0003.png
|               |-- Training-m4s1-img0004.png
|               |-- Training-m4s1-img0005.png
|               |-- Training-m4s1-img0006.png
|               |-- Training-m4s1-img0007.png
|               |-- Training-m4s1-img0008.png
|               |-- Training-m4s1-img0009.png
|               |-- Training-m4s1-img0010.png
|               |-- Training-m4s1-img0011.png
|               |-- Training-m4s1-img0012.png
|               |-- Training-m4s1-img0013.png
|               |-- Training-m4s1-img0014.png
|               |-- Training-m4s1-img0015.png
|               |-- Training-m4s1-img0016.png
|               |-- Training-m4s1-img0017.png
|               |-- Training-m4s1-img0018.png
|               |-- Training-m4s1-img0019.png
|               |-- Training-m4s1-img0020.png
|               |-- Training-m4s1-img0021.png
|               |-- Training-m4s1-img0022.png
|               |-- Training-m4s1-img0023.png
|               |-- Training-m4s1-img0024.png
|               |-- Training-m4s1-img0025.png
|               |-- Training-m4s1-img0026.png
|               |-- Training-m4s1-img0027.png
|               |-- Training-m4s1-img0028.png
|               |-- Training-m4s1-img0029.png
|               |-- Training-m4s1-img0030.png
|               |-- Training-m4s1-img0031.png
|               |-- Training-m4s1-img0032.png
|               |-- Training-m4s1-img0033.png
|               |-- Training-m4s1-img0034.png
|               |-- Training-m4s1-img0035.png
|               |-- Training-m4s1-img0036.png
|               |-- Training-m4s1-img0037.png
|               |-- Training-m4s1-img0038.png
|               |-- Training-m4s1-img0039.png
|               |-- Training-m4s1-img0040.png
|               |-- Training-m4s1-img0041.png
|               |-- Training-m4s1-img0042.png
|               |-- Training-m4s1-img0043.png
|               |-- Training-m4s1-img0045.png
|               |-- Training-m4s1-img0046.png
|               |-- Training-m4s1-img0047.png
|               |-- Training-m4s1-img0048.png
|               |-- Training-m4s1-img0049.png
|               |-- Training-m4s1-img0050.png
|               |-- Training-m4s1-img0051.png
|               |-- Training-m4s1-img0052.png
|               |-- Training-m4s1-img0053.png
|               |-- Training-m4s1-img0054.png
|               |-- Training-m4s1-img0055.png
|               |-- Training-m4s1-img0056.png
|               |-- Training-m4s1-img0057.png
|               |-- Training-m4s1-img0058.png
|               |-- Training-m4s1-img0059.png
|               |-- Training-m4s1-img0060.png
|               |-- Training-m4s1-img0061.png
|               |-- Training-m4s1-img0062.png
|               |-- Training-m4s1-img0063.png
|               |-- Training-m4s1-img0064.png
|               |-- Training-m4s1-img0065.png
|               |-- Training-m4s1-img0066.png
|               |-- Training-m4s1-img0067.png
|               |-- Training-m4s1-img0068.png
|               |-- Training-m4s1-img0069.png
|               |-- Training-m4s1-img0070.png
|               |-- Training-m4s1-img0071.png
|               |-- Training-m4s1-img0072.png
|               |-- Training-m4s1-img0073.png
|               |-- Training-m4s1-img0074.png
|               |-- Training-m4s1-img0075.png
|               |-- Training-m4s1-img0076.png
|               |-- Training-m4s1-img0077.png
|               |-- Training-m4s1-img0078.png
|               |-- Training-m4s1-img0080.png
|               |-- Training-m4s1-img0081.png
|               |-- Training-m4s1-img0082.png
|               |-- Training-m4s1-img0083.png
|               |-- Training-m4s1-img0084.png
|               |-- Training-m4s1-img0085.png
|               |-- Training-m4s1-img0086.png
|               |-- Training-m4s1-img0087.png
|               |-- Training-m4s1-img0089.png
|               |-- Training-m4s1-img0090.png
|               |-- Training-m4s1-img0091.png
|               |-- Training-m4s1-img0092.png
|               |-- Training-m4s1-img0093.png
|               |-- Training-m4s1-img0094.png
|               |-- Training-m4s1-img0095.png
|               |-- Training-m4s1-img0096.png
|               |-- Training-m4s1-img0097.png
|               |-- Training-m4s1-img0098.png
|               |-- Training-m4s1-img0099.png
|               |-- Training-m4s1-img0100.png
|               |-- Training-m4s1-img0101.png
|               |-- Training-m4s1-img0102.png
|               |-- Training-m4s1-img0105.png
|               |-- Training-m4s1-img0106.png
|               |-- Training-m4s1-img0107.png
|               |-- Training-m4s1-img0108.png
|               |-- Training-m4s1-img0110.png
|               |-- Training-m4s1-img0111.png
|               |-- Training-m4s1-img0112.png
|               |-- Training-m4s1-img0113.png
|               |-- Training-m4s1-img0114.png
|               `-- Training-m4s1-img0115.png
|-- labeled-data
|   `-- m4s1
|       |-- CollectedData_Pranav.csv
|       |-- CollectedData_Pranav.h5
|       |-- img0000.png
|       |-- img0001.png
|       |-- img0002.png
|       |-- img0003.png
|       |-- img0004.png
|       |-- img0005.png
|       |-- img0006.png
|       |-- img0007.png
|       |-- img0008.png
|       |-- img0009.png
|       |-- img0010.png
|       |-- img0011.png
|       |-- img0012.png
|       |-- img0013.png
|       |-- img0014.png
|       |-- img0015.png
|       |-- img0016.png
|       |-- img0017.png
|       |-- img0018.png
|       |-- img0019.png
|       |-- img0020.png
|       |-- img0021.png
|       |-- img0022.png
|       |-- img0023.png
|       |-- img0024.png
|       |-- img0025.png
|       |-- img0026.png
|       |-- img0027.png
|       |-- img0028.png
|       |-- img0029.png
|       |-- img0030.png
|       |-- img0031.png
|       |-- img0032.png
|       |-- img0033.png
|       |-- img0034.png
|       |-- img0035.png
|       |-- img0036.png
|       |-- img0037.png
|       |-- img0038.png
|       |-- img0039.png
|       |-- img0040.png
|       |-- img0041.png
|       |-- img0042.png
|       |-- img0043.png
|       |-- img0044.png
|       |-- img0045.png
|       |-- img0046.png
|       |-- img0047.png
|       |-- img0048.png
|       |-- img0049.png
|       |-- img0050.png
|       |-- img0051.png
|       |-- img0052.png
|       |-- img0053.png
|       |-- img0054.png
|       |-- img0055.png
|       |-- img0056.png
|       |-- img0057.png
|       |-- img0058.png
|       |-- img0059.png
|       |-- img0060.png
|       |-- img0061.png
|       |-- img0062.png
|       |-- img0063.png
|       |-- img0064.png
|       |-- img0065.png
|       |-- img0066.png
|       |-- img0067.png
|       |-- img0068.png
|       |-- img0069.png
|       |-- img0070.png
|       |-- img0071.png
|       |-- img0072.png
|       |-- img0073.png
|       |-- img0074.png
|       |-- img0075.png
|       |-- img0076.png
|       |-- img0077.png
|       |-- img0078.png
|       |-- img0079.png
|       |-- img0080.png
|       |-- img0081.png
|       |-- img0082.png
|       |-- img0083.png
|       |-- img0084.png
|       |-- img0085.png
|       |-- img0086.png
|       |-- img0087.png
|       |-- img0088.png
|       |-- img0089.png
|       |-- img0090.png
|       |-- img0091.png
|       |-- img0092.png
|       |-- img0093.png
|       |-- img0094.png
|       |-- img0095.png
|       |-- img0096.png
|       |-- img0097.png
|       |-- img0098.png
|       |-- img0099.png
|       |-- img0100.png
|       |-- img0101.png
|       |-- img0102.png
|       |-- img0103.png
|       |-- img0104.png
|       |-- img0105.png
|       |-- img0106.png
|       |-- img0107.png
|       |-- img0108.png
|       |-- img0109.png
|       |-- img0110.png
|       |-- img0111.png
|       |-- img0112.png
|       |-- img0113.png
|       |-- img0114.png
|       `-- img0115.png
|-- log
|-- mouse.py
|-- training-datasets
|   `-- iteration-0
|       `-- UnaugmentedDataSet_openfield-filteredDec4
|           |-- CollectedData_Pranav.csv
|           |-- CollectedData_Pranav.h5
|           |-- Documentation_data-openfield-filtered_95shuffle1.pickle
|           `-- openfield-filtered_Pranav95shuffle1.mat
`-- videos
    |-- m3v1mp4.mp4
    |-- m3v1mp4DLC_resnet101_openfield-filteredDec4shuffle1_200.h5
    |-- m3v1mp4DLC_resnet101_openfield-filteredDec4shuffle1_200includingmetadata.pickle
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_1200.h5
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_1200_labeled.mp4
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_1200includingmetadata.pickle
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600.h5
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600_filtered._labeled.mp4
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600_labeled.mp4
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600filtered.csv
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600filtered.h5
    |-- m3v1mp4DLC_resnet50_openfieldDec4shuffle1_426600includingmetadata.pickle
    `-- plot-poses
        `-- m3v1mp4
            |-- hist.png
            |-- hist_filtered..png
            |-- plot-likelihood.png
            |-- plot-likelihood_filtered..png
            |-- plot.png
            |-- plot_filtered..png
            |-- trajectory.png
            `-- trajectory_filtered..png

18 directories, 276 files
(deeplabcut)[jalal@scc-k11 openfield-filtered]$

when I change the shuffle=3 in train I get this error:


(deeplabcut)[jalal@scc-k11 openfield-filtered]$ python mouse.py 
DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)
Loaded, now creating training data...
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/labeled-data/m3v1mp4/CollectedData_Pranav.h5  not found (perhaps not annotated)
Annotation data was not found by splitting video paths (from config['video_sets']). An alternative route is taken...
The following folders were found: ['m4s1']
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//train  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//test  already exists!
The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/training-datasets/iteration-0/UnaugmentedDataSet_openfield-filteredDec4  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/labeled-data/m3v1mp4/CollectedData_Pranav.h5  not found (perhaps not annotated)
Annotation data was not found by splitting video paths (from config['video_sets']). An alternative route is taken...
The following folders were found: ['m4s1']
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//train  already exists!
/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle1//test  already exists!
The training dataset is successfully created. Use the function 'train_network' to start training. Happy training!
The training datafile  /projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/dlc-models/iteration-0/openfield-filteredDec4-trainset95shuffle3/train/pose_cfg.yaml  is not present.
Probably, the training dataset for this specific shuffle index was not created.
Try with a different shuffle/trainingsetfraction or use function 'create_training_dataset' to create a new trainingdataset with this shuffle index.
Traceback (most recent call last):
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/evaluate.py", line 296, in evaluate_network
    Snapshots[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mouse.py", line 9, in <module>
    deeplabcut.evaluate_network(path_config_file, plotting=True)
  File "/projectnb/dnn-motion/jalal/.conda/envs/deeplabcut/lib/python3.6/site-packages/deeplabcut/pose_estimation_tensorflow/evaluate.py", line 298, in evaluate_network
    raise FileNotFoundError("Snapshots not found! It seems the dataset for shuffle %s and trainFraction %s is not trained.\nPlease train it before evaluating.\nUse the function 'train_network' to do so."%(shuffle,trainFraction))
FileNotFoundError: Snapshots not found! It seems the dataset for shuffle 1 and trainFraction 0.95 is not trained.
Please train it before evaluating.
Use the function 'train_network' to do so.
(deeplabcut)[jalal@scc-k11 openfield-filtered]$ cat mouse.py 
import os
os.environ["DLClight"]="True"
import deeplabcut

path_config_file = "/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/config.yaml"
deeplabcut.load_demo_data(path_config_file)
deeplabcut.create_training_dataset(path_config_file, augmenter_type='imgaug')
deeplabcut.train_network(path_config_file, shuffle=3, displayiters=10, saveiters=100, gputouse=0, maxiters=200)
deeplabcut.evaluate_network(path_config_file, plotting=True)
videofile_path = ['/projectnb/ivcgroup/jalal/mouse1M/openfield-filtered/videos/m3v1mp4.mp4'] #Enter the list of videos to analyze.
deeplabcut.analyze_videos(path_config_file,videofile_path, videotype='.mp4', gputouse=0)
deeplabcut.filterpredictions(path_config_file, videofile_path, shuffle=3, videotype='.mp4', filtertype='arima', ARdegree=5, MAdegree=2)
deeplabcut.create_labeled_video(path_config_file, videofile_path, shuffle=3, filtered=True, save_frames=True, draw_skeleton=True)
deeplabcut.plot_trajectories(path_config_file, videofile_path, shuffle=3, filtered=True)

How can I use shuffle=3 in filterpredictions without getting error or in train?

Try with a different shuffle/trainingsetfraction or use function ‘create_training_dataset’ to create a new trainingdataset with this shuffle index.

  • Do you have a network that is labeled shuffle 3? None of your code you shared suggests you have this, ie you’re not analyzing shuffle 3, so drop shuffle=3 and it should work fine.