Train/Test RMSE vs. iteration plots for multianimal DLC

Hi!

I’m trying to plot the standard train/test RMSE vs. iteration plots for a ma_DLC project on a single animal. To do this, I first changed the ‘snapshotindex’ to ‘all’ in the config.yaml file and proceeded to train the network so as to save 10 intermediate iterations:

dlc.train_network(config_path,
shuffle = shuffle_idx,
displayiters = 10000,
saveiters = 10000,
max_snapshots_to_keep = None,
maxiters = 100000,
gputouse = 0)

Next, I evaluated the results:

dlc.evaluate_network(config_path_uniformSampler,
Shuffles = [1,2,3],
plotting = True,
gputouse = 0)

Finally, I cross-validated the inference parameters to obtain the final estimation of RMSE:

dlc.evaluate_multianimal_crossvalidate(config_path_uniformSampler,
Shuffles=[1, 2, 3],
pbounds = {‘pafthreshold’: (0.05, 0.7),
‘detectionthresholdsquare’: (0, 0.9),
‘minimalnumberofconnections’: (0, 21)},
edgewisecondition=True,
leastbpts=0,
init_points=20,
n_iter=50,
target=‘rmse_test’)

However, when I look under /evaluation-results/iteration-#/shuffle#/results.csv, only the last saved training iteration is displayed. Is there a way to obtain the results for all saved iterations (similar to what you get after evaluate_network in a non-ma_DLC project)?

Many thanks for your help and answers,
Nejc

I’m noticing the same thing. Specifically, I set “snapshotindex: all” in the config file. After running evaluate_network(), all of the _full.pickle and _meta.pickle file are saved but after running evaluate_multianimal_crossvalidate() only the last iteration is saved.