Problem with network refinement

I’m following the network refinement workflow. I have manually selected outlier frames and adjusted tracker errors using the GUI, and I seem to have successfully merged these datasets with my original frame selection. According to the workflow (described here: https://github.com/DeepLabCut/DeepLabCut/blob/4e7306baefe7a4f1b107251dae07e138ce02215c/docs/functionDetails.md#g-train-the-network), the new training set should be stored in a new iteration folder. However, my new training set, as well as new .csv, .h5, .mat, and .pickle files, are stored in the iteration-0 folders of their respective subdirectories (see screenshot; I specified this to be shuffle2 in the GUI “create training dataset” panel).

Why this is relevant: when I started retraining (with shuffle=2 selected in the train function), all new snapshots are being stored in the shuffle1 folder. I am concerned that the newly augmented training set (shuffle2) is not being used in retraining.

Am I correct in assuming that shuffle2 is not being used for retraining? If so, what is the root of this problem? Does the confusion stem from failure to create a new iteration folder? Or have I just goofed on making another adjustment that would tell DLC to use shuffle2 instead of shuffle1?

One more note: I just realized that I forgot to adjust the initial weight in the train pose_cfg.yaml file to the last snapshot of my first training session using shuffle1, but I don’t see how this could cause the issue described above.

I and the Bettas thank you v much for any help you can give!

AmyScreen Shot 2020-11-13 at 12.20.44 PM

I had something similar happen once (where a merge did not advance the iteration). It hasn’t happened again.

Here’s what I would suggest:

  1. Open the label frames GUI, and load some of the new frames you extracted and refined. If the data look good, then the data merged into CollectedData successfully, and only the iteration didn’t advance.
  2. You can manually advance the iteration by changing it in config.yaml. Change that to 2.
  3. Create your training set, train, evaluate. It should all be in iteration-2 folders.
1 Like

The iteration will not advance, if there is some problem (which will be stated in the terminal). This is indeed really rare (and often some path/incomplete data annotation issue, as this is what that step checks!)

Thanks everyone. The problem was triggered by the presence of a .DS_Store file in my labeled-data directory. Got rid of these wherever present in the model and now everything seems to be working fine. The iteration updated on its own and I’m retraining now. Cheers!