How to move partial labeled frame to a new project & how to quickly evaluate a trained dataset on analyzing new frames without retraining

Hello All,

I have labeled thousands of frames in a project, in which each labeled folder contains 600 frames. Now, I would like to use some of labeled frames (about 300 frames from each folder) in another new project. So, I tried to copy one entire folder to the new project and deleted 300 frames, but I found there were still same 600 frames in new “labeled folder” after executing “check_label” command. My thought was that the save labeling information in h5. file wasn’t changed so that I didn’t completely delete 300 frames. Thus, my question is that how should I move partial frames from one labeled folder to another project?

Also, for a trained dataset, how can I quickly learn its performance (test error) on analyzing new frames that are totally different from the frame used to train (differernt backgroup, coat-color, body size mice). In other words, I may not need to merge new frames to trained dateset and retrain it, then get its performance (test error).

Thanks so much in advance!!!

Hey,

Short answer: anotation file (.h5) and images need to match. Not sure how 300 frames can become 600 though…

Anyway, you can always delete frames from any labeled folder (and move folders across projects). If you delete folders then you can just run deeplabcut.dropannotationfileentriesduetodeletedimages to drop the entries in the h5 file. See help here: https://github.com/AlexEMG/DeepLabCut/blob/199f387b50df1f32ded6ac5d54b68f08f009661b/deeplabcut/generate_training_dataset/trainingsetmanipulation.py#L153

Btw. the inverse also exist: you can drop images that were not annotated with:

dropimagesduetolackofannotation

See: https://github.com/AlexEMG/DeepLabCut/blob/199f387b50df1f32ded6ac5d54b68f08f009661b/deeplabcut/generate_training_dataset/trainingsetmanipulation.py#L184

When you copy/paste indidual annotation folders, the following functions are also very handy: deeplabcut.comparevideolistsanddatafolders allows you to check if all folders have a corresponding key in the config.yaml file. If not you can use deeplabcut.adddatasetstovideolistandviceversa to fix that.

With regard to: “Also, for a trained dataset, how can I quickly learn its performance (test error) on analyzing new frames that are totally different from the frame used to train (differernt backgroup, coat-color, body size mice). In other words, I may not need to merge new frames to trained dateset and retrain it, then get its performance (test error).”

You can of course evaluate DLC on new data. But there is currently no function to evaluate a model on another set of annotated frames. Could be a valuable addition, if someone is interested.

Cheers,
Alexander

P.S. More info on helper functions here: https://github.com/AlexEMG/DeepLabCut/blob/199f387b50df1f32ded6ac5d54b68f08f009661b/docs/HelperFunctions.md

Hello Alex,

Greatly appreciate for your detailed responses and they are really helping!! I thought the reason why 300 frames became 600 was that I haven’t modified the h5. and just deleted frames from annotation folders directly. Thus, the annotation information are sill there.

Yes, it would greatly improve testing efficiency of DLC if we can learn its performance on another anontated frames without mutiple hours’ traning.

Many thanks!!

Jinxin

you can also just run deeplabcut.analyze_videos on the new data and look at the performance, with plot_trajectories, etc.

I just realized the function https://github.com/AlexEMG/DeepLabCut/blob/master/deeplabcut/pose_estimation_tensorflow/predict_videos.py#L610 can of course be used to run a DLC model on a new folder of labeled data…

Great news!! Thank you so much!

Hi, the link to code references a function GetPoseDynamic. Can you specify which function can be used to run a DLC model on a new folder of labelled data?

if you have more labeled-data, you need to just create a new training set and train again.

I’m trying to evaluate the already trained model on new labelled data (test data)

there is no function for that, but you can look at the evaluation code and edit as you like: https://github.com/DeepLabCut/DeepLabCut/blob/b7b2da93eed3a0c9b892f64b890174a4f9967008/deeplabcut/pose_estimation_tensorflow/evaluate.py