Training variable number of animals in multi-animal

In designing an analysis strategy for one project, I have a question about how multi-animal deeplabcut is trained. Here’s my situation: my videos have variable numbers of individuals. The vast majority of videos have just one or two animals, but some have up to six animals. I don’t need DLC to identify individuals (I can barely do that myself), so this would be a very generalized model.

As an extreme/unlikely example: If I set up my config file to have six different individuals, can I only train it on videos with one animal? If I can, do I label all points under animal 1 in all frames? Does DLC apply what it knows about animal 1 to animals 2-6?

I guess what I’m asking is if it pools labelling/training info among individuals.

At the other extreme, I could fully label only (mostly) videos with 6 individuals. The problem is that I have so few of those right now that it may just be faster to manually digitize those, and limit DLC use to videos with 1-3 animals.

Does DLC apply what it knows about animal 1 to animals 2-6?

Hi @backyardbiomech yes! You can train it with however many animals (max) are in the training dataset images, then for inference, you can ask for more, or less.

i.e., I can train 2.2 with only 1 mouse and it will find 3 mice during inference. :slight_smile:

Holy crap that will make this sooooo much easier. Thank you, @MWMathis!

1 Like

Of course! I can also point to how best to do this (I should make a proper video tutorial in the future).
i…e, if you train with 1 animal, then your config.yaml would be:

individual:
- individual1

and then if you run analyze_videos with a video that has > 1 mouse, you will see the detections are picked up for all that is visible (ignore the shoddy pose estimation quality - I only trained for 200 iterations on a handful of frames to quickly demo for you :wink: )

ezgif.com-video-to-gif (6)

Then before you convert to tracklets you can add more names in the config.yaml file, and change topktoretrain in inference_cfg.yaml file to be > 1 (i.e. .inf or if maximum, known set it there, i.e. 5, etc).

i.e. here I just added other random names, and then set topktoretain: 3

Hi, I was just running into this issue in our experimental design and was searching to see if the answer was already ‘out there’ before asking. So this is really helpful! I just want to clarify something. From what I read here, I am interpreting that as long as we are using 2.2 (we have 2.2b8 right now) and have multianimal selected, then we can train on just videos of a single animal and then use that trained network for any number of simultaneously recorded animals. Is that correct? Or would it make the network better to take the time to train on videos with several fish (and then subsequently scale up or down). Thanks!

Yes, with caveats.
First, upgrade to the latest version (2.1.10.2 as of now). It includes all of the multi-animal features plus lots of bug fixes and other benefits over 2.2b8 (no longer being developed).

Second, if you only train on 1 animal, and evaluate on those test images, your cross-validation outputs will be way off, and you will end up with good point detections but poor consistency in tracking the different individual animals in those videos. I.e., it will learn how to track one animal, but not how to track multiple when they are close together in one view.

If you will regularly be analyzing videos with multiple fish, train on videos with multiple fish. I’ve found actually much easier (other than the labeling part) to train and evaluate and cross-validate on some videos with many or the maximal number of animals, and then use that model to analyze videos with fewer or single animals, compared to training on videos with a single animal but trying to analyze videos with multiple.

Thank you for the advice!
It sounds like the cost-benefit seems to shake out in favor of spending the time to label more animals upfront.
Also, I did not realize that 2.1.10.2 incorporated ma-tracking (I guess I had been tunnel-visioned on 2.2).