Ma-dlc for single animal project

Usage & Issues
,
#1

Hello,

My goal is to track a single fish, I want to track the edges of the fish’s head and also to track points on its tail. I am using 2.2b8 version. I would very appreciate if someone could help me with the following questions.

  1. does the “skeleton” feature can improve the performances of single animal tracking?

  2. so far I opened a project with “multianimalproject: true” in the config.yml. while doing so, I am struggling to understand if I should set all of the body parts as “uniquebodyparts”? or as “multianimalbodyparts”? (under only one individual)

  3. also, sometimes the tip of the tail cannot be seen when the fish is moving. When that happens, should I skip this label? or its better to mark the end of the visible tail as a consistent label?

  4. I am using a low dim videos (high saturated grayscale images) - when the fish is on the side I am seeing a different figure (hard to asses depth). because of that, is it ok to also use labels such as “center of the head” which will be marked in different locations due to the posture of the fish? or labels that are suppose to be the edges of the fish without considering the posture of the fish?

  5. if the answer to 4 is “yes”, should I add those labels also to the skeleton?

Thanks!

#2

Hi, Tom:

  1. Yes, I have found improved single animal tracking with maDLC when I have lots of occlusions, low video quality, or complex backgrounds. However, the training and analysis is much slower, and there are more steps to building the model. I suggest trying single animal first, and if you find that it’s getting a lot of false positives (e.g. tracking points off the animal), that’s a good sign that the skeleton features might be helpful.
  1. No, If you want the skeleton tracking, you have to put the animal body parts with multianimalbodyparts. The unique parts are mostly for other things in the image (e.g. the corners of the arena, or objects). You just want to make sure that you have only one individual listed.
  1. Don’t mark it if you can’t see it. At least at first. Deeplabcut can guess if you train it to use your guesses, but don’t do that on your first model build. You can interpolate through the skipped frames later. Or, of course, you could select body parts that are never occluded/blurred and extrapolate out to the edge from there.
  1. It’s possible to mark the volumetric center of an animal. I use deeplabcut for 3D reconstruction with wild birds, so every frame sometimes has a different angle, and from different cameras. I have a mark for “head”, which I envision as a sphere, and I try to mark the center of the sphere. Same for “rump”. It works surprisingly well once you have labeled a large enough variety of images. But you have to be very consistent at visualizing your marks during labeling. And yes, you should add this to the skeleton if you want to use the skeleton for training. Eventually the model will generalize that body part and essentially learn to mark it from many angles, just like how it can learn to mark dog shoulders on many breeds of dogs.
1 Like
#3

Thanks!!

You mean setting “multianimalproject: false” right from the beginning? or just working with multianimal project with just one individual? Is there a difference?

Also, what do you mean by :

What are those steps? The pipeline using the GUI should be different?

#4

Creating a multi animal project (multianimalproject: true) does more than just change that text and track different numbers of individuals. It changes the structures of a lot of files, and changes how tracks are built, no matter if you are tracking one or one hundred animals. And you can’t swap between multi- and single- structures mid project. So when I said single animal project, yes, I meant to set that to false. You do not get the skeleton information in the tracking with that, but it’s faster and easier to use.

The additional steps occur during refining the model. Multi-animal projects require several extra steps because now the model is not just looking at the best “head” and the best “tail” in each frame, but is looking at all of the areas it thinks might be “heads” and “tails”, and tries to keep track of all of them. Then there are steps of trying to assemble the parts into animals (that’s where the skeleton comes in) within frames, and then keep track of animal identities across frames. That’s why some of the files have different structures, which means that you can’t simply extract outliers from your video analysis output.

The multi-animal functions have also not yet been fully described/published, so a lot of users, myself included, have run into difficulties trying to figure out what settings to manipulate and use. I’ve figured out the specifics for my purposes (with a lot of help from the developers and from other users), but it took a lot of time, and I have a background in 3D animal tracking software.

It’s doable, but definitely only try multi-animal if you 1. need it because you have multiple animals, or 2. have tried to build a single animal DLC project but can’t get good results because of the video or background.

2 Likes
#5

Also, upgrade to the latest version of deeplabcut (2.1.10.2). All of the multi-animal parts are now included there, with lots of bug fixes and other useful things.

1 Like
#6

Hi Brandon,

Thanks for your answer. We have a similar case and we are considering to start a multi-animal project with a single animal.
My question is, if we try single animal first, and then realize that the skeleton features might be helpful, do we need to label all frames again after we create a new a multi-animal project? Or can we use the labeled frames for the new ma-Project?

#7

Hi, Gökçe Ergün
You do not need to re-label. In fact, there is even a page of instructions for converting from single to multi-animal so you can see the process.

I really can’t reiterate this enough, though. Do not start a multi-animal project when you don’t really need it until you are comfortable using single-animal DLC and can confirm that you can’t get good tracking with single-animal DLC. Go through several refine iterations with a single-animal model, read through all of the papers and docs, and get a good understanding of how DLC works before jumping to multi-animal. There are many more parameters to keep track of with maDLC, and getting good tracking with maDLC can be easy, but getting publishable tracking can be more difficult.

1 Like