[Multi-animal] have markers, NO skeleton/box

Hi guys,

I am trying to draw skeleton for multi-animal project with deeplabcut v2.2b8 under Ubuntu 18.04.

I am kind of experienced with Deeplabcut, and everything goes perfect here:

deeplabcut.create_video_with_all_detections(.....)

I successfully generate video with all body-parts markers.

However, when I try to draw the skeleton as follows:

deeplabcut.convert_detections2tracklets(.....)
deeplabcut.convert_raw_tracks_to_h5(.....)
deeplabcut.filterpredictionsis(.....)
deeplabcut.create_labeled_video(.....)

I get the video totally same with the original video, but no skeleton, even no body-parts markers.

I checked the generated csv file after filterpredictionsis(), the pixel xy results are empty for each picture frame!

deeplabcut.refine_tracklets(...) also shows only same with original video, without any skeleton or marker to refine.

In summary, create_video_with_all_detections() works perfect, but seems that convert_detections2tracklets() fails, even if the printed output seems all right:

deeplabcut.convert_detections2tracklets(path_config_file, videofile_path, videotype=‘mp4’, track_method=tracktype, overwrite=True)
Using snapshot-8800 for model /home/…/dlc-models/iteration-0/maDLC-2020-11-12Nov13-trainset95shuffle1
Processing… /home/…/videos/video.mp4
/home/…/videos already exists!
Analyzing /home/…/videos/VID_xxxxshuffle1_8800.h5
681it [00:01, 638.33it/s]
The tracklets were created. Now you can ‘refine_tracklets’.

I found several unsolved issues with the similar problems as Missing tracking points, No labels in final step, No marker appear on Refine Tracklet.
If you need any further information to solve this problem, please don’t hesitate to let me know.

Thanks in advance.

Hi @stefen, the steps above are crucial and should be taken slowly. Did you run cross-validation after evaluation? This is important prior to converting detections to tracklets; you could try utils.make_labeled_video.create_video_from_pickled_tracks() after convert_detections2tracklets() to make sure animals are nicely assembled. Only then should you move to creating the final h5 file, whose default values of min_tracklet_len and max_gap may need tuning to your particular setup.

Thank you very much for the quick response! @jeylau

you could try utils.make_labeled_video.create_video_from_pickled_tracks() after convert_detections2tracklets()

utils.make_labeled_video.create_video_from_pickled_tracks() will generate a new folder with pictures of each frame and a video, but all of them are same with original video, without any body-parts marker or skeleton.

Then I did the pipeline again trying to draw the skeleton, but still failed as before.

Alright, then we now know that the assembly of the detections into animals is not functioning. Please use evaluate_multianimal_crossvalidate() before convert_detections2tracklets(). This will attempt to select optimal parameters for the stages coming afterwards. I think that post may clarify what those parameters do. If that still does not solve it, please get back to me :slight_smile:

Thank you @jeylau! Very clear declaration, it really helped my problem.

btw, just for making sure, could we randomly label an animal with different individual id in madlc?
e.g. I have many monkeys, one is labelled as individual 5 in last labelled frame, but new frame I label it as individual 7.
I guess it’s not a good idea?

Thank you!

If you can tell them apart, you should label consistently. If you cannot, it does not matter what ID you give them.

Could this have an impact for the tracking?
I have several animals in my training data and i selected frames at random.
Since the animals cannot really be distinguished i labelled them consistently per ID, but ignored the ID consistency.
The detection works very well, however when trying to track them, the skeleton maps body parts across animals. Thanks in advance!

I had exactly same experience as @pbugneac .
Really want to figure out whether because of label ID, or just bad parameters in inference_cfg.yaml.

Thank you very much in advance! @MWMathis. @jeylau

1 Like

@stefen, what helped me to some extent was setting edgewisecondition to False. The default is True and they highly recommend this. However, it seems the distances this calculates between body parts are too high and so it connects body parts across IDs.
Without it, the model is much more accurate and we see almost no ID swaps. Also the rmse for test is much lower. Give it a try!
@MWMathis could this explain the major ID swaps?

@pbugneac thank you very much for this kind experience, that setting looks reasonable! (sorry for my late response)
Maybe because I only labelled 10 frames, it only has very little improvement.
May I ask how much frames you labelled in your case?

@stefen good to hear!
I labelled around 150 frames and planning to increase that to 300.
10 frames is definitely too little, but then again it depends on how complex are your images.
I managed to get some promising results but not yet entirely accurate, so trying to improve overall accuracy at the moment, by adding more frames and playing around with some of the settings.

1 Like