Support for plugins for iterative animal behavior labeling and ML

Hi,

I’m a data scientist at St Jude, working on classifying animal behavior (mice). I’ve been using deeplabcut to track the animal’s pose, before playing with some ML methods to classify behaviors from annotated videos. We’ve been considering using or building behavior annotation tools in napari for this process.

Building an ML classifier is obviously an iterative process – label, train, inspect results, relabel if needed, etc. We’ve been thinking about how to make this as streamlined as possible. This means making the train-inspect-relabel loop as tight as possible. To be a bit more explicit, the ideal process would be something like: the labeling would be performed, the model trained, inference run, the results overlaid in the video in napari, and perhaps frames in the video are suggested to relabel, then retrain, etc. The suggested frames to relabel is a form of ‘active learning’, and can be a vastly more efficient way to train an ML model. But to begin with, just something that can handle both the video labeling and ML training/inference would be a useful start.

I wonder what support napari has or will have for adding such analysis plugins? I do see in the roadmap that support for functional plugins in on the cards. But what form does this take exactly? What would be needed to be developed on our part?

I also wonder if you know of any specific behavior classification plugins that are currently in development in napari along the lines of what I’ve described? I haven’t seen anything listed here: Issues · napari/napari · GitHub. I do know that deeplabcut, for instance, is planning on using napari for its labeling GUI in a future release. Behavior classification can use the DLC tracked poses as input to its algorithm, so having the pose labeling, and then the behavior labeling and classification all take place within napari would be very nice.

Thanks!

Regards,
Ben Lansdell

2 Likes

Hi Ben

We’re definitely excited to team up with the DLC folks to get their annotation working inside napari, and @kevinyamauchi has worked on some cool proofs of concepts, checkout this great tutorial on annotating points with napari, and maybe he can give you an update on the latest there?

Something like the active learning framework you just described are really exciting, and one could imagine building something like that using ideas from this interactive affine registration plugin affinder that @jni is working on (still under active in development).

Now is a great time though to be thinking about building a behavior classification plugin as we’ve recently expanded our plugin hook specifications with GUI hooks and analysis hooks that allow you to connect your custom code. You can read more about getting going making a plugin here.

I hope these links help and let us know how it goes, I know a lot of people would be excited about this functionality

Best
Nick

2 Likes

Hi Nick,

Thanks a lot for the info and links. I’ll take a look at the affinder plugin, and the details on hooks.

Cheers,
Ben

Hello @benlansdell ! Sounds like you’re working on some cool stuff.

As @sofroniewn mentioned, I had been working on a couple annotation tools for behavior movies. I have recently changed jobs, so I am not currently working on them, but I am more than happy to share with you what I’ve done and either pass them off to you or help you (or anybody else interested) get going!

I just realized that I have some changes that I haven’t yet committed, so I will do so and reply back with a couple of links and comments.

Best regards,
Kevin

3 Likes

Hi Nick,
I have a related question. I saw in a post that you were looking for an efficient way to get movies (avi,mp4, …) into napari. Have you found a more efficient way?
Thanks,
Abbas

Unfortunately, I havn’t made any real progress, but maybe some others have

Hi Nick and Abbas,

we have a plugin for working with videos in napari: napari_video · PyPI.

It uses opencv to read the video - it’s not thoroughly tested but it works well for us on osx and windows.

Best,
Jan

1 Like

Very exciting! I was trying to visit the GitHub homepage from PyPI, but got a 404. Can you update the link. Thanks!!

Sorry about that - link should work now.

1 Like