Reconstruct 3D from 2D images

Hi, There.

I have used deeplabcut to capture motion from 2d images(frame).
But, although the deeplabcut software captures from 2d frame images, how can I build the 3d structure from the 2d captured images?

Is it possible through the software? If it were not, could you let me know what the program can do that and If it can do that, could you let me know how to do that?

Thanks!

2 Likes

Hi @Walter_sh12,

I might misunderstand your question/intent and I am not familiar with DeepLabCut but generally spoken you cannot reconstruct volumetric information (3D) from a simple 2D image if there is no other data existing which codes the information for the third dimension. This will be true for other software as well.
If you have a simple 2D image plane showing a still image (from a video) a normal camera will not give you this additional information you would need to do such reconstruction.
An example where a reconstruction would be possible is, if e.g. the pixel intensity would stand as representative information for the height of a certain position of a terrain in units. Then a reconstruction of a landscape from such an image would be an option.

1 Like

Are you only recording with one camera? As @biovoxxel said, you’ll need more info in order to reconstruct 3D.
A “simple” way for 3D reconstruction is using multiple cameras (number depends a bit on your setup and what you are filming). There are toolboxes available in order to reconstruct 3D from multiple cameras, our lab is currently looking into argus, but the main idea is that the cameras need to have a fixed position and be calibrated in relation to each other, which then allows to reconstruct 3D.

There are also some pointers on the basic idea of 3D reconstruction in this preprint:
Using DeepLabCut for 3D markerless pose estimation across species and behaviors

2 Likes

Thanks for good advice!
We are looking for the software that reconstruct the 3d structure and will do that through your mention.
But we don’t know that deeplabcut can do that. Could you let me know what the toolboxes that you mentioned name it is?

Thanks!

you can train an individual network with DeepLabCut on multiple camera angles, but you need to know the camera placement for proper triangulation (see preprint mentioned above), i.e in deeplabcut you have openCV already installed, and can use their camera calibration tools: https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html then yan can project the two (+) camera views into 3D space

here is a good notebook that can help guide you: https://github.com/udacity/CarND-Camera-Calibration

2 Likes

Hey Mathis, there is no mentioning of “triangulation” in the preprint mentioned above. I am having problems finding the relevant function in OpenCV.

This package (https://github.com/lambdaloop/anipose) that seems it is still being developed, has some easy to use functions to calibrate cameras and triangulate. I hope to try it this week after I collect some calibrations videos. It has a nice functions to run videos in batches too (2d).

DLC also runs videos in batches, the function analyze_videos allows for an input folder so any videos that get placed there (or in sub folders thereof), can be analyzed automatically.

Here is the script to batch process: https://github.com/AlexEMG/DLCutils/blob/master/scale_analysis_oversubfolders.py

here is a simple script to run over many projects: https://github.com/AlexEMG/DLCutils/blob/master/scale_raining_and_evaluation.py

Hello, I’m the developer of anipose. It should be ready to use for 3D tracking with DeepLabCut! I’ve setup the documentation last week, but there may still be some rough edges on first setup.

If you end up using it, please let me know if you have any issues, either through email or by filing a github issue. I will do my best to help out, and update anipose as needed to make it work.

1 Like

Thanks for your good advice!
By the way, when I reconstruct the 3D image from the videos, I can’t understand the process only with the git memo.
So, could you let me know if therer are any useoverguide about anipose?(such as how to make folders / config.toml file etc)
Thanks!

Have you tried to follow the 2D setup guide for anipose?

It goes into depth on some example folder structures and how to setup the configuration file.

Let me know if you try to follow it and there is something unclear, and I can help guide you further.


I typed the command as you mentioned on the guide, but it was wrong and I don’t know how to fix it.
How can I correct this error?
Thanks!

Ah sorry, for running anipose, you need to run it from the command line directly, rather than from an ipython session. In addition, you need to setup the configuration file and run anipose from the same folder as the project.

What about inputs?
I commanded " anipose analyze the path of camera.avi " . Is that correct?
If not, How do I command the input?

Hi @Walter_sh12,

I apologize for the delayed reply.

Anipose is only setup to run in batch mode, over a whole group of videos rather than a single one. If you’d like to just analyze one video then it’s easier to use DeepLabCut directly.

For analyzing a group of videos, once you’ve placed them in some folder structure (presumably they are organized in some way when recording), you may run any of the anipose commands.

For instance, to run DeepLabCut on a whole set of 2D videos, you may do:
anipose --config path/to/config.toml analyze
…where you would replace the path/to/config.toml part with the actual path to your config.toml file.

If you’re in the same directory as the config.toml file, you may run anipose analyze directly and it will detect the config file automatically.

Just to be clear, DLC also runs over as many videos as you want (and always has ;), so it’s very easy to batch process this…ie we run hours of video per day in a fully automated fashion. As soon as a new video hits the folder, you simply need to have a cronjob run, and/or a script like this:

deeplabcut.analyze_videos allows for folder input, and you can set a destination output of analyzed files. Here is a graphical overview from our user guide:

Yes, I don’t mean to imply that DeepLabCut cannot process multiple videos. If you just need to analyze a set of videos in a folder then DeepLabCut is clearly sufficient (and perhaps better suited than anipose for the task).

The advantage of anipose in this regard is that it naturally handles nested folder structures, organizes different data into different folders, and can aggregate all the data into a single file with the names of the folders in the file. Some examples of nested structures it can handle are described here: https://github.com/lambdaloop/anipose/blob/master/docs/start_2d.md

This can all be done manually for a specific pipeline, but we have found for own uses that coding it in a more general way with a configuration file has been quite helpful.

1 Like

Thanks for your advice!
And I typed the commend as you mentioned before, it works!
Well, When I compute angles, it prints well done but on the path there are no file or directory.
How can I correct this probelm?
I think that the model_folder in config.toml file is something wrong, How do I set this path?
Is it ok that the path is trained network about the video on the past?
Thanks!

Workflow for 3D DLC analysis can be found at: https://optogeneticsandneuralengineeringcore.gitlab.io/ONECoreSite/projects/

Please email us with questions/problems/suggestions and acknowledge the ONE Core (email and suggested wording in the link).

Hi everyone!

A very nice and simple solution for reconstruction from 2D to 3D coordinates is the program ‘argus 3D for the people’. http://argus.web.unc.edu/

It is also python based and even has a GUI, if this is preferred :wink:

However I have another question regarding the 3D analysis.

I have successfully gained a track of an animal in 3D/4D space. How is it possible for me to smooth this track/signal to reduce the variance of the position of the tracking coordinates?

I found only 2D smoothing of graphs etc. so far, but I would assume that 3D smoothing or more likely 4D (including time) is a very common topic in animal behavior tracking. Is there possibly a python package or similar to do this?

Thank you very much :slight_smile: