Requirements for aligning video frames across multiple cameras for 3D Reconstruction

Hello,

I have had great success in setting up DeepLabCut using an overhead camera for 2D analysis of rats in a research lab setting. I’m considering using multiple cameras for 3D reconstruction, and have a question about how to set up my video acquisition. I’m new to computer vision and have a fairly basic question. Because I’m dealing with moving animals, I’m wondering how precisely synchronized the frame times must be across cameras to properly train the network and/or to reconstruct x,y,z coordinates? I’m recording at about 15-20 fps, but there is always some jitter in the exact frame times and additionally isn’t it likely to have some temporal offset between the cameras? Is this the type of thing that can be aligned and accounted for fairly well post-hoc without too much difficulty, or is it necessary (or strongly advised) for me to create a system that will acquire all images simultaneously across all cameras? Are there some general guidelines regarding what is tolerable?

Thank you for any and all advice!

Brian

Hey Brian,

I would strongly suggest to record frames at the same time (based on a common clock). Post-hoc dealing with misalignment especially with low frame rates (i.e. less than 100Hz) in the case of fast movements is difficult (and easily becomes messy).

Just image your frames is of by 100 ms a rodent can move a lot in that time, the 3D triangulation will be bad; of course you could calculate filtered 2D coordinates and then triangulate from those based on the intermediate time points… However, this can all be avoided by a common frame clock.

Cheers,
Alexander

2 Likes

Hi Alexander,

Thanks so much for your reply - that lays out the problem and solution very clearly. I suspected I might need a solution like that. I’m still searching, although my preliminary searches seem to indicate it is going to be much more expensive to keep videos robustly synchronized throughout ~3 hour long experiments.

By any chance are you (or anyone else reading this!) aware of a reasonably priced system or even DIY method that could handle those requirements?

Best,
Brian

white-matter has a good sync system, especially if you need more than 2 cameras, but the frame rate is not super high: http://www.white-matter.com/ecube-accessories

1 Like

Thanks for the recommendation, I really appreciate your work in making it easy for new users to get started.

I have contacted white-matter as well as another supplier to get quotes and discuss exact specifications. Excited to move forward with this.

Cheers,
Brian

1 Like

A lot of the FLIR cameras support synchronized capture via an external sync cable (you need to build this or have someone build it for you); either a “primary” camera can send TTL pulses to trigger the other cameras, or an external signal generator can be used to trigger all cameras. FLIR’s tech doc is here, but it is basically only explaining the process of setting up their own cameras using their SDK. These cameras will support high frame rate capture (assuming your computer hardware can keep up!) if you need it.

1 Like

@BrianJLane we also just released our software to capture from multiple cameras with timestamps, which might be useful to you. https://github.com/AdaptiveMotorControlLab/Camera_Control

1 Like