Tracking in 3d with stereo camera setup

Greatings everyone,

first let me briefly explain the experimental setup: we have a transparent cube illuminated by led panels from 2 sides and 2 cameras are filming flying insects at a 90 degree to each other. The aim is to reconstruct the 3d flight trajectories from the video data. The amount of insects should not matter but for the initial phases anywhere from 1 to 1-2 dozen is expected. My search so far did not find anything similar described on the forum. Is there a way for Trackmate to compute what we need? My very basic approach was to track on each camera separately and then after camera calibration to triangulate the points in 3d. Besides doing this manually(too much work for big amount of insects), the stereo correspondence is not trivial to compute. For example one Track in the first camera is broken into many segments and I don’t know how to ‘assign’ them to the single big corresponding segment on the other camera(ideal results are as many tracks as insects with no fragmentation). Some additional information: Trackmate runs within Matlab. I’m open to ideas and thanks in advance.

Quick addition: I have read that the Kalman filter can be modified to work for 3d. Where exactly in the tracker code is the Kalman code declared, so I can experiment with it a little?


Hi @Voydwalker

The setup you describe matches those in the field of computer vision if I am correct. Two cameras at 90º generates 2D frames, that are used to derive the 3D location of spots.

TrackMate does not do that out of the box. It was built for microscopy setups, where 3D images (‘stacks’) are acquired by imaging each 2D plane, and moving the focus plane through the sample. So the particle detectors built in there directly inspect the 3D image to find the 3D positions of the particles.

In your case you have to combine two sets of 2D positions to generate one set of 3D positions, then track them. TrackMate does not do that, it can only track your particles once you have the 3D positions.

I do not know where to find code that achieve such a challenge. I would start with APIs that specialise in Computer Vision (as opposed to Image Analysis for Life Sciences) such as openCV.

As for the Kalman tracker code in TrackMate, you can find it here:

1 Like

just to add some other documents and links to the suggestions of @tinevez, Matlab has some examples and some codes that I think could be a good starting point for your task.

something like that.

Hoping to be useful,
have a nice day
Emanuele Martini