Missing spots when comparing CSV and XML files for Trackmate

When you’ve completed a Trackmate analysis, there are basically two ways to export your data on spots. One is clicking the ‘Save’ button at the very end of the process, which creates an xml file with all the Trackmate data. The other method is clicking the ‘Analysis’ button at the bottom-right a few panels earlier, that brings up 3 tables with spot, link and track statistics respectively. The tables can then be saved as csv files.

The problem with the former method is that importing an xml file of any reasonable video into matlab takes absolutely ages (and is even sometimes impossible to work with!), as done with the ‘trackmateSpots’ function included in the MIJ extension for matlab. The xml files are simply too large.

That’s why I decided to use the latter method; importing a csv file is much faster. There’s a problem though! No matter what Trackmate-processed video I test this on, there are always spots missing in the csv file that are present in the xml.

I would like to know why, and how this can be amended (or alternatively if there is a fast way of importing the xml file into matlab). You can reproduce the problem with the files I provide here:

They are the original unprocessed video (Original.tif), the Trackmate-processed video (Processed.tif), the corresponding xml file (Trackmate XML.xml) and spots csv file (Spots in track statistics.csv) (along with the other two csv files), and finally the matlab program I wrote to check for missing spots (difchecker.m).

If you download the folder, then update the directory variable in difchecker.m to point to the downloaded folder before running it, you should get a result like this:

 XML imported. Found 4013 spots in the file.
 CSV imported. Found 3995 spots in the file.
 18 (0.449%) spots missing: 
ID:225 at index 220 
ID:226 at index 221 
ID:227 at index 222 
ID:228 at index 223 
ID:229 at index 224 
ID:494 at index 489 
ID:509 at index 504 
ID:3164 at index 3159 
ID:3772 at index 3767 
ID:3794 at index 3789 
ID:3821 at index 3816 
ID:3832 at index 3827 
ID:3891 at index 3886 
ID:3897 at index 3892 
ID:3913 at index 3908 
ID:3945 at index 3940 
ID:3982 at index 3977 
ID:4003 at index 3998

The program returns the spot IDs of the missing spots.

Indeed, no matter what file I run this on, there are always spots missing. The amount varies depending on the video too; most of the time it’s low, but on one it had around 60% of the spots missing! Either way, this is a large problem for me as I need to be able to import all the spot data into Matlab, and without waiting hours for it to do so. Any help would be greatly appreciated.

Would you perhaps be able to help me, @tinevez?

Yes! But probably not this week :confused: is that ok?

1 Like

Sure, no problem! Thank you for taking the time mate.

Hello.

I checked the data (beautiful movie! what is it?)

But I could not find a problem: There is 3995 spots in the CSV file and they are properly all imported.

What I suspect is that the problem belongs with the CSV file you saved.
It is true that in the original TrackMate XML file there 4013 spots. But when you click on the TrackMate Analysis button, TrackMate generates a table with only the spots that are part of a track. Lonely spots are not displayed in the table.

Could you confirm this?

1 Like

The movie shows epidermal fish keratocytes (whose nuclei are fluorescently labeled blue) navigating an environment with adhesive protein micropatterns (labeled yellow).

I checked, and indeed, the CSV files exclude the spots not in any track! You are absolutely correct! Thank you so much, I thought I might be missing something but did not connect the dots as it were.

:+1:
There is way to export ALL the spots regardless of whether they belong in a track. Just shif + click on the analysis button.

1 Like

Just for completeness: this trick is documented in the mouse-over tooltip of the Analysis button:

1 Like

Thanks to the both of you. Don’t need it anymore, but I wrote a notebook in Python that takes in a Trackmate XML file and converts it to spot, edge, track and calibration data, which is what I was using.

Linking it here in case anyone is interested, it’s free to use and open-source.

1 Like

I think this deserves to go on TrackMate documentation page.
Would you be ok @serendipity?
We still have to wait for the new Fiji website but this will come.

1 Like

Sure man! Do as you like with it.