Odd Error Parsing Trackmate File in Matlab

An odd error because this is a pipeline that has worked fine in the past on another machine (with different versions of everything likely) I have a script generated trackmate xml file (which I just tried to attach, I can’t tell if I succeeded). An attempt to load it in matlab with trackmateGraph gives the error:
Error using containers.Map/subsref
The specified key is not present in this container.

Error in trackmateEdges (line 165)
vDescriptions{ l } = ef( vn ).name;

Error in trackmateGraph (line 93)
trackMap = trackmateEdges(filePath, edgeFeatureList);

I cant quite follow what its doing but it seems to be looking up the titles of the edge table fields {‘SPOT_SOURCE_ID’} {‘SPOT_TARGET_ID’} {‘DISPLACEMENT’} {‘VELOCITY’} in a map that only seems to contain two keys {‘DISPLACEMENT’} {‘VELOCITY’} and obviously failing.

I have a vague recollection that I may have gotten this to work by commenting out this code in the past.

If I do this now however I now get a (maybe unrelated) error in the matlab graph code

Building graph. Error using matlab.internal.graph.constructFromTable (line 16)
First variable in edge information table must be a 2-column array (numeric, cell, or string)
of node names.

Error in digraph (line 278)
matlab.internal.graph.constructFromTable(…

Error in trackmateGraph (line 120)
G = digraph( nt, spotTable );

EndNodes is indeed a 3kx9k connectivity matrix (?) not a somethingx2 matrix as it seems to expect. Matlab 2019b and the Trackmate and loading code in an up to date Fiji install.

Any thoughts on where to look to start debugging this would be appreciated since this is all pretty deep in the weeds of the loading innards. Thanks!

38043760_movie1 1-15 sstabilizedimage_bksubmin_subtraction.tifexportModel.xml (1.9 MB)

1 Like

Hello Anthony.

I can reproduce the issue and I think I have found the reason why.
For some reasons your xml file is missing some edge feature declaration.
Right now you have:

      <EdgeFeatures>
        <Feature feature="VELOCITY" name="Velocity" shortname="V" dimension="VELOCITY" isint="false" />
        <Feature feature="DISPLACEMENT" name="Displacement" shortname="D" dimension="LENGTH" isint="false" />
      </EdgeFeatures>

(lines 15 to 18 in the xml file).

But TrackMate is also looking for the declaration of SPOT_SOURCE_ID and SPOT_TARGET_ID.

If you paste these 2 lines:

        <Feature feature="SPOT_SOURCE_ID" name="Source spot ID" shortname="Source ID" dimension="NONE" isint="true" />
        <Feature feature="SPOT_TARGET_ID" name="Target spot ID" shortname="Target ID" dimension="NONE" isint="true" />

after the first two above, then the file opens normally in MATLAB.

1 Like

Interesting, appreciate you looking at it, any thought if there are saving gotchas that could be causing this. This is a pipeline that’s been fallow for a while and is about to be revived for hundreds of data sets. The script generating the xml files (below) is basically the example scripting trackmate script with a few features added but I don’t really understand the intricacies of how to interpret the code that is saving the xml file so there might be some newbie mistake there (though they open fine in trackmate itself obviously I just double checked).
Thanks! -A

from fiji.plugin.trackmate import Model
from fiji.plugin.trackmate import Settings
from fiji.plugin.trackmate import TrackMate
from fiji.plugin.trackmate import SelectionModel
from fiji.plugin.trackmate import Logger
from fiji.plugin.trackmate.detection import DogDetectorFactory
from fiji.plugin.trackmate.tracking.sparselap import SparseLAPTrackerFactory
from fiji.plugin.trackmate.tracking import LAPUtils
from fiji.plugin.trackmate.action import ExportTracksToXML
from os import listdir
from ij import IJ
import java.io.File as File
import fiji.plugin.trackmate.visualization.hyperstack.HyperStackDisplayer as HyperStackDisplayer
import fiji.plugin.trackmate.features.FeatureFilter as FeatureFilter
import sys
import fiji.plugin.trackmate.features.track.TrackDurationAnalyzer as TrackDurationAnalyzer
import fiji.plugin.trackmate.features.track.TrackSpeedStatisticsAnalyzer as TrackSpeedStatisticsAnalyzer
import fiji.plugin.trackmate.features.track.TrackSpotQualityFeatureAnalyzer as TrackSpotQualityFeatureAnalyzer
import fiji.plugin.trackmate.features.edges.EdgeVelocityAnalyzer as EdgeVelocityAnalyzer

import fiji.plugin.trackmate.io.TmXmlWriter as TmXmlWriter

outputFolder= “D:\core\aditistuff\correctwindowed30_15_withnorm\bksub\”
filelist=listdir(outputFolder)

for file in filelist:
curfile=File(outputFolder,file)
print "processing "+file
imp = IJ.openImage(curfile.getAbsolutePath())

#----------------------------
# Create the model object now
	#----------------------------
	   
	# Some of the parameters we configure below need to have
	# a reference to the model at creation. So we create an
	# empty model now.
model = Model()
	   
	# Send all messages to ImageJ log window.
model.setLogger(Logger.IJ_LOGGER)
	
	#------------------------
	# Prepare settings object
	#------------------------
	      
settings = Settings()
settings.setFrom(imp)
	      
	# Configure detector - We use the Strings for the keys
settings.detectorFactory = DogDetectorFactory()
settings.detectorSettings = { 
	    'DO_SUBPIXEL_LOCALIZATION' : True,
	    'RADIUS' : 7.5,
	    'TARGET_CHANNEL' : 1,
	    'THRESHOLD' : 3.0,
	    'DO_MEDIAN_FILTERING' : False,
}  
	   #.15 was ok on div but was losing some and way overseg for others
	    #was 2.25 for other bksub
	# Configure tracker - We want to allow merges and fusions
settings.trackerFactory = SparseLAPTrackerFactory()
settings.trackerSettings = LAPUtils.getDefaultLAPSettingsMap() # almost good enough
settings.trackerSettings['ALLOW_TRACK_SPLITTING'] = False
settings.trackerSettings['ALLOW_TRACK_MERGING'] = False
settings.trackerSettings['GAP_CLOSING_MAX_DISTANCE']= 0.0
settings.trackerSettings['LINKING_MAX_DISTANCE']= 20.0
	  
	# Configure track analyzers - Later on we want to filter out tracks 
	# based on their displacement, so we need to state that we want 
	# track displacement to be calculated. By default, out of the GUI, 
	# not features are calculated. 
	   
	# The displacement feature is provided by the TrackDurationAnalyzer.
settings.addTrackAnalyzer(TrackSpeedStatisticsAnalyzer())
settings.addTrackAnalyzer(TrackDurationAnalyzer())
settings.addTrackAnalyzer(TrackSpotQualityFeatureAnalyzer())
settings.addEdgeAnalyzer(EdgeVelocityAnalyzer())	
	# Configure track filters - We want to get rid of the two immobile spots at 
	# the bottom right of the image. Track displacement must be above 10 pixels.
	
	## Commented for debug
#filter2 = FeatureFilter('TRACK_DISPLACEMENT', 10, True)
#lowest acceptable threshold for now rest done in post production
filter2 = FeatureFilter('TRACK_DURATION', 2, True)

settings.addTrackFilter(filter2)
	   
	#-------------------
	# Instantiate plugin
	#-------------------
	   
trackmate = TrackMate(model, settings)
	      
	#--------
	# Process
	#--------
	   
ok = trackmate.checkInput()
if not ok:
	   sys.exit(str(trackmate.getErrorMessage()))
	   
ok = trackmate.process()
if not ok:
    sys.exit(str(trackmate.getErrorMessage()))
	   
	      
	#----------------
	# Display results
	#----------------
	    
	#selectionModel = SelectionModel(model)
	#displayer =  HyperStackDisplayer(model, selectionModel, imp)
	#displayer.render()
	#displayer.refresh()

#outFile = File(outputFolder, curfile.getName()+"exportTracks.xml")
#ExportTracksToXML.export(model, settings, outFile)
	
outFile = File(outputFolder, curfile.getName()+"exportModel.xml")
writer = TmXmlWriter(outFile)
writer.appendModel(model)
writer.appendSettings(settings)
writer.writeToFile()
print "All Done!"

	#----------------
	# Display results
	#----------------
	     
selectionModel = SelectionModel(model)
displayer =  HyperStackDisplayer(model, selectionModel, imp)
displayer.render()
displayer.refresh()
	
# Echo results with the logger we set at start:
model.getLogger().log(str(model))

Yep, I forgot to declare the features that matter in this script >.<

The problem is that the GUI does plenty of things for you, that you have to do manually when you make a script. And some of these features are critical when it comes to stuff like loading saving.

I take this as an indication that I should add facilities in the code that will make scripting easier and safer.
Thanks Anthony

1 Like

There are always wheels inside the wheels, :slight_smile: but what declaration in the demo script would get these tags into the xml files? Its no problem for me to nuke all the existing xml files and start over, I’d rather do that (and know the logic of what’s missing in the script for future reference). Thanks!

Alternatively, do these lines just populate that map? If its a bug rather than something I can easily fix in my trackmate script I’d rather hack the matlab loader for now, rather than add an extra wheel of a script to fiddle with the hundreds of xml files.

I just tried to add the EdgeTargetAnalyzer which seems to put the missing tag in the xml file and avoids the map related error (in unmodified loader) but i’m still getting the second error (error below and new xml file attached). Could this be an unrelated MATLAB version behavior change? I’d put money that in the past I’d successfully loaded similar xml files by just commenting out that map unit setting code this was definitely an earlier MATLAB version, I’m on 2019b now.

Thanks,
Building graph. Error using matlab.internal.graph.constructFromTable (line
38043760_movie1 1-15 sstabilizedimage_bksubmin_subtraction.tifexportModel.xml (2.0 MB) 16)
First variable in edge information table must be a 2-column
array (numeric, cell, or string) of node names.

Error in digraph (line 278)
matlab.internal.graph.constructFromTable(…

Error in trackmateGraph (line 120)
G = digraph( nt, spotTable );

Ok I will look into it.
ButI remember adding the missing 2 lines in the TrackMate xml fixed the issue.

Now as you said we have a problem with the 100s of xml file you already generated.We do not want to re-run the script for such a large dataset.
As a remediation I would make a grep based script to fix them on the fly. I am uncomfortable with changing the MATLAB scripts. They assumed the XML file was correct, and by putting workarounds for missing declarations, we open the way to downstream problems and harder to fix.

Anthony, this works for me with MATLAB 2019b:


>> G = trackmateGraph('38043760_movie1 1-15 sstabilizedimage_bksubmin_subtraction.tifexportModel.xml')
Importing spot table. Done in 9.1 s.
Importing edge table. Done in 28.3 s.
Building graph. Done in 0.3 s.

G = 

  digraph with properties:

    Edges: [3012×6 table]
    Nodes: [4964×10 table]

How odd, started from scratch replaced all the fiji/scripts/ files with fresh downloads just in case, and I’m still getting the error from the matlab graph code on the exact same fixed file… If you can’t replicate it must be some very specific weirdness, I’ll dig into it a bit more and maybe remote desktop into the lab and see if I can replicate on a different PC… The hundreds of files don’t exist yet, I’m trying to make sure I only have to run them once :wink: Thanks for poking at this, will keep you posted if I figure it out…

G=trackmateGraph(‘38043760_movie1 1-15 sstabilizedimage_bksubmin_subtraction.tifexportModel.xml’);
Importing spot table. Done in 13.2 s.
Importing edge table. Done in 48.9 s.
Building graph. Error using matlab.internal.graph.constructFromTable (line 16)
First variable in edge information table must be a 2-column array (numeric, cell, or string) of node
names.

Error in digraph (line 278)
matlab.internal.graph.constructFromTable(…

Error in trackmateGraph (line 120)
G = digraph( nt, spotTable );

Got it, problem 1 was indeed the missing tags and was fixed by the additional analyzer.
Problem 2 was something stupid and local, an unrelated and ill engineered library I downloaded was shadowing cell2mat and breaking the graph generation step. Oy, that’s 2 hours of my life I can’t get back. Thanks for all your help would have taken twice as long to zero in on it if I wasn’t sure it should be working…

1 Like