TrackMate memory issue when run in MATLAB

Hello,

I’ve used the TrackMate (v3.4.2) plugin to sucessfully analyze my data. The tracking component finishes in 1.5 s with little to no change in memory usage. Here is the output of the TrackMate plugin:

TrackMate v3.4.2 started on:
Tue, 13 Dec 2016 18:05:10
Image data:
For the image named: 10min_4X_20fps_8-bit_001.tif.
Matching file 10min_4X_20fps_8-bit_001.tif in folder: /home/ajw68/Documents/Projects/gradient/experiments/microfluidics/cropped/split_out/
Geometry:
  X =    0 -  650, dx = 1.00000
  Y =    0 -  650, dy = 1.00000
  Z =    0 -    0, dz = 1.00000
  T =    0 -  598, dt = 1.00000
Spot feature analyzers:
 - Manual spot color analyzer provides: Spot color; is manual.
 - Spot descriptive statistics provides: Mean, Median, Min, Max, Total int., Stdev.
 - Spot radius estimator provides: Diam.
 - Spot contrast and SNR provides: Constrast, SNR.
Edge feature analyzers:
 - Edge target provides: Source ID, Target ID, Cost.
 - Edge mean location provides: T, X, Y, Z.
 - Edge velocity provides: V, D.
 - Manual edge color analyzer provides: Edge color; is manual.
Track feature analyzers:
 - Branching analyzer provides: N spots, Gaps, Longest gap, Splits, Merges, Complex.
 - Track duration provides: Duration, T start, T stop, Displacement.
 - Track index provides: Index, ID.
 - Track location provides: X, Y, Z.
 - Velocity provides: Mean V, Max V, Min V, Median V, V std.
 - TRACK_SPOT_QUALITY provides: Mean Q, Max Q, Min Q, Median Q, Q std.
Computing spot features.
Computation done in 28 ms.
Computing edge features:
  - Edge target in 0 ms.
  - Edge mean location in 0 ms.
  - Edge velocity in 0 ms.
Computation done in 1 ms.
Computing track features:
  - Branching analyzer in 0 ms.
  - Track duration in 0 ms.
  - Track index in 0 ms.
  - Track location in 0 ms.
  - Velocity in 0 ms.
  - TRACK_SPOT_QUALITY in 0 ms.
Computation done in 0 ms.
Starting detection using DoG detector
with settings:
  - target channel: 1
  - threshold: 0.2
  - do median filtering: true
  - radius: 2.0
  - do subpixel localization: true
Starting detection process using 8 threads.
Detection processes 8 frames simultaneously and allocates 1 thread per frame.
Found 132747 spots.
Detection done in 27.5 s.
Computing spot quality histogram...
Histogram calculated in 0.3 s.
Initial thresholding with a quality threshold above 0.0 ...
Starting initial filtering process.
Retained 132747 spots out of 132747.
Calculating spot features...
Computing spot features.
Computation done in 4573 ms.
Calculating features done in 4.6 s.
Performing spot filtering on the following features:
No feature threshold set, kept the 132747 spots.
Starting tracking using Simple LAP tracker
with settings:
  Linking conditions:
    - max distance: 15.0
    - no feature penalties
  Gap-closing conditions:
    - max distance: 15.0
    - max frame gap: 1
    - no feature penalties
  Track splitting not allowed.
  Track merging not allowed.
Starting tracking process.
Found 4624 tracks.
Tracking done in 1.5 s.

However, I’ve moved it over to a MATLAB script with the same settings, and the memory blows up. I’m using 2016b. I’ve given it up to 27 GB and it always takes up all of the memory. Here is the code I’m using:

Miji(false);
imp = ij.ImagePlus(fullfile('.', tif_out));

model = fiji.plugin.trackmate.Model();
model.setLogger(fiji.plugin.trackmate.Logger.IJ_LOGGER);

settings = fiji.plugin.trackmate.Settings();
settings.setFrom(imp);

settings.detectorFactory = fiji.plugin.trackmate.detection.DogDetectorFactory();
map = java.util.HashMap();
map.put('DO_SUBPIXEL_LOCALIZATION', true);
map.put('RADIUS', 2);
map.put('TARGET_CHANNEL', 1);
map.put('THRESHOLD', 0.2);
map.put('DO_MEDIAN_FILTERING', true);
settings.detectorSettings = map;

filter1 = fiji.plugin.trackmate.features.FeatureFilter('QUALITY', 0.0, true);
settings.addSpotFilter(filter1);

settings.trackerFactory  = fiji.plugin.trackmate.tracking.oldlap.SimpleLAPTrackerFactory();
settings.trackerSettings = fiji.plugin.trackmate.tracking.LAPUtils.getDefaultLAPSettingsMap();
settings.trackerSettings.put('LINKING_MAX_DISTANCE', 15);
settings.trackerSettings.put('GAP_CLOSING_MAX_DISTANCE', 15);
settings.trackerSettings.put('MAX_FRAME_GAP', java.lang.Integer(1));
settings.trackerSettings.put('ALLOW_TRACK_SPLITTING', false);
settings.trackerSettings.put('ALLOW_TRACK_MERGING', false);

trackmate = fiji.plugin.trackmate.TrackMate(model, settings);

ok = trackmate.checkInput();
if ~ok
    display(trackmate.getErrorMessage());
end

ok = trackmate.process();
if ~ok
    display(trackmate.getErrorMessage());
end

fiji.plugin.trackmate.action.ExportTracksToXML.export(model, settings, res_out);

One difference I know is happening is that the Z and T dimensions are swapped when I run it through matlab. I vaguely remember seeing how to fix this somewhere, but I’ve looked around and cannot find it. Does anyone know how to fix the geometry?

The swapping of Z and T dimensions is hard-coded inside the TrackMate GUI component GuiUtils, which makes it unsuitable to use directly from a script:

You can however just copy that line into your own code:

imp.setDimensions( dims[ 2 ], dims[ 4 ], dims[ 3 ] );

Have you seen the ImageJ-Matlab project? It is meant to replace the old Miji in the long term.

1 Like

Thanks @imagejan! I also found the original post describing how to do this. I will try swapping the dimensions and see if it fixes the problem.

Yes, I am using Miji because of the problems I’m having with ImageJ-Matlab. I made another post about these issues.

1 Like

I was able to get the geometry correct with the following code:

Miji(false);
imp = ij.ImagePlus(fullfile('.', tif_out));

% width, height, nChannels, nSlices, nFrames
dims = imp.getDimensions();

% nChannels, nSlices, nFrames
imp.setDimensions(dims(3), dims(5), dims(4));

Running TrackMate now works better, in that it gets past creating the Cost matrix and uses less memory. However, there are clearly some problems since it still uses much more memory than the ImageJ version (~8 GB vs <1 GB), and has now been running for several minutes, whereas the plugin finishes in 1.5s.

I’ll let it run, but I’m not sure if it is ever going to finish. This looks like the same thing as this problem.

Note that in that post, @tinevez mentions that it might be a memory issue related to the number of spots. However, in my case, the plugin version finishes on the same data without issue.

Here is the output of the logger:

Starting detection process using 8 threads.
Detection processes 8 frames simultaneously and allocates 1 thread per frame.
Found 132747 spots.
Starting initial filtering process.
Computing spot features.
Computation done in 14 ms.
Starting spot filtering process.
Starting tracking process.
Frame to frame LAP solved in 3.8 s.
Cost matrix for track segments created in 18.7 s.

UPDATE: I also tried reducing the linking distance to 9px. It uses a little bit less memory (~6GB), and the steps finish a little bit faster, but it still hangs after the cost matrix. Here is the output of the logger:

Starting detection process using 8 threads.
Detection processes 8 frames simultaneously and allocates 1 thread per frame.
Found 132747 spots.
Starting initial filtering process.
Computing spot features.
Computation done in 14 ms.
Starting spot filtering process.
Starting tracking process.
Frame to frame LAP solved in 3.5 s.
Cost matrix for track segments created in 16.7 s.

This is new to me. Let’s try shooting at random.
First, can you try to use only 1 thread for calculation. To do so, do this after creating the trackmate:

trackmate.setNumThreads(1)

and copy/paste the log output.

First, I tried calling ImageJ; instead of Miji(false);. Since the ImageJ gui shows up in this case, I can see that it is getting stuck at the “Solving for final tracks…” step. This line does not show up in the logger window. One processor stays pegged at 100% during this time.

(By the way, how do I terminate a process that is stuck like this? I’ve just been killing my matlab process, which is a pain.)

I set trackmate.setNumThreads(1) and got the same behavior, except the detection step ran on one processor:

Starting detection process using 1 thread.
Detection processes 1 frame simultaneously and allocates 1 thread per frame.
Found 132747 spots.
Starting initial filtering process.
Computing spot features.
Computation done in 13 ms.
Starting spot filtering process.
Starting tracking process.
Frame to frame LAP solved in 11.9 s.
Cost matrix for track segments created in 14.1 s.

@tinevez, about to head out for vacation over here, and perhaps you are as well, but I was wondering if there was anything I could do to help debug this problem. I’d really like to use TrackMate if possible.

Nothing yet. I do not know how to even start with this one. Let’s talk about it when you come back.
Best
jy

If it’s helpful, I can email you with a dropbox link to an example data that causes the hang up. It’s about 250 MB.

That would be great actually. With the script?

The script is in the first post (let me know if you need something I’ve left out). I am emailing you a link to the data.

Ok I could get it to work. It took me ages and in the process I have found the source of the bug reported for ‘ImageJ-matlab’.

But first things first:

  • Do you have Java 8 installed on your system? If not install it.
  • Check that in your Fiji you have the java8 site enabled. Is it the case? If not, check it and run the long update until the end. You need that latest version of Fiji and TrackMate for this to work.

Then, most likely, your Miji won’t work anymore. The reason is that MATLAB ships that v1.7 of Java, and Fiji and friends now all depend on v1.8. This is why the Miji and friends fail.

So you need to launch MATLAB with the JVM pointing to the latest java 1.8 version. This is explained here.

Then tell us what you found.

1 Like

Java 8 is installed, the Java-8 site is enabled on Fiji, and MATLAB is using java 1.8. I did all of these things before using ImageJ-matlab or Miji. Basically, you can read from the beginning of this thread assuming that I already did all of these things.

Miji still works with this configuration. Here is the output:

>> version -java

ans =

Java 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14 with Oracle Corporation OpenJDK 64-Bit Server VM mixed mode

>> Miji


Use MIJ.exit to end the session

--------------------------------------------------------------
MIJ 1.3.6-fiji: Matlab to ImageJ Interface
--------------------------------------------------------------
More Info: http://bigwww.epfl.ch/sage/soft/mij/
Help: MIJ.help
JVM> 1.3.6-fiji
JVM> Version: 1.8.0_111
JVM> Total amount of memory: 540160 Kb
JVM> Amount of free memory: 403339 Kb
ImageJ> Version:1.51h
ImageJ> Memory:152MB of 25000MB (<1%)
ImageJ> Directory plugins: /usr/share/Fiji/plugins/
ImageJ> Directory macros: /usr/share/Fiji/macros/
ImageJ> Directory luts: /usr/share/Fiji/luts/
ImageJ> Directory image: Not specified
ImageJ> Directory imagej: /usr/share/Fiji/
ImageJ> Directory startup: /usr/share/Fiji/
ImageJ> Directory home: /home/ajw68/
--------------------------------------------------------------
Status> ImageJ is running.
--------------------------------------------------------------

ans =

MIJ@24841372

Let me know if any of it looks suspicious.

Has this update been released? I tried running my script using ImageJ-Matlab instead of Miji and nothing has changed.

No @ajw,

I was referring to the Java8 on MATLAB bug. So the issue is something else unfortunately.

Sorry @tinevez, now I’m confused! When you said

you were not referring to getting TrackMate to work on the tif stack I sent you within MATLAB?

Were you able to run a test on the tif stack I sent you?

Now the code runs but I have a “OutOfMemory error”:

Starting detection process using 8 threads.
Detection processes 8 frames simultaneously and allocates 1 thread per frame.
Found 132747 spots.
Starting initial filtering process.
Computing spot features.
Computation done in 3 ms.
Starting spot filtering process.
Starting tracking process.
  Frame to frame LAP solved in 3.5 s.
  Cost matrix for track segments failed on following error: LAPTracker: Not enough memory.
  Skipping track segment LAP.
  Track segment LAP aborted after 4.6 s.
Computing track features:
Computation done in 3 ms.
Starting track filtering process.
Computing edge features:
Computation done in 0 ms.

even though Miji declares that there is a lot of memory available.
This might be caused by the fact that MATLAB on the other hand states that it can only allocates up to 2GB to Java heap (in MATLAB preferences).

What do you have for this limit on your side?

Yes, I had this problem initially. You need to increase the Java heap past what the gui allows.

I set the java heap space to 27GB by adding the line

JavaMemHeapMax=I27000

to ~/.matlab/R2016b/matlab.prf.

The other way to do this is to create a file called java.opts in your startup directory, and add the line -Xmx27g to it.

1 Like

Hi @tinevez, have you been able to increase the memory and run it successfully?

I failed on my PC: not enough RAM whatever I do.
I am moving to a Mac with more RAM now, I will keep you posted.

If you can spare the time: is there anyway you could measure the memory consumption over time while you run TrackMate? I suspect it is a memory problem.