Memory not clearing over time in batch macro

imagej
macro
memory

#1

Hello,

I’ve written a macro that iterates through all Tiff files in a given directory and runs some operations on them (specifically runs the Thunderstorm plugin), based on the names of the images. The thing does work, but even though I close() every image, the memory isn’t clearing. After 3 movies or so, the script stops with an “out of memory error”. If I monitor the memory use, it just ramps up every time a new movie is opened.
I’ve tried running the garbage collector after every close(), but that doesn’t do anything.

Here’s the macro in question:

//This script will cycle through all the subfolders in a particular directory
//and run THunderstorm on all the tiff files in there, then save the results as .csv files
   dir = getDirectory("Choose a Directory "); 
   setBatchMode(true); 
   count = 0; 
   countFiles(dir); 
   n = 0; 
   processFiles(dir);
   print(count+" files processed");
   
   function countFiles(dir) {
      list = getFileList(dir);
      for (i=0; i<list.length; i++) {
          if (endsWith(list[i], "/"))
              countFiles(""+dir+list[i]);
          else
              count++;
      }
  }

   function processFiles(dir) {
      list = getFileList(dir);
      for (i=0; i<list.length; i++) {
          if (endsWith(list[i], "/"))
              processFiles(""+dir+list[i]);
          else {
             showProgress(n++, count);
             path = dir+list[i];
             processFile(path);
          }
      }
  }

  function processFile(path) {
       if ((endsWith(path, ".tif") == true) && (indexOf(path, "Avg_") < 0) && (indexOf(path, "BL") < 0) && (indexOf(path, "_avg") <0) && (indexOf(path, "cnp1") > 0)) {
           run("TIFF Virtual Stack...", "open=path");
           run("Run analysis", "filter=[Wavelet filter (B-Spline)] scale=2.0 order=3 detector "+
	    "detector=[Local maximum] connectivity=8-neighbourhood threshold=1.5*std(Wave.F1) "+
	    "estimator=[PSF: Integrated Gaussian] sigma=1.4 method=[Weighted Least squares] fitradius=4 mfaenabled=false "+
	    "renderer=[Averaged shifted histograms] magnification=12.9 colorizez=true shifts=2 "+
	    "repaint=50 threed=false");
		   run("Show results table", "action=drift magnification=5.0 method=[Cross correlation] save=false steps=10 showcorrelations=true");
		   run("Export results", "filepath=["+dir+list[i]+"_Results.csv"+"] file=[CSV (comma separated)] "+
			    "sigma=true intensity=true chi2=true offset=true saveprotocol=true x=true y=true bkgstd=true id=true uncertainty_xy=true frame=true");
		   selectWindow("Averaged shifted histograms");
		   setMinAndMax(0, 20);
		   saveAs("PNG", dir+"cnp1.png");
		   run("Cyan Hot");
		   saveAs("PNG", dir+"color_cnp1.png");					
		   close();
       }         
	   
	    if ((endsWith(path, ".tif") == true) && (indexOf(path, "Avg_") < 0) && (indexOf(path, "BL") < 0) && (indexOf(path, "_avg") < 0) && (indexOf(path, "sad1") > 0)) {
	       run("TIFF Virtual Stack...", "open=path");
           run("Run analysis", "filter=[Wavelet filter (B-Spline)] scale=2.0 order=3 detector "+
	    "detector=[Local maximum] connectivity=8-neighbourhood threshold=1.5*std(Wave.F1) "+
	    "estimator=[PSF: Integrated Gaussian] sigma=1.4 method=[Weighted Least squares] fitradius=4 mfaenabled=false "+
	    "renderer=[Averaged shifted histograms] magnification=12.9 colorizez=true shifts=2 "+
	    "repaint=50 threed=false");
		   run("Export results", "filepath=["+dir+list[i]+"_Results.csv"+"] file=[CSV (comma separated)] "+
			    "sigma=true intensity=true chi2=true offset=true saveprotocol=true x=true y=true bkgstd=true id=true uncertainty_xy=true frame=true");
		   selectWindow("Averaged shifted histograms");
		   setMinAndMax(0, 20);
		   saveAs("PNG", dir+"sad1.png");
		   close(); 	   
	                }
	    if ((endsWith(path, ".tif") == true) && (indexOf(path, "Avg_") < 0) && (indexOf(path, "BL") < 0) && (indexOf(path, "_avg") < 0) && (indexOf(path, "sad1") < 0) && indexOf(path, "cnp1") < 0) {
	    	run("TIFF Virtual Stack...", "open=path");
            run("Run analysis", "filter=[Wavelet filter (B-Spline)] scale=2.0 order=3 detector "+
	    "detector=[Local maximum] connectivity=8-neighbourhood threshold=1.5*std(Wave.F1) "+
	    "estimator=[PSF: Integrated Gaussian] sigma=1.4 method=[Weighted Least squares] fitradius=4 mfaenabled=false "+
	    "renderer=[Averaged shifted histograms] magnification=12.9 colorizez=true shifts=2 "+
	    "repaint=50 threed=false");
		    run("Show results table", "action=drift magnification=5.0 method=[Cross correlation] save=false steps=10 showcorrelations=true");
		    run("Export results", "filepath=["+dir+list[i]+"_Results.csv"+"] file=[CSV (comma separated)] "+
			    "sigma=true intensity=true chi2=true offset=true saveprotocol=true x=true y=true bkgstd=true id=true uncertainty_xy=true frame=true");
		    selectWindow("Averaged shifted histograms");
		    setMinAndMax(0, 20);
		    saveAs("PNG", dir+"POI.png");
		    run("Red Hot");
		    saveAs("PNG", dir+"color_POI.png");					
		    close();    
	          
      }
      
  }

A few more details, I’m running it on a Windows 7 machine, java.version: 1.6.0_24
The movies I’m opening are micromanager movies and they’re quite large, about 10k frames per movie (super resolution movies).

I’m not good at ImageJ scripting by any means, so if anyone has an idea on how to fix it, I’d be extremely grateful. Also, I got the first part of the script (the part that iterates through a directory) online.

Cheers,
David


#2

What’s this “Run analysis” command that you’re using? I’d suppose that this might keep a reference to the open image, so that the memory doesn’t get freed…


#3

It’s part of the Thunderstorm plugin. Is there any way to fix that with ImageJ? The plugin itself has no options for memory clearing.
I also can’t get rid of this command as it’s sort of the main point of the whole script.


#4

If the issue is really in the plugin, there’s no way to fix that within ImageJ. The plugin needs to be updated in this case. If you want to dig into the code, you can have a look at the source in this repository:

I invited the author, Martin Ovesny, to join this discussion. Maybe he’ll be able to help.


#5

All right, if I run the plugin manually and do all the operations in my macro, then close everything and run the garbage collector (or click on the memory monitor window), the memory successfully clears. I assume that means it’s not an issue with Thunderstorm itself, then?


#6

Did you try running your macro with setBatchMode(false) to see if the images actually get closed as expected and the memory freed up?


#7

That seems to have fixed it! Thanks! I can now run it on a whole folder and it will finish with cleared memory.
What does the batch mode even do, then? Like I said, I got that part of the code from a different source. I didn’t want to change it since whoever wrote it obviously knew what they were doing better than I did :slight_smile:


#8

Do you mean it frees the memory when not running in batch mode, but still has the issue when running with batch mode? That seems to me like an ImageJ issue indeed. @Wayne did you observe similar behavior in the past?

It prevents newly opened images from being displayed, making the processing a lot faster, usually.

See also the documentation at https://imagej.net/developer/macro/functions.html#setBatchMode


#9

Actually, the whole macro functions as it should if I disable batch mode, the memory is properly cleared after it finishes a file. I am fine with batch mode being disabled.

Thanks for the help!


#10

Additional suggestion: If you want to keep batch mode, you can also
run("Collect Garbage");
after you close the images before the next loop starts.


#11

I don’t think this will help. Garbage collection should be automatically triggered before the JVM runs out of memory, so this line just adds additional clutter to your script and probably won’t solve the original issue.

But I might be wrong.


#12

Yes, this is absolutely how things are supposed to work - but I have also run into similar issues before and found this addition does help a little, even in situations where garbage collection should already have been taken care of.