Can you clear specific items from Clij2 GPU memory/General Memory issues

Hi all,
I made an ImageJ macro utilizing Clij2 to process some large behavior movies for easier automated tracking (using the absolute difference function on Clij2 for BG subtraction).

In order to make it work with Clij I have to break the movie up into smaller chunks, process these, and then concatenate the processed files in a separate macro.

If I run the macro on a folder of movies, the first movies are able to save all 12 pieces correctly, but as I get further into the set of movies, I start having memory issues where Clij2 cannot save certain files because
"CLIJ Error: Creating an image or kernel failed because your device ran out of memory. "

During the processing of each chunk of movie, I’m creating 6 total images in the GPU’s memory. I could save memory if I could specifically clear out certain files during the GPU processing step, instead of just clearing everything out of memory at the end of each run of the loop.

More generally, my system memory is 98% utilized while running this script. I’m a novice coder and haven’t had to worry too much about memory demands up until this point. I have tried calling GarbageCollecter within the macro but it doesn’t seem to be helping much. Are there other tricks I could use for managing system memory when batch processing large movies?

Generally – everything works OK if I run it overnight without trying to simultaneously use my computer, but it would be nice to allow it to run in the background during the day as well.

Thanks!

/* Auto-process behavior movies for ToxTrac
 * Author: Liz M. Haynes
 * contact: ehaynes2@wisc.edu
 */

//get directories from user
myDir1 = getDirectory("Choose Source Directory "); 
myDir2 = myDir1+"BG_Subtracted"+File.separator;
  File.makeDirectory(myDir2);
  if (!File.exists(myDir2)){
  	exit("Cannot Make Background Subtracted Folder");
  }
imageList = getFileList(myDir1); 

//batch processing loop
setBatchMode(true); 
for (i=0; i<imageList.length; i++) { 
 if (endsWith(imageList[i], ".avi")) {
  showProgress(i+1, imageList.length);
  myPath = myDir1 + imageList[i]; 
  print(myPath);
  run("AVI...", "open=[" + myPath+"] use");
  print("Opening \"" + myPath + "\"");
   id = getTitle();
   id = replace(id, "\\.avi", "");
  myDir3 = myDir2+id+File.separator;
  File.makeDirectory(myDir3);
  //initialize GPU
   run("CLIJ2 Macro Extensions", "cl_device=[GeForce GTX 980 Ti]");
   Ext.CLIJ2_clear();

//reduce size of stack by half (run this twice = 5fps, 3x = 2.5fps)
  slicenumber = nSlices;
  run("Slice Remover", "first=1 last=slicenumber increment=2");
  run("Slice Remover", "first=1 last=slicenumber increment=2");
  
  slicenumber = nSlices;
  
  id = getTitle();
 
 //duplicate first (empty tank) image (make sure to do this on the right first image!)
  run("Duplicate...", "title=bg");
   run("8-bit");
  id2 = getTitle();
 //duplicate again as holder for making a stack
    run("Duplicate...", "title=bg_stk");
   run("8-bit");
  id3 = getTitle();


  //split stack further into 12 parts so GPU can handle it
    selectWindow(id); 
    endStackNum = slicenumber/12;
    beginStackNum = 1;
    stackInt = slicenumber/12;
    
  //Establish a counter so you know how many times to repeat the while loop
   imageCounter = 1;
   
  while(imageCounter<13){
  selectWindow(id);
  run("Duplicate...", "duplicate range=beginStackNum-endStackNum");
    rename("stk"+imageCounter);
    run("8-bit");

	  selectWindow("stk"+imageCounter); 
	  current = getTitle();
	
	//get difference of image
	Ext.CLIJ2_push(current);
	Ext.CLIJ2_reportMemory();
	
	Ext.CLIJ2_convertFloat(current, float);
	print("pushing \"" + current + "\"");  
	Ext.CLIJ2_push(id2);
    print("pushing \"" + id2 + "\""); 

	//make the bg image a stack so absDiff doesn't throw an error
	Ext.CLIJ2_imageToStack(id2, id3, stackInt);
	Ext.CLIJ2_convertFloat(id3, float2);
    print("making \"" + id3 + "\""); 

	image3 = "diff_"+id+"_"+imageCounter;
	Ext.CLIJ2_absoluteDifference(float, float2, image3);
	Ext.CLIJ2_reportMemory();
	
	Ext.CLIJ2_convertUInt8(image3, i3_8);
		print("Difference is now \"" + i3_8 + "\""); 
		Ext.CLIJ2_saveAsTIF(i3_8, myDir3+image3);
		
		Ext.CLIJ2_clear();
		imageCounter++;
		beginStackNum = beginStackNum + stackInt;
		endStackNum = endStackNum + stackInt;
		print(beginStackNum);
		print(endStackNum);
		
  
 }

 
}

}

1 Like

Hey Liz @lizhaynes,

welcome in #clij county :wink:

Yeah, that shouldn’t be necessary. You may know, I also process thousands of light sheet imaging stacks in a row. without that kind of issues. We’ll figure out how to achieve this.

You could call CLIJ2_release(image); to remove one image from GPU memory as soon as you don’t need image anymore. Check out the basics tutorial to see this in action.

If you want to get the highest performance/speed out of your workflow, I recommend not releasing intermediate processing steps, because reusing memory is faster than reallocating memory. See also this exercise using py-clesperanto (the yet experimental successor of clij).

I can just speculate what’s going on: You call run("Duplicate...", often; and in a loop. With setBatchMode(true); these duplicates might not be deleted in CPU memory. If you had batch-mode off, many images would be shown on your screen. Thus, I recommend commenting out the batchmode thingy and working on your code until it doesn’t open so many windows anymore. A quick way of achieving this could be calling close("*"); at the beginning of your loop - it does that same as ‘CLIJ2_clear();’ but with CPU memory.

I also think you could spare some of the duplicate operations. E.g. you could use pushCurrentSlice instead of duplicating a slice and pushing it. Furthermore, you duplicate an image and save its name in variable id3. But id3 isn’t pushed to the GPU. Instead there is a stack created using imageToStack that has the same name as id3 but exists in the GPU memory.

If you explain a bit how your background subtraction works, I’m happy to give you further hints on how to implement it using clij :slight_smile: Furthermore, how large are your videos (in GB) and what GPU are you using, how much memory does it have?

Cheers,
Robert

Thanks Robert!
Another person pointed out that batch mode only suppresses the display, but it still equivalent to having an open image. I had assumed that by not displaying the images they would not be sitting and taking up memory in the same way, so I think this probably accounts for a lot!

I have a GeForce GTX 980 Ti with 6GB of memory, and 16GB system memory. The movies are around 3.2GB each, and are encoded in AVI (from the camera, I don’t have much control over this. They’re not fluorescence images, though, but movies of adult fish swimming so don’t be too worried about image quality!)

Essentially, I want to remove the tank from the background of image to get a result that contains just the fish on a mostly white background.

This makes it much easier for another automated analysis software I’m using to detect only the fish and ignore the non-ideal experimental setup :slight_smile:

I’m duplicating the first image in the series, which is before I put the fish in the tanks. I then get the absolute difference of each frame in the movie and the background frame.

I will try adding in some window closing into my loops and see if that solves the problem, and then move on to refining some other steps. I’ll report back!

1 Like

Ok cool, then let me suggest something: Don’t open the AVI file with the “use” flag, because if it’s not a virtual stack, you can write in it. Then you can push a single slice, copy it on the GPU, so that you can then push more slices (with the same name) and compute the absolute difference with the first slice. Finally pull the single slice back into the slice of the AVI stack where it came from. An example of this strategy is shown here:

Let me know if you get stuck, I’m happy to assist :slight_smile:

Cheers,
Robert

Here is the new solution I implemented from your suggestions. It is considerably more refined in comparison.

It solves the issue I was having specifically with Clij2, but unfortunately general memory usage by ImageJ is still high while running and seems to start when I begin running the macro, but before I even specify a directory, so I think it’s some strange ImageJ gremlin and not necessarily anything in the macro.

Thanks for the help, this is a much better solution. It takes about 6.5 minutes per stack to run compared to a previous 12 minutes per stack.

/* Auto-process behavior movies for ToxTrac
 * Author: Liz M. Haynes
 * contact: ehaynes2@wisc.edu
 */

//get directories from user
start = getTime();
myDir1 = getDirectory("Choose Source Directory "); 
myDir2 = myDir1+"BG_Subtracted"+File.separator;
  File.makeDirectory(myDir2);
  if (!File.exists(myDir2)){
  	exit("Cannot Make Background Subtracted Folder");
  }
imageList = getFileList(myDir1); 

//batch processing loop
setBatchMode(true); 
for (i=0; i<imageList.length; i++) { 
 if (endsWith(imageList[i], ".avi")) {
  showProgress(i+1, imageList.length);
  myPath = myDir1 + imageList[i]; 
  print(myPath);
  run("AVI...", "open=[" + myPath+"] use");
  print("Opening \"" + myPath + "\"");
   id = getTitle();
   id = replace(id, "\\.avi", "");
  myDir3 = myDir2+id+File.separator;
  File.makeDirectory(myDir3);
  //initialize GPU
   run("CLIJ2 Macro Extensions", "cl_device=[GeForce GTX 980 Ti]");
   Ext.CLIJ2_clear();

//reduce size of stack by half (run this twice = 5fps, 3x = 2.5fps)
  slicenumber = nSlices;
  run("Slice Remover", "first=1 last=slicenumber increment=2");
  run("Slice Remover", "first=1 last=slicenumber increment=2");
  
  slicenumber = nSlices;
  
  id = getTitle();
	bg_subtraction(id);

  function bg_subtraction(id) { 
	// function description
	selectImage(id);
	run("8-bit");
	run("Duplicate...", "title=bg");
	Ext.CLIJ2_push("bg");
	close("bg");
	input = id;
	for(z = 0; z < nSlices; z++) {
	Stack.setSlice(z + 1);
	// push current slice
	Ext.CLIJ2_pushCurrentSlice(input);
	// process it
	Ext.CLIJ2_absoluteDifference(input, "bg", result);
	// pull it back 
	Ext.CLIJ2_pullToCurrentSlice(result, input);
	}
	//Ext.CLIJ2_saveAsTIF(result, myDir3+id);
	Ext.CLIJ2_clear();
	selectWindow(id);
	id = replace(id, "\\.avi", "");
	run("Invert LUT");
	mySavePath = myDir3 + File.separator + id + "processed.avi";
	run("AVI... ", "compression=None frame=[5.48] save=[" + mySavePath+"]");
	close("*");
	print((getTime()-start)/1000);
	
	}
 
}
print((getTime()-start)/1000);
}

1 Like