CLIJ combined with StarDist memory issue

Howdy all,

Hoping someone (@haesleinhuepf and @uschmidt83 in particular :slight_smile: ) can help me out with an issue i am hitting.

I have some code that is using both CLIJ for nice fast filtering and binary operations and Stardist to extract nuclei. Either part works perfectly fine. But if i use CLIJ, then process with Stardist and then go back to CLIJ i get a memory exception error.

This demo code demonstrates the problem (at least on my system). If you run it and don’t tick the start dist box you will get a median and a gaussian blur image. If you tick the stardist box you wil lget the error when it tries to do the CLIJ filtering after running stardist.

Dialog.create("Run Stardist?");
Dialog.addCheckbox("Run Stardist?", false);
Dialog.show();

runStardist=Dialog.getCheckbox();

run("CLIJ2 Macro Extensions", "cl_device=");

run("Blobs (25K)");
image1="blobs.gif";
Ext.CLIJ2_push(image1);	
image2="Median";
radius=5;
Ext.CLIJ2_median2DBox(image1, image2, radius, radius);
Ext.CLIJ2_pull(image2);
Ext.CLIJ2_clear();

if(runStardist==true){
run("Command From Macro", "command=[de.csbdresden.stardist.StarDist2D], args=['input':'Median', 'modelChoice':'Versatile (fluorescent nuclei)', 'normalizeInput':'true', 'percentileBottom':'1.0', 'percentileTop':'99.8', 'probThresh':'0.6000000000000001', 'nmsThresh':'0.8', 'outputType':'Both', 'nTiles':'1', 'excludeBoundary':'2', 'roiPosition':'Automatic', 'verbose':'false', 'showCsbdeepProgress':'false', 'showProbAndDist':'false'], process=[false]");
}

Ext.CLIJ2_push(image2);
image3="Gaussian";
sigma=2.0;
Ext.CLIJ2_gaussianBlur2D(image2, image3, sigma, sigma);
Ext.CLIJ2_pull(image3);
Ext.CLIJ2_clear();

Any help would be greatly appreciated as using CLIJ in this pipeline will improved the speed by about 80%.

Cheers
Cam

1 Like

Sorry should say that that is Fiji macro code.

Hey @Cameron.Nowell,

I could imagine that this is related to TensorFlow blocking the whole GPU memory. There are online resources explaining how to fix this in Python. Maybe @uschmidt83, @frauzufall or @mweigert know how to configure this in Java/ImageJ? I’m also curious!

The workaround: Using Stardist on CPU might also fix this issue.

Cheers,
Robert

3 Likes

Ok so that’s a work around. Using the CPU for TensorFlow works and doesn’t seem to be too much of a speed hit to StarDist. Works on both that demo code above and my production code.

A bit of googling seems to hint that this is a know bug in TensorFlow and there is no fix for it as yet.

1 Like

Hi all,

Yep, I ran into this issue as well, and found the workaround.
Even running StarDist on the tiny blobs.gifwith TensorFlow 1.15 GPU almost completely fills the 24 GB of the RTX 6000 card, preventing further processing with CLIJ.

Just pinging @uschmidt83 @mweigert @frauzufall if anyone perhaps has looked into this issue already?

Thanks,
Bram

1 Like

I quickly looked at the TensorFlow Java API (is this latest version?) and found ConfigProto and GPUOptions, which seem to mirror the Python TensorFlow 1.x API to limit GPU memory (here’s how we do that).

Can someone try that in Java?

Best,
Uwe