Speeding up ImageJ macro py parallel image loading and processing (Preloader)

haesleinhuepf · March 2, 2019, 1:05pm

Dear *,

I’m working on a procedure for speeding up ImageJ macros by parallel loading and processing of images.

The idea is loading image2 while image1 is processed. Afterwards, image3 will be loaded while image2 is processed and so on.

I just programmed a prototype which performs pretty well. I can spare 30% of overall processing time.

You can try it yourself by downloading the preloader.jar file, putting it in the jars subdirectory of your Fiji installation and restarting Fiji. Try this macro to reproduce my log window above:

github.com

haesleinhuepf/preloader/blob/master/src/main/macro/preloaderDemo.ijm#L55


time = getTime();
for (i = 0; i < lengthOf(fileList2); i++) {
	run("Close All");
	
	filename = fileList2[i];
	if (i < lengthOf(fileList2) - 1) {
		nextFilename = fileList2[i + 1];
	}
	
	loadStartTime = getTime();
	run("Preloader", "current=" + filename + " next=" + nextFilename);
	deltaTime = getTime() - loadStartTime;
	print("Loading a single file with preloading took " + deltaTime + " msec");


	// process the image
	run("Mean 3D...", "x=3 y=3 z=3");
}
deltaTime = getTime() - time;
print("Loading " + lengthOf(fileList1) + " files with preloading and processing them took " + deltaTime + " msec" );

Also: Find the source code of the plugin here.

I was wondering if something like this aleady exists somewhere in the ImageJ/Fiji universe. If not, might there be a place for the preloader in the Fiji distribution?

Cheers,
Robert

ThomasBoudier · March 4, 2019, 12:29am

Hi @haesleinhuepf,

Maybe not fully related but with the same idea of Batch Processing, you can have a look to TAPAS. The idea is to separate the data to process (that can be eventually pre-loaded) and the processing itself.

Best,

Thomas

haesleinhuepf · March 5, 2019, 5:39pm

Just a little update FYI: The Preloader and his little brother the RawPreloader now ship with the clij update site. I think speedup via preloading and speedup via GPU-acceleration go well hand in hand.

Debora_KellerOlivier · March 20, 2019, 11:40am

hmm a beginner here trying to start with macros et al (by copy/pasting existing ones and modifying) but I have to process multiple SIM images (biiig) and was thinking of including your preloader in my humble macro…

Dialog.create("Batch Processing - SIM Thresholding");
	Dialog.addCheckbox("Use Bio-Formats Importer ", true);
Dialog.show();
		Bio=Dialog.getCheckbox();
		
input = getDirectory("location where images are stored");



//select location where images are stored
output = getDirectory("Location for results");

list = getFileList(input);

setBatchMode(true);

//loop to sequentially open images
for (im=0; im<list.length; im++){{
full = input + list[im];


if (Bio==true){
run("Bio-Formats Importer", "open=full autoscale color_mode=Composite view=Hyperstack stack_order=XYCZT");
} else {
open(full);
}

filename=getTitle();
//Get the name without the extension
ShortFileName=substring(filename, 0, lastIndexOf(filename,"."));
run("Threshold and 16-bit Conversion", "auto-scale");
}
rename(ShortFileName+" _THR ");
print(ShortFileName+" _THR " + " done");
out = output + ShortFileName + " _THR ";
saveAs("tiff", out);
run("Close All");
number=number+1;


setBatchMode(false);


print("All done - " + number + " images processed");

should I then just add “run(“Preloader”, “current=” + filename + " next=” + nextFilename);" on my line 23 within the for loop (instead of the bioformats importer)?

Cheers

Debbi

haesleinhuepf · March 20, 2019, 12:06pm

Hi @Debora_KellerOlivier,

I’m very sorry, but the preloader isn’t compatible with BioFormats yet. But you’re not the first one asking for this. I’ll try to have a look into this during spring and let you know if there is something to test.

Thanks for the suggestion btw. I love seeing users pushing developers in the right direction!

Cheers,
Robert