Batch processing macro

I’ve posted on this forum before about writing this batch processing macro and received a lot of help before but I’ve got one last problem that needs a fresh pair of eyes.

This macro is designed to run through a folder of C01 files, separate them by their layers (d0, d1 & d2) and then run the analytical functions for counting various fluorescent markers and then counting any overlap.
The problem is that rather than looping through all of the C01 files, it becomes stuck on the first one, but when the results are exported, the filenames correspond to all files in the folder but all have the same result.
Below is the code:

inDir = getDirectory("C:/Users/laure/OneDrive/Documents/MSc Molecular Biology & Biotechnology/Masters Project - Dombrowski Lab/Project Data/LD006 CX5/CellHealthProfiling.V4_03-15-20_01;35;58/CEM-133432_200315130001/Test files");
outDir = getDirectory("C:/Users/laure/OneDrive/Documents/MSc Molecular Biology & Biotechnology/Masters Project - Dombrowski Lab/Project Data/LD006 CX5/CellHealthProfiling.V4_03-15-20_01;35;58/CEM-133432_200315130001/Results"); 

list = getFileList(inDir);

C01list = newArray(0);
for (i = 0; i < list.length; i++) {
if (endsWith(list[i], ".C01")) {
	C01list = Array.concat(C01list, list[i]);
 }
} 


d0_List = newArray(0);
d1_List = newArray(0);
d2_List = newArray(0);

for (j = 0; j < C01list.length; j++){
if (matches(C01list[j], ".*d0.*")) {
	d0_List = Array.concat(d0_List, C01list[j]);
}

if (matches(C01list[j], ".*d1.*")) {
	d1_List = Array.concat(d1_List, C01list[j]);
   	}

if (matches(C01list[j], ".*d2.*")) {
	d2_List = Array.concat(d2_List, C01list[j]);
}
}


Array.sort(d0_List);
Array.sort(d1_List);
Array.sort(d2_List);

setBatchMode(true);

for(img = 0; img < d0_List.length; img++){

run("Bio-Formats Importer", "open=[" + inDir + File.separator + d0_List[img] + "] color_mode=Default rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
DAPI = getTitle();
run("Bio-Formats Importer", "open=[" + inDir + File.separator + d1_List[img] + "] color_mode=Default rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
OLIG = getTitle();
run("Bio-Formats Importer", "open=[" + inDir + File.separator + d2_List[img] + "] color_mode=Default rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
MBP = getTitle();

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);

print("The image being processed is " + imgName);

selectWindow(DAPI);
run("8-bit");
setThreshold(45, 255); //was 45 in windows
run("Convert to Mask", "method=Default background=Dark calculate black"); 

run("Analyze Particles...", "size=30-350 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_Particles"); 
		
selectWindow(OLIG);
run("8-bit");
setThreshold(80, 255); //was 80 in windows
run("Convert to Mask", "method=Default background=Dark calculate black");

run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_OLIG_Particles"); 
			

selectWindow(MBP);
run("8-bit");
setThreshold(150, 255); //was 150 in Windows
run("Convert to Mask", "method=Default background=Dark calculate black");

run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_MBP_Particles");


imageCalculator("AND create", DAPI, OLIG); 
DAPI_OLIG = getTitle();
run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_OLIG_Outline");
	
imageCalculator("AND create", DAPI_OLIG, MBP); 
DAPI_OLIG_MBP = getTitle();
run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_OLIG_MBP_Overlap");

run("Close All");

}

setBatchMode(false);

selectWindow("Summary");
saveAs("Results", outDir + File.separator + imgName + ".csv");

Dialog.create("Progress"); 
Dialog.addMessage("Macro Complete!");
Dialog.show; 

I have consulted with another colleague and we both think the the problem comes from these couple of lines of script below:

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);

This script is directly linked to the output but I have run out of ideas on how to fix/work around it. I tried creating indexes for each marker, as written below, and inserting the string into the output but that didn’t work.

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);
d1Index = indexOf(OLIG, "d");
imgName1 = substring(OLIG, 1, d1Index);
d2Index = indexOf(MBP, "d");
imgName2 = substring (MBP, 2, d2Index);

I also tried taking the macro back to basics and using “if matches” once I had created the C01list array to separate out each of the layers and analyse them but that also didn’t work as it wouldn’t export a summary.

Any help would be greatly appreciated as I’m trying to get this up and running before my MSc thesis deadline in three weeks.

Thanks!
Lauren

To clarify, both the tiff save and the results table both include the measurements from the first layer of the first file, and are all the exact same size?
Or do you get some number of files out per layer for the first file, and then not subsequent files?

All tiff files saved and the results table have the measurements from the first layer of the first image but on the results table, the names of each result differ as though it had done as it was meant to and analysed all of the images in that file.

I don’t really know what your file structure looks like as I am not familiar with the format, but have you printed these to make sure they are what you think they are?

I don’t see a clear results command, but I guess that isn’t a problem right now.

Can we also assume the print statement showing the imgName is working? I guess that would answer the first question.

Also, what happens when you turn off batch mode and step through the macro one line at a time? Do the wrong images pop up?

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);

Do the print statements for these match what you expect when you step through the macro one line at a time?

Separate from the other issues, though:

selectWindow("Summary");
saveAs("Results", outDir + File.separator + imgName + ".csv");

That part is outside your loop. So you do expect to get only one CSV file for the entire script.

The analytical commands should return a results table at the end, this macro was built on top of an existing one that had the same commands for analysing the data but it just did one file at a time and then you had to enter the next file name, I’m just trying to automate it.

Yes, the print statement for imgName is working.

I commented everything out after the first set of commands related to DAPI and it still processed the same image repeatedly for all “d0” layers.

That part being outside the loop was deliberate, I wanted one document where all the results were compiled rather than an excel file per result, I made that mistake the first time and ended up with numerous excel files.

I put this up because whilst I have been teaching myself how to write macro scripts, I only started learning how to do it a couple of months ago so I’m still a novice when it comes to anything to do with coding - this is just what I’ve put together with a lot of help from other people.

Thanks for getting back to me so quickly.

Lauren

1 Like

Hi Lauren,

Sorry, I didn’t realise you had replied on that other thread, sorry you have still been struggling all this time.

The

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);

Section is a red herring I think, it is however, what made me realise the code was mine because I get pernickity about file names. All it does istake the below file name
CEM-133432_200229080001_B02f01d2.C01
and make it
CEM-133432_200229080001_B02f01
It does this by finding the d and just retuning what’s in front of it.

It’s really a silly preference I probably should have not put in the macro. Like you could just take those two lines out and replace all the “imgName” appearances with the DAPI variable and it wouldn’t matter.

So you are saying that the saved images you get are always of your first image? Could you maybe provide two sets of images so 6 files so someone can test out this behaviour? Is the CSV file okay?

I think the issue is how you are opening the files and how BioFormats reads this particular format.
Based on the code, you think you are opening each individual channel, but that isn’t how bioformats treats the files.

As a test, take any individual file of the 6 sample files, and drag it into FIJI. No matter which one you use, it will find the other 5 and import as one batch of 6 files.

Therefore, when you are running your macro, you are always grabbing the same exact object (a collection of 6 channels as 2 figures). Looking at how to adjust the BioFormats importer code now.

Ah, think this might do it.


With the macro recorder:

run("Bio-Formats Importer", "open="+namehere+" color_mode=Default open_files rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");

Note the addition of the “open_files” option.

1 Like

Hi,

Don’t worry about it Laura, I thought I would just try to figure it out myself. I think the only reason my colleague & I thought it was that script was because we couldn’t see any problem with the rest of it.

Also, I’ll try both those edits and hopefully get this up and running - I really appreciate you both getting back to me.

Thanks again to both of you!
Lauren

1 Like

Hey guys,

Just thought I’d update you both, its finally up and running! I added a few lines to the analytical functions to sort the brightness and contrast, fixed the thresholds and it works! I now need to run it against my manual processing and the original macro to see how accurate it is but thank you so, so much for all the help. I definitely would not have got this far without your help. The (mostly) final version is below.

Thank you!
Lauren

inDir = getDirectory("C:/Users/laure/OneDrive/Documents/MSc Molecular Biology & Biotechnology/Masters Project - Dombrowski Lab/Project Data/LD006 CX5/CellHealthProfiling.V4_03-15-20_01;35;58/CEM-133432_200315130001/Test files");
outDir = getDirectory("C:/Users/laure/OneDrive/Documents/MSc Molecular Biology & Biotechnology/Masters Project - Dombrowski Lab/Project Data/LD006 CX5/CellHealthProfiling.V4_03-15-20_01;35;58/CEM-133432_200315130001/Results"); 

list = getFileList(inDir);

C01list = newArray(0);
for (i = 0; i < list.length; i++) {
if (endsWith(list[i], ".C01")) {
	C01list = Array.concat(C01list, list[i]);
}
} 


d0_List = newArray(0);
d1_List = newArray(0);
d2_List = newArray(0);

for (j = 0; j < C01list.length; j++){
if (matches(C01list[j], ".*d0.*")) {
	d0_List = Array.concat(d0_List, C01list[j]);
}

if (matches(C01list[j], ".*d1.*")) {
	d1_List = Array.concat(d1_List, C01list[j]);
}

if (matches(C01list[j], ".*d2.*")) {
	d2_List = Array.concat(d2_List, C01list[j]);
}
}


Array.sort(d0_List);
Array.sort(d1_List);
Array.sort(d2_List);

setBatchMode(true);

for(img = 0; img < d0_List.length; img++){

run("Bio-Formats Importer", "open=[" + inDir + File.separator + d0_List[img] + "] color_mode=Default open_files rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
DAPI = getTitle();
run("Bio-Formats Importer", "open=[" + inDir + File.separator + d1_List[img] + "] color_mode=Default open_files rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
OLIG = getTitle();
run("Bio-Formats Importer", "open=[" + inDir + File.separator + d2_List[img] + "] color_mode=Default open_files rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
MBP = getTitle();

dIndex = indexOf(DAPI, "d" );
imgName = substring(DAPI, 0, dIndex);

print("The image being processed is " + imgName);

selectWindow(DAPI);
run("8-bit");
run("Brightness/Contrast...");
run("Enhance Contrast", "saturated=0.35");
run("Apply LUT");
setThreshold(45, 255); //was 45 in windows
run("Convert to Mask", "method=Default background=Dark calculate black"); 

run("Analyze Particles...", "size=30-350 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_Particles"); 
		
selectWindow(OLIG);
run("8-bit");
run("Brightness/Contrast...");
run("Enhance Contrast", "saturated=0.35");
run("Apply LUT");
setThreshold(80, 255); //was 80 in windows
run("Convert to Mask", "method=Default background=Dark calculate black");

run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_OLIG_Particles"); 
			

selectWindow(MBP);
run("8-bit");
run("Brightness/Contrast...");
run("Enhance Contrast", "saturated=0.35");
run("Apply LUT");
setThreshold(150, 255); //was 150 in Windows
run("Convert to Mask", "method=Default background=Dark calculate black");

run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_MBP_Particles");


imageCalculator("AND create", DAPI, OLIG); 
DAPI_OLIG = getTitle();
run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_OLIG_Outline");
	
imageCalculator("AND create", DAPI_OLIG, MBP); 
DAPI_OLIG_MBP = getTitle();
run("Analyze Particles...", "size=30-250 show=Outlines clear summarize add");
saveAs("Tiff", outDir + File.separator + imgName + "_DAPI_OLIG_MBP_Overlap");

run("Close All");

}

setBatchMode(false);

selectWindow("Summary");
saveAs("Results", outDir + File.separator + imgName + ".csv");

Dialog.create("Progress"); 
Dialog.addMessage("Macro Complete!");
Dialog.show;
2 Likes