Dear all,
Thanks to export measurement tool, it’s really easy to export the data.
Now, I have a questino that if I can script to average and standard deviation from a few features in detection measurement. Single case has got more than a million detections so I would like to average and STD from eg)DAPI nuc mean. Also if I can do batch processing to combine these averaged and STD value from other cases that would be great.
Thanks for your support.
Hi @inhwa,
It should be possible, but I feel like it would be much easier to simply export your desired measurements with the MeasurementExporter and calculate the standard deviation and average in another tool such as Excel, or Python (scikit-learn, pandas, numpy, …). What do you think?
Hi, @melvingelbard ,
Thanks for your suggestion.
But I have more than 150 cases which have around 1million data point per case. So when I tried to export the interesting features from detection measurement, it couldn’t cope to transfer 150 cases at the same time with csv format.
When we look at the histogram from the detection measuremnt, we can see below which shows mean and std. dev value. So I wonder if we can script to export these mean and std. Dev values from certain features of detection measurement in order to export the avearged and STD values from 150 cases.
Thanks a lot
I believe you can get the same numbers through scripting with the following:
import qupath.lib.analysis.stats.Histogram
var pathObjects = getDetectionObjects()
var columnName = "Cell: DAPI min" // Change this to your column of interest
var stats = Histogram.getMeasurementValues(pathObjects, columnName);
var avg = 0
for (int i=0; i < stats.length; i++) {
avg += stats[i]
}
avg /= stats.length
var sd = 0
for (int i=0; i < stats.length; i++) {
sd = sd + Math.pow(stats[i] - avg, 2);
}
print "Mean: " + avg
print "Std.Dev: " + Math.sqrt(sd/stats.length)
Hi, @melvingelbard
It has worked nicely.
Now I wonder if we can also adopt some of script from the exporting detection measurement in order to export this eman and std. dev value from a few features and combine all the cases’ data into one csv file. Thanks for your help again.
import qupath.lib.analysis.stats.Histogram
var pathObjects = getDetectionObjects()
var columnName = "H3342: Cell: Mean" // Change this to your column of interest
var stats = Histogram.getMeasurementValues(pathObjects, columnName);
var avg = 0
for (int i=0; i < stats.length; i++) {
avg += stats[i]
}
avg /= stats.length
var sd = 0
for (int i=0; i < stats.length; i++) {
sd = sd + Math.pow(stats[i] - avg, 2);
}
print "Mean: " + avg
print "Std.Dev: " + Math.sqrt(sd/stats.length)
///how can I combine this script below to the above?
import qupath.lib.gui.tools.MeasurementExporter
import qupath.lib.objects.PathCellObject
// Get the list of all images in the current project
def project = getProject()
def imagesToExport = project.getImageList()
// Separate each measurement value in the output file with a tab ("\t")
def separator = "\t"
// Choose the columns that will be included in the export
// Note: if 'columnsToInclude' is empty, all columns will be included
def columnsToInclude = new String[]{"Name", "Class", "H3342: Cell: Mean"}
// Choose the type of objects that the export will process
// Other possibilities include:
// 1. PathAnnotationObject
// 2. PathDetectionObject
// 3. PathRootObject
// Note: import statements should then be modified accordingly
def exportType = PathCellObject.class
// Choose your *full* output path
def outputPath = "M:/measurements.tsv"
def outputFile = new File(outputPath)
// Create the measurementExporter and start the export
def exporter = new MeasurementExporter()
.imageList(imagesToExport) // Images from which measurements will be exported
.separator(separator) // Character that separates values
.includeOnlyColumns(columnsToInclude) // Columns are case-sensitive
.exportType(exportType) // Type of objects to export
.exportMeasurements(outputFile) // Start the export process
print "Done!"
But the measurement exporter will have a row for each detection (‘one row’ for one detection), whereas the average and standard deviation is for all the detections altogether (‘one row’ for all detections).
Hi, @melvingelbard
Thanks for your reply. I think I have made a confusion.
I would like to export this mean and std dev. values from 150 cases to one csv files as a batch.
would that be possible to be done via scriptting?
Thanks again
Ah! I understand, the following should then work (Run > Run for project):
import qupath.lib.analysis.stats.Histogram
var pathObjects = getDetectionObjects()
var columnName = "Cell: Min caliper" // Change this to your column of interest
var stats = Histogram.getMeasurementValues(pathObjects, columnName);
var avg = 0
for (int i=0; i < stats.length; i++) {
avg += stats[i]
}
avg /= stats.length
var sd = 0
for (int i=0; i < stats.length; i++) {
sd = sd + Math.pow(stats[i] - avg, 2);
}
print "Mean: " + avg
print "Std.Dev: " + Math.sqrt(sd/stats.length)
var separator = "\t"
var path = "C:/Users/Mel/Desktop/measurements.csv" // Change this to your output path!
File file = new File(path)
var exists = file.exists()
file.withWriterAppend { fw ->
if (!exists)
fw.writeLine(["Image name", "Average", "Standard deviation"].join(separator))
fw.append([getProjectEntry().getImageName(), avg, sd].join(separator))
fw.append(System.getProperty("line.separator"))
}
print "Done!"
Hi, @melvingelbard
Can I ask you a question, please?
the script was worked in the dummy run in v.0.2.1.
But in the real analysis under v.0.2.3, it says
INFO: Mean: NaN
INFO: Std.Dev: NaN
INFO: Done!
I run the same script.
I would appreciate your help.
Thanks a lot
Not sure, but the names do have to be exactly correct. If you have renamed your measurements even slightly, it will not work.
I adjusted the script to write to the Project folder by default, and to work for N number of measurements.
// Source: https://forum.image.sc/t/script-for-averaging-detection-measurement-data/48805/9?u=research_associate
columnName = ["CK: Cell: Mean", "CD68: Cell: Mean"] // Change this to your column of interest
imageName = getProjectEntry().getImageName()
//Save to the project folder
var path = buildFilePath(PROJECT_BASE_DIR, "Detection Summary measurements.csv")
//print "Mean: " + avg
//print "Std.Dev: " + Math.sqrt(sd/stats.length)
var pathObjects = getDetectionObjects()
var separator = ","
File file = new File(path)
header = "Image name,"
var exists = file.exists()
file.withWriterAppend { fw ->
if (!exists){
columnName.each{
header =header+[it+" Average", it+" Standard deviation"].join(separator)+separator
}
fw.writeLine(header)
}
dataLine = getProjectEntry().getImageName()+","
columnName.each{
stats = Histogram.getMeasurementValues(pathObjects, it);
var avg = 0
for (int i=0; i < stats.length; i++) {
avg += stats[i]
}
avg /= stats.length
var sd = 0
for (int i=0; i < stats.length; i++) {
sd = sd + Math.pow(stats[i] - avg, 2);
}
sd = Math.sqrt(sd/stats.length)
dataLine = dataLine + [avg, sd].join(separator)+separator
}
fw.append(dataLine)
fw.append(System.getProperty("line.separator"))
}
print "Done!"
import qupath.lib.analysis.stats.Histogram
Also I went comma separated rather than tab separated, because US Excel did not like the tabs by default 
Dear @Research_Associate
I have just tried the script. It has worked.
But the standard deviation value does not match with the histogram.
This is the export of mean and STD. The mean data matches as 4254.083 but STD does not.
Hope to hear from you.
Thanks for your help
I suspect something is up with the STD Dev calculation then, but I did not check the math.
It looks like
Math.sqrt(
was not added in front of the SD calculation, just in front of the printed text, so adding that in should fix it.
I have adjusted my code above to add in the square root.
Dear @Research_Associate
Thank you so much for your help.
I got it and it works nicely.
You are amazing!!!
Thanks again,
Well, the bulk of the work was @melvingelbard, I just swooped in at the end and adjusted a few things!
Can I ask you one more question, please?
Is it possible to calculate mean and STD by parent annotation within same detection features? For example, we have column name “CK: Cell: Mean” which has got 2 different classes such as tumor, stroma.
I would appreciate your help.
Thanks a lot