Export annotations in parallel (AI -> Export training regions functionality in groovy)

Hi! An extension that is retired in 0.2.0+ versions AI -> Export training regions had an ability to export images and annotation labels in parallel. The documentation of the 0.2.3 version highlights the groovy scripts as an alternative, e.g.: Full labeled image. I would like to reproduce the functionality of the retired extension and to export images with annotation labels in parallel ideally.

Here goes a simplified script for a sequential export compiled from the documentation and other other questions:

import qupath.lib.images.servers.LabeledImageServer
import qupath.lib.objects.classes.PathClassFactory
import qupath.lib.objects.classes.PathClass

// Get a current project and all the images it consists of
def project = getProject()
def imageList = project.getImageList()

// Define output path (relative to project)
def outputDir = buildFilePath(PROJECT_BASE_DIR, 'export')
mkdirs(outputDir)

// Set downsample to 1 for a full resolution
double downsample = 1

// Find out all labels accross project images automatically and draw in arbitrary order.
Map<PathClass, Integer> labels = new LinkedHashMap<>();
int label = 1
for (entry in imageList) {
    for (def annotation : entry.readHierarchy().getAnnotationObjects()) {
        def pathClass = annotation.getPathClass();
        if (pathClass != null && !labels.containsKey(pathClass))
            labels.put(pathClass, label++);
    }
}
print 'Annotation labels: ' + labels

// Process each image from the project
for (entry in imageList) {
    def imageData = entry.readImageData()

    def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())
    def path_image = buildFilePath(outputDir, name + ".tif")
    def path_mask = buildFilePath(outputDir, name + "-mask.png")

    // Save the image
    def server = getCurrentServer()
    // Write the full image downsampled by a factor of downsample
    def requestFull = RegionRequest.createInstance(server, downsample)
    print 'Saving image: ' + path_image
    writeImageRegion(server, requestFull, path_image)

    // Save the mask
    // Create an ImageServer where the pixels are derived from annotations
    def labelServer = new LabeledImageServer.Builder(imageData)
      .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
      .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
      .addLabels(labels)  // Choose output labels (the order matters!)
      .multichannelOutput(false) // If true, each label refers to the channel of a multichannel binary image (required for multiclass probability)
      .build()

    print 'Saving mask: ' + path_mask
    // Write the mask image
    writeImage(labelServer, path_mask)
}

Now, when a sequential script runs, I would like to export images in parallel.

  • GParsPool imports are not available in the groovy script.
  • I found an old gist where .parallelStream().forEach is used.
    When I try to wrap the last loop into parallelStream:
imageList.parallelStream().forEach { entry ->
    def imageData = entry.readImageData()
    ...
}

I get ERROR: Script error: null which is completely unclear how to debug.

My questions are:

  • In case I’m reinventing the wheel, point me, please, to a port of AI -> Export training regions into a groovy script.
  • How can I improve my script to write images in parallel?

Yes, debug methods arising from streams and closures aren’t very informative. From looking at the script, my guess is that the line

def server = getCurrentServer()

should be replaced by

def server = imageData.getServer()
3 Likes

Thank you @petebankhead! Worked like a charm!

Here’s the full script for anyone who would like to simulate Export training regions in groovy:

import qupath.lib.images.servers.LabeledImageServer
import qupath.lib.objects.classes.PathClassFactory
import qupath.lib.objects.classes.PathClass
import com.google.gson.GsonBuilder

// Get a current project and all the images it consists of
def project = getProject()
def imageList = project.getImageList()

// Define output path (relative to project)
def outputDir = buildFilePath(PROJECT_BASE_DIR, 'export')
mkdirs(outputDir)

// --------------------------------------------------------------------------

// A downsample will be derived from a requested pixel size in µm if the value of requestedPixelSize is > 0.
// Define output resolution in calibrated units (e.g. µm if available)
double requestedPixelSize = 1.0
// Or set the requestedPixelSize <= 0 and define manually how much to downsample during the export
double downsample = -1

// Derive requestedPixelSize form downsample or downsample from requestedPixelSize basing on a resolution of the current image.
// We suppose all images in the project are with the same resolution.
def currentImage = getCurrentImageData()
double pixelSize = currentImage.getServer().getPixelCalibration().getAveragedPixelSize()
if (requestedPixelSize > 0) {
    downsample = requestedPixelSize / pixelSize
} else {
    requestedPixelSize = downsample * pixelSize
}
def downsample_string = String.format("%.2f", downsample)
print 'Using downsample: ' + downsample_string

// --------------------------------------------------------------------------

// Define user labels if a union of classes is required or an order of drawing matters.
def user_labels = [
    background: 1,
    other_tissue: 2,
    healthy_tissue: 3
]
// Convert user defined labels into PathClasses
def labels = [:]
user_labels.each { k, v ->
    labels.put(PathClassFactory.getPathClass(k), v)
}

// Or find out all labels across project images automatically and draw in an arbitrary order.
/*
Map<PathClass, Integer> labels = new LinkedHashMap<>();
int label = 1
//labels.put(PathClassFactory.getPathClassUnclassified(), label++);
for (entry in imageList) {
    for (def annotation : entry.readHierarchy().getAnnotationObjects()) {
        def pathClass = annotation.getPathClass();
        if (pathClass != null && !labels.containsKey(pathClass))
            labels.put(pathClass, label++);
    }
}
print 'Annotation labels: ' + labels
*/

// --------------------------------------------------------------------------

// Save metadata in a json format
def gson = new GsonBuilder()
    .setPrettyPrinting()
    .create()
class Annotation {
    Double requestedPixelSizeMicrons
    Double downsample
    Classification[] classifications
}
class Classification {
    String name
    Integer label
}
def classifications = []
labels.each {k, v -> 
    classifications.add(
        new Classification(
            name: "$k",
            label: v
        )
    )
}
annotation = new Annotation(
    requestedPixelSizeMicrons: requestedPixelSize,
    downsample: downsample,
    classifications: classifications
)
print "Saving metadata: " + outputDir + "/training.json"
def writer = new FileWriter(outputDir + "/training.json")
gson.toJson(annotation, writer);
writer.close()

// --------------------------------------------------------------------------

// Process images in parallel
imageList.parallelStream().forEach { entry ->
    def imageData = entry.readImageData()

    def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())
    def path_image = buildFilePath(outputDir, name + "-" + downsample_string + ".tif")
    def path_mask = buildFilePath(outputDir, name + "-" + downsample_string + "-mask.png")
    
    // Save the image
    def server = imageData.getServer()
    // Write the full image downsampled by a factor of downsample
    def requestFull = RegionRequest.createInstance(server, downsample)
    print 'Saving image: ' + path_image
    writeImageRegion(server, requestFull, path_image)
    
    // Save the mask
    // Create an ImageServer where the pixels are derived from annotations
    def labelServer = new LabeledImageServer.Builder(imageData)
      .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
      .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
      .addLabels(labels)  // Choose output labels (the order matters!)
      .multichannelOutput(false) // If true, each label refers to the channel of a multichannel binary image (required for multiclass probability)
      .build()
    
    print 'Saving mask: ' + path_mask
    // Write the mask image
    writeImage(labelServer, path_mask)
    
}

/*
// To debug take a small slice of images and run the script sequentially
for (entry in imageList[0..1]) {
    def imageData = entry.readImageData()
    ...
}
*/

print 'The script has completed'
1 Like