TileExporter using bounding box

[Mike_Nelson]

Hi all

I know this has come up before in a few places, but I was creating a quick importer to pair with the TileExporter function when it became apparent that the exporter was exporting a lot more tiles than I expected (full scripts below).

    new TileExporter(imageData)
        .downsample(downsample)     // Define export resolution
        .imageExtension('.png')     // Define file extension for original pixels (often .tif, .jpg, '.png' or '.ome.tif')
        .tileSize(224)              // Define size of each tile, in pixels
        //.labeledServer(labelServer) // Define the labeled image server to use (i.e. the one we just built)
        .annotatedTilesOnly(true)  // If true, only export tiles if there is a (labeled) annotation present
        //.includePartialTiles(false)
        .overlap(0)                // Define overlap, in pixel units at the export resolution
        .writeTiles(pathOutput)     // Write tiles to the specified directory
    removeObject(it, true)
    print 'Done!'

For an object like the above, I would have expected the tiles to roughly follow the outline of the annotation rather than the bounding box. That is not what happens, as when I re-create tile detections based off of the exported images, I get the following.

I think the last I recall, the recommendation was to use Python and a labeled image to verify the tiles overlapped significantly enough with the annotation, but I was expecting to only have to deal with X% overlap (the group is fine with any overlap, but there needs to be some overlap), not the 0% overlap in many of the tiles.

How to export only areas within annotations as whole picture or tiles

@sophia1 the main ways to export images (possibly including annotations) are summarized at https://qupath.readthedocs.io/en/latest/docs/advanced/exporting_images.html https://qupath.readthedocs.io/en/latest/docs/advanced/exporting_annotations.html The main thing to decide is whether QuPath should do the tiling or not… which may depend partially upon whether your regions are too big to export in one go. To export a rectangle corresponding to a single annotation, use // Write the region of th…

I wanted to make sure I had not missed anything new that could streamline the process or remove tiles with zero annotation overlap.
I also noticed that even with the annotatedTilesOnly set to true, I was getting tiles for my tumor, stroma, and unclassified annotations. I was able to work around that fairly easily, though, by first removing them, and then re-adding such annotations after exporting the tiles.

Script to export tiles to sub-folders based on the class of the annotation.

import qupath.lib.images.servers.LabeledImageServer

def imageData = getCurrentImageData()

// Define output path (relative to project)
def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())


// Define output resolution
double requestedPixelSize = 10.0

// Convert to downsample
double downsample = 1.0 //requestedPixelSize / imageData.getServer().getPixelCalibration().getAveragedPixelSize()
/*
// Create an ImageServer where the pixels are derived from annotations
def labelServer = new LabeledImageServer.Builder(imageData)
    .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
    .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
    .addLabel('Other', 1)      // Choose output labels (the order matters!)
    .addLabel('Islet', 2)
    .multichannelOutput(false)  // If true, each label is a different channel (required for multiclass probability)
    .build()
*/
// Create an exporter that requests corresponding tiles from the original & labeled image servers
annotations = getAnnotationObjects()
print annotations
removeObjects(annotations, true)

annotations.each{
    if(it.getPathClass() == null){return}
    addObject(it)

    className = it.getPathClass().toString()
    pathOutput = buildFilePath(PROJECT_BASE_DIR, 'tiles', name, className)
    mkdirs(pathOutput)
    new TileExporter(imageData)
        .downsample(downsample)     // Define export resolution
        .imageExtension('.png')     // Define file extension for original pixels (often .tif, .jpg, '.png' or '.ome.tif')
        .tileSize(224)              // Define size of each tile, in pixels
        //.labeledServer(labelServer) // Define the labeled image server to use (i.e. the one we just built)
        .annotatedTilesOnly(true)  // If true, only export tiles if there is a (labeled) annotation present
        //.includePartialTiles(false)
        .overlap(0)                // Define overlap, in pixel units at the export resolution
        .writeTiles(pathOutput)     // Write tiles to the specified directory
    removeObject(it, true)
    print 'Done!'
}
addObjects(annotations)
fireHierarchyUpdate()

Script to create Detection tiles based on the exported images. Classification taken from folder name

def imageData = getCurrentImageData()

// Define output path (relative to project)
def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())

basePath = buildFilePath(PROJECT_BASE_DIR, 'tiles', name)
baseDir = new File(basePath);
files = baseDir.listFiles();
classes = []
newTiles = []
files.each{ classes<<it.getName().toString()}
print classes
classes.each{c->
    tempPath= buildFilePath(PROJECT_BASE_DIR, 'tiles', name, c)
    dir = new File(tempPath);
    files = dir.listFiles();
    files.each{f->
        toParse = f.getName()
        subString = (toParse =~ /\[(.+)\]/).findAll()*.first()
        //extract the X and Y coordinates and tile size
        s = (subString =~ /\d+/).findAll()
        x = Double.parseDouble(s[0])
        y = Double.parseDouble(s[1])
        size = Double.parseDouble(s[2])
        def roi = new RectangleROI(x,y,size,size, ImagePlane.getDefaultPlane())
        newTiles << PathObjects.createDetectionObject(roi, getPathClass(c))
    }
}

//pathOutput = buildFilePath(PROJECT_BASE_DIR, 'tiles', name, className)
/*
    double x = Double.parseDouble(xy[0])
    double y = Double.parseDouble(xy[1])

    def roi = new RectangleROI(x-size/2,y-size/2,size,size, ImagePlane.getDefaultPlane())
    
    newTiles << PathObjects.createDetectionObject(roi)
*/    
addObjects(newTiles)
resolveHierarchy()
import qupath.lib.regions.ImagePlane
import qupath.lib.roi.RectangleROI;

[Mike_Nelson]

Here is a hacky work-around within QuPath to delete files that have zero overlap with the annotation being used to generate them.
A Python script was posted here.

Be warned, this is not very efficient, probably better to do it in Python if you can code there

//Primary "stuff to change" is in the labelServer - you will need to adjust to your classes.
//Otherwise, creates a "tiles" folder in the project folder.
//Within the "tiles" folder there will be one folder per class with the exports from that class.

def imageData = getCurrentImageData()

// Define output path (relative to project)
def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())


// Convert to downsample
double downsample = 1.0 //requestedPixelSize / imageData.getServer().getPixelCalibration().getAveragedPixelSize()

// Create an ImageServer where the pixels are derived from annotations
def labelServer = new LabeledImageServer.Builder(imageData)
    .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
    .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
    .addLabel('Tumor', 1)      // Choose output labels (the order matters!)
    .addLabel('Stroma', 2)
    .multichannelOutput(false)  // If true, each label is a different channel (required for multiclass probability)
    .build()

// Create an exporter that requests corresponding tiles from the original & labeled image servers
annotations = getAnnotationObjects()
print annotations
removeObjects(annotations, true)

annotations.each{
    if(it.getPathClass() == null){return}
    addObject(it)

    className = it.getPathClass().toString()
    pathOutput = buildFilePath(PROJECT_BASE_DIR, 'tiles', name, className)
    mkdirs(pathOutput)
    new TileExporter(imageData)
        .downsample(downsample)     // Define export resolution
        .imageExtension('.png')     // Define file extension for original pixels (often .tif, .jpg, '.png' or '.ome.tif')
        .tileSize(224)              // Define size of each tile, in pixels
        .labeledServer(labelServer) // Define the labeled image server to use (i.e. the one we just built)
        .annotatedTilesOnly(true)  // If true, only export tiles if there is a (labeled) annotation present
        //.includePartialTiles(false)
        .overlap(0)                // Define overlap, in pixel units at the export resolution
        .writeTiles(pathOutput)     // Write tiles to the specified directory
    removeObject(it, true)
    print 'Done!'
}
addObjects(annotations)
fireHierarchyUpdate()

//Past here is a rather hacky way of deleting any tiles that have 0 overlap with the annotation.
thing = annotations[0]
basePath = buildFilePath(PROJECT_BASE_DIR, 'tiles', name)
baseDir = new File(basePath);
files = baseDir.listFiles();
classes = []
files.each{ classes<<it.getName().toString()}
classes.each{c->
    params = new ImageJMacroRunner(getQuPath()).getParameterList()
    tempPath= buildFilePath(PROJECT_BASE_DIR, 'tiles', name, c)
    dir = new File(tempPath);
    
    files = dir.listFiles().findAll{!it.getName().toString().contains("-labelled")};
    files.each{f->
        
        f= f.toString().replaceAll('\\\\', '/')
        //print f
        //print ParameterList.getParameterListJSON(params, ' ')
        macro = 'setBatchMode(true);n="'+f+'"; m=substring(n,0,n.length()-4); m=m+"-labelled.png"; open(m); getStatistics(nPixels, mean, min, max); if (max == 0){ close();File.delete(m);File.delete(n); };'

        ImageJMacroRunner.runMacro(params, imageData, null, thing, macro)
        
    }
    macro ='selectWindow("Log"); run("Close");'
    ImageJMacroRunner.runMacro(params, imageData, null, thing, macro)
}


import qupath.imagej.gui.ImageJMacroRunner
import qupath.lib.images.servers.LabeledImageServer

For anyone like me who would be even more lost in Python, maybe this will help.

[petebankhead (Pete)]

My guess (since my memory isn’t that good) is that bounding boxes were used for efficiency, avoiding more expensive computations checking for overlap. In retrospect, the computations probably aren’t that expensive and are likely to be worth it.

Re. annotatedTilesOnly… am I right to think that you’re intuition expects this to mean that only tiles containing a classified annotation with a corresponding label should be export?

I think the trouble is that sometimes (and perhaps even often) one might want to export all tiles with one kind of classification (e.g. Region*), but not assign these any labels.

I suppose the workaround there would be to give those tiles the same label as the background… but that doesn’t necessarily give the same results (e.g. in the case of overlaps) – although perhaps it does when the labels are passed on a suitable order.

I’ll need to give that a bit more thought. I’m not sure annotationTilesOnly is just too blunt an instrument for the job, and more nuanced options are needed, or if the switch proposed above would be enough. Certainly my intuition is often confused by this option and its behavior, which can’t be a good sign.

[Mike_Nelson]

@petebankhead

Re. annotatedTilesOnly … am I right to think that you’re intuition expects this to mean that only tiles containing a classified annotation with a corresponding label should be export?

The text after it in the example code is:
.annotatedTilesOnly(false) // If true, only export tiles if there is a (classified) annotation present
I figured it would mean any classified annotation would be processed, leaving unclassified (null class) annotations un-tiled. This shows up in the TileExporter example here unrelated to the labelServer. Instead, the tileExporter was also processing an unclassified annotation I threw in as a test (users being users).

Anecdotally, the computations using JTS seem pretty fast (it turned out we needed a rectangular tile exporter, so I modified some other code to get the image below). I did not notice appreciably more time when adding in the check, though I haven’t tried it for a million tiles either.