QuPath-Python Import/Export instance segmentation masks for binary classification

Dear community,

I am trying to import and export from QuPath instance segmentation masks for binary (background or RoundObject) classification problems. Does anyone have good ideas/scripts to do this?

The underlying idea is that

  1. I want to use external instance segmentation algorithms (like CellPose or Mask R-CNN in PyTorch) to give initial mask predictions
  2. Followingly, to import these masks to QuPath to correct incorrectly segmented objects.
  3. To export these masks again from QuPath, in order to retrain my model and make it better.
  4. Repeat steps 1-3, to make annotating as fast as possible

In the end, I would like to convert my data to PNG instance segmentation masks (see for instance the photos from the FudanPed database) where the background consists of zeros, and for each new instance a new number is introduced (so object1: 1; object2: 2; etc. ). However, I am also open to convert it to JSON/GeoJSON/other formats in between.

Related posts I could find were:

I got a bit lost in the many different posts but most if not all appeared to be about semantic segmentation (though I didn’t manage to test all; I had some errors when running certain scripts).

Images of FudanPed database, with 2 persons on it (mask will appear black, but note that when reading the array, the persons have a value of 1 and 2 respectively).


An image of the masks from myself I’m trying to read (for easy visualization, download + import to QuPath, then you will see the objects)

1 Like

The most recent export is on the main docs site: Exporting annotations — QuPath 0.2.3 documentation

Importing objects from such masks can depend on the size of the images (do you need tiling?) but for simple cases, scripts like:

which is based off of Does anyone have an importing masks script corresponding to the working export tiles script - #3 by melvingelbard
might work for png/tif imports.

1 Like

The importing works amazingly, thank you very much. I’ve spent a lot of time trying to figure it out.

For the exporting, which script do you mean from the website? If I am right, the ‘individual annotations’ gives single-object cropped exports, the ‘Full labeled image’ gives semantic segmentation and the ‘Labeled tiles’ also appears to be giving semantic segmentation only (no instance segmentation, if I am correct. Though I am not sure about this last case as I am just getting zero’ed Tif images when opening it with numpy from pillow in python).

Or did you mean using one of the Shapes export formats and then converting in python? (I haven’t ventured into that option yet)

Thanks a lot for your help already!

1 Like

It sounded like you wanted the fully labeled image.
https://qupath.readthedocs.io/en/latest/docs/advanced/exporting_annotations.html#full-labeled-image

That will export either annotations or detections depending on how you set it up.

1 Like

For instance segmentation, add useUniqueLabels() to the builder (you might then need to add a filter to include only the annotations you require):

(I think we haven’t got around to adding that to the documentation yet…)

1 Like

Ah, right! Missed that. Been using the code recently, myself.
https://github.com/qupath/qupath/blob/43aad4ecda893a7eb03c30774e64da5b9547bc86/qupath-core/src/main/java/qupath/lib/images/servers/LabeledImageServer.java#L288

1 Like

You guys are amazing. I’m going to try it tomorrow morning :). Thanks a lot for such wonderful help

1 Like

Both the importer and the exporter are working amazingly.

Only change I still had to make was to have the importer import the data as annotations instead of detections; otherwise the exporter would not include them. For anyone else looking for this solution, the two scripts are also now here below.

Importer:

//QP 0.2.3 compliant
//Target a directory of PNG exports from CellPose and import those masks as detection or annotation objects into QuPath.

def directoryPath = /C:\Users\MyUserName\RestOfPathToDirectory/ // TO CHANGE

clearAllObjects()
double downsample = 1 // TO CHANGE (if needed)
ImagePlane plane = ImagePlane.getDefaultPlane()

File folder = new File(directoryPath);
File[] listOfFiles = folder.listFiles();

currentImport = listOfFiles.find{ GeneralTools.getNameWithoutExtension(it.getPath()).contains(GeneralTools.getNameWithoutExtension(getProjectEntry().getImageName())) &&  it.toString().contains(".png")}

path = currentImport.getPath()
    /*if (!path.endsWith(".png"))
        return*/
def imp = IJ.openImage(path)

print ' path'  + path

int n = imp.getStatistics().max as int
if (n == 0) {
    print 'No objects found!'
    return
}
print ' n '  + n
def ip = imp.getProcessor()
    if (ip instanceof ColorProcessor) {
        throw new IllegalArgumentException("RGB images are not supported!")
    }
def roisIJ = RoiLabeling.labelsToConnectedROIs(ip, n)

    def rois = roisIJ.collect {
        if (it == null)
            return
        return IJTools.convertToROI(it, 0, 0, downsample, plane);
    }
    rois = rois.findAll{null != it}
    
    // Convert QuPath ROIs to objects
    def pathObjects = rois.collect {
        return PathObjects.createAnnotationObject(it)//createDetectionObject(it) //switching between createAnnotationObject(it) and createDetectionObject(it) to vary in which form to import
    }
    addObjects(pathObjects)
    
resolveHierarchy()
print "Import completed"

import ij.gui.Wand
import qupath.lib.objects.PathObjects
import qupath.lib.regions.ImagePlane
import ij.IJ
import ij.process.ColorProcessor
import qupath.imagej.processing.RoiLabeling
import qupath.imagej.tools.IJTools
import java.util.regex.Matcher
import java.util.regex.Pattern

Exporter:

/**
 * Script to export instance and/or semantic segmentation masks for full images.
 *
 * See the comments to change the script between instance or semantic segmentation.
 * 
 * On the website of QuPath it is described as ' Full labeled image' exporting.
 * https://qupath.readthedocs.io/en/latest/docs/advanced/exporting_annotations.html
 * 
 * @author Pete Bankhead
 */

def imageData = getCurrentImageData()

// Define output path (relative to project)
def outputDir = buildFilePath(PROJECT_BASE_DIR, 'export')
mkdirs(outputDir)
def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())
def path = buildFilePath(outputDir, name + "-labels.png")

// Define how much to downsample during export (may be required for large images)
double downsample = 1

// Create an ImageServer where the pixels are derived from annotations
def labelServer = new LabeledImageServer.Builder(imageData)
  .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
  .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
  
  // In order to use instance segmentation, use the following line of code
  .useUniqueLabels()         // Assign labels based on instances, not classifications
  
  // In order to use semantic segmentation, use the following lines of code. Each indicating one class' pixel value.
//  .addLabel('Tumor', 1)      // Choose output labels (the order matters!)
//  .addLabel('Immune cells', 2)
//  .addLabel('Other', 3)

  .multichannelOutput(false) // If true, each label refers to the channel of a multichannel binary image (required for multiclass probability)
  .build()

// Write the image
writeImage(labelServer, path)

P.S. I’m marking the first post as the solution, but it’s of course your posts together that formed the solution. Thanks.

1 Like

Ah, you’ll be wanting the useDetections() option of the builder for that :slight_smile:

2 Likes

One more question on this topic. I am currently trying to export (preferably tiled) instance segmentation masks with >256 instances in it, preferably in PNG format. I am not succeeding.

Exporting in .tif format without tiling is working for a full image, but subsequent conversion with pillow to PNG has some problems due to a bug: Saving grayscale 16 bit PNG fails · Issue #2970 · python-pillow/Pillow · GitHub . Direct exporting as a PNG gives files which cannot be opened/visualized by pillow/standard windows Photos app (my guess would be that they are corrupted).

For tiled images it isn’t working for .tif nor PNG, returning the error “You’ve requested 777 output channels, but the maximum supported number is 256”. I did write a cutter script and a merger script for cutting bounding boxes annotations to tiles and merging again (incl. nms etc.) in python and I could extend this to masks in order to perform tiling in Python, but if there is already a working variant in QuPath then I will happily use that one.

Does either of you know how to create such (I assume 16-bit) [preferably tiled] PNG images?

When it comes to importing, I didn’t test if this already works for tiled images with >256 instance segmentations, but I would be eager to know if either of you has an answer to this.

You need to use .multichannelOutput(false) to export more than 256 different labels; then you’ll get a labelled image rather than a multichannel binary image as output (more details here).

2 Likes

Yes, it continues to give masks that cannot be opened. For exporting full images I used the script given 4 posts back (Feb 26th, 09.45h) and for tiled images:

/**
 * Script to export instance and/or semantic segmentation masks for full images.
 *
 * See the comments to change the script between instance or semantic segmentation.
 * 
 * On the website of QuPath it is described as 'Full labeled image' exporting.
 * https://qupath.readthedocs.io/en/latest/docs/advanced/exporting_annotations.html
 * 
 * @author Pete Bankhead
 */

def imageData = getCurrentImageData()

// Define output path (relative to project)
def outputDir = buildFilePath(PROJECT_BASE_DIR, 'export')
mkdirs(outputDir)
def name = GeneralTools.getNameWithoutExtension(imageData.getServer().getMetadata().getName())
def path = buildFilePath(outputDir, name + "-labels.png")

// Define how much to downsample during export (may be required for large images)
double downsample = 1

// Create an ImageServer where the pixels are derived from annotations
def labelServer = new LabeledImageServer.Builder(imageData)
  .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
  .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
  
  // In order to use instance segmentation, use the following line of code
  .useUniqueLabels()         // Assign labels based on instances, not classifications
  
  // In order to use semantic segmentation, use the following lines of code. Each indicating one class' pixel value.
//  .addLabel('Tumor', 1)      // Choose output labels (the order matters!)
//  .addLabel('Immune cells', 2)
//  .addLabel('Other', 3)

  .multichannelOutput(false) // If true, each label refers to the channel of a multichannel binary image (required for multiclass probability)
  .build()

// Write the image
writeImage(labelServer, path)

The result appears to be the same kind of an image. For the tiled image for example:

Maybe noteworthy is that within the tile, there are not >256 instances, only in the image before tiling (though I would like to also have in the tile, possibly over >256 instances).

Can you just use .tif as the output format instead? I’m not sure why you need to convert it to PNG at all.

With Python I prefer to use imageio for reading/writing rather than Pillow directly, and it has good .tif support. There’s also pypng (although I haven’t tried it).

Note that, when useUniqueLabels() is applied, the labels are applied globally (rather than per-tile) – that way objects that might overlap a tile boundary can still be identified even if they span multiple export tiles.

1 Like

.tif extension is working (for both full images and for tiled images), thanks. I will see if I can run my DL model with tif images.

I’ll have a look at imageio and pypng as well, thanks!