RareCellFetcher- a tool for annotating rare cells in QuPath

Here’s a tool to help find and annotate rare cells in a tissue section for training a machine learning classifier.

One of the difficulties in building a classifier for rare cells is how long it can take to search for and annotate a sufficient number of example cells. With this, you can start with a preliminary object classifier with just a few examples of your classes. This script searches for cells that it thinks are your class of interest based on your current classifier and enables you to easily label them with just a single mouse click or key stroke. Overall, this is a much faster way to generate training data. This is inspired by the “fetch” command in Cell Profiler Analyst.

RareCellFetcher was written in collaboration with Michael Nelson, a graduate of our 2019 QuPath workshop in San Diego and all around helpful person.

HOW TO USE:

  1. Detect cells, perform whatever measurements you need, and draw annotation objects around a few cells of each class. Then train an Object Classifier (see here: https://github.com/qupath/qupath/wiki/Classifying-objects). Leave the classifier window open.

  2. Run this script. It will bring up this window, which will automatically populate with a list of all of the valid classes:
    image

  3. Select which class you’d like it to pull examples of (in the dropdown labeled “Fetch which class”). It will select a random cell among those that are currently labeled as that class. It will avoid ones currently inside a labeled annotation.

  4. Select which class to assign that cell to in the list on the left. You can use the up and down arrows to switch between classes. Then, click “Assign” or press A to make a labeled annotation object over that cell. It will automatically move to a new cell. Or, you can click “Skip” (or press S) to move on to a different cell without labeling this one.
    image

  5. Update the classifier occasionally to see your improved results. Or, you can auto-update with every new assignment, though updating the classifier is typically slower than fetching the next cell.

  6. If you make a mistake with an assignment, you can click “Back” or press B to go back through previous cells and alter their annotation. If you accidentally lose track of the current cell, click “Highlight” or press H to highlight the cell and bring it to the center of your screen.

  7. Once you are happy with your classifier, you can save all of your training annotations with “Save and Remove Annotations”. A new TrainingAnnotations file will be created and all annotations that have a set class will be deleted. If you click this by accident, you can immediately press Ctrl+Z to undo. If you want to reload the annotations later, click “Reload current image annotations.”

  8. At any time, you can click “Help Guide” to bring up this explanation. :slight_smile:

Script can be found here: https://github.com/saramcardle/Image-Analysis-Scripts/blob/master/QuPath%20Groovy%20Scripts/QuPath%200.2.0%20m8/RareCellFetcher.groovy

or here:

/*
Function to help you annotate single, rare cells.
Run this after detecting cells and beginning to train an object classifier.
Shows you random cells that it thinks are in your desired class. You can assign them to a class,
and it automatically moves to the next cell. Update the classifier regularly to see your improved results.

Inspired by "fetch" command in Cell Profiler Analyst.

Written by Sara McArdle of the La Jolla Institute and edited by Michael Nelson, 2020.
 */
import javafx.application.Platform
import qupath.lib.gui.QuPathGUI
import qupath.lib.gui.scripting.QPEx
import javafx.scene.control.ListView
import javafx.collections.FXCollections
import javafx.scene.layout.GridPane;
import javafx.scene.control.ChoiceBox
import javafx.geometry.Insets
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.stage.Stage
import javafx.scene.control.Label
import javafx.scene.control.Button
import javafx.beans.value.ChangeListener
import javafx.geometry.HPos
import javafx.scene.input.KeyCode
import javafx.scene.control.Hyperlink
import javafx.scene.control.Alert
import javafx.scene.control.Alert.AlertType
import javafx.scene.control.ButtonType
import java.awt.Desktop

//Made a list to hold tracking information from cells that are analyzed.
//No tracking data is collected by the developers
List pastCells = []
//function to get a random cell of a desired class (interest) that isn't already annotated
//chooses a cell, copies it's ROI to an annotation
def getNextCell(interest, classifications, pastCells){
    def cells= QPEx.getCellObjects()
    def celltypes=cells.findAll{it.getPathClass()==getPathClass(interest)}
    def unassigned=celltypes.findAll{it.getParent().getPathClass()==null}
    
    Random rnd = new Random()
    def selection=unassigned[rnd.nextInt(unassigned.size())]

    def roi=selection.getROI()
    tempAnnot = PathObjects.createAnnotationObject(roi)

    QPEx.getCurrentHierarchy().selectionModel.setSelectedObject(tempAnnot,false)
    QPEx.getCurrentViewer().setCenterPixelLocation(tempAnnot.getROI().getCentroidX(),tempAnnot.getROI().getCentroidY())
    
    //add the current cell to a list of cells so that the user can backtrack if something was incorrectly skipped or assigned
    pastCells << tempAnnot
}

//get existing classes
def cells= QPEx.getCellObjects()
if (cells.size()==0) {
    print("Must have detection objects")
}
def classifications = new ArrayList<>(cells.collect {it.getPathClass()} as Set)
if (classifications.size()<2){
    print("Must have cells assigned to at least 2 classes")
}

def classStr=classifications.collect{it.toString()}
def classObs= FXCollections.observableArrayList(classStr)

//list view for assigning classes (and label)
ListView<String> classListView = new ListView<String>(classObs)
if (classStr.size()<6) {
    classListView.setPrefHeight((classStr.size() * 24) + 4)
} else {
    classListView.setPrefHeight((6 * 24) + 4)
}
Label assignmentLabel = new Label("Assign to which class:")

//drop down for choosing which class to fetch (and label)
ChoiceBox classChoiceBox = new ChoiceBox(classObs)
classChoiceBox.setMaxWidth(Double.MAX_VALUE)
Label fetchLabel = new Label("Fetch which class: \n\n\n")

Hyperlink helpLink = new Hyperlink("Help guide")
helpLink.setOnAction{e->
    if (Desktop.isDesktopSupported()){
        Desktop.getDesktop().browse(new URI("https://forum.image.sc/t/rarecellfetcher-a-tool-for-annotating-rare-cells-in-qupath/33654"))
    }
}

//Export and remove annotations. Needed to move this up in the code so that the assign button could trigger it going active
Button saveButton = new Button("Save and Remove Annotations")
saveButton.setOnAction {e ->

    //Need to add an alert of some sort here, since right now if you double click you will overwrite
    //with an empty file and delete everything.
    Alert alert = new Alert(AlertType.CONFIRMATION, "All classified annotations will be saved\nand then deleted. Proceed?", ButtonType.YES, ButtonType.NO, ButtonType.CANCEL);
    alert.showAndWait();

    if (alert.getResult() == ButtonType.YES) {


        toRemove = getAnnotationObjects().findAll{classifications.contains(it.getPathClass())}
        toExport = toRemove.collect {new qupath.lib.objects.PathAnnotationObject(it.getROI(), it.getPathClass())}
        name = getProjectEntry().getImageName()+".training"
        dirpath = buildFilePath(PROJECT_BASE_DIR, 'TrainingAnnotations')
        mkdirs(dirpath)
        path = buildFilePath(PROJECT_BASE_DIR, 'TrainingAnnotations', name)
    

        new File(path).withObjectOutputStream {
            it.writeObject(toExport)
        }
    
        removeObjects(toRemove,true)
        resolveHierarchy()
    }
    
}
saveButton.setMaxWidth(Double.MAX_VALUE)
saveButton.setDisable(true) //start disabled until a class is fetched the first time


//Restore removed annotations
Button reloadButton = new Button("Reload current image annotations")
reloadButton.setOnAction {e ->

    Alert alert = new Alert(AlertType.CONFIRMATION, "All annotations from the save file will be placed \nback into the image! Proceed?", ButtonType.YES, ButtonType.NO, ButtonType.CANCEL);
    alert.showAndWait();

    if (alert.getResult() == ButtonType.YES) {

        name = getProjectEntry().getImageName()+".training"
        dirpath = buildFilePath(PROJECT_BASE_DIR, 'TrainingAnnotations')
        path = buildFilePath(PROJECT_BASE_DIR, 'TrainingAnnotations', name)
        //I was not able to find a cleaner way to do this without throwing an error if the file did not exist.
        File file = new File(path)
        if (file.exists()){
            file.withObjectInputStream {
            toReload = it.readObject()
            addObjects(toReload)
            resolveHierarchy()
        }
    

        }
    }
}
reloadButton.setMaxWidth(Double.MAX_VALUE)



//Set up a button to assign a class
Button assignButton = new Button("Assign (A)")
assignButton.setOnAction {e ->
    if (classListView.selectionModel.selectedItem) { //only assign if a class is chosen
        tempAnnot.setPathClass(getPathClass(classListView.selectionModel.selectedItem))
        addObjects(tempAnnot)
        getCurrentHierarchy().insertPathObject(tempAnnot, true)
        getNextCell(classChoiceBox.selectionModel.getSelectedItem().toString(), classifications, pastCells)
        saveButton.setDisable(false)
    }
}
assignButton.setMaxWidth(Double.MAX_VALUE)
assignButton.setDisable(true) //start disabled until a class is fetched the first time

//skip button for if you do not like a cell and do not want to annotate it
Button skipButton = new Button("Skip (S)")
skipButton.setOnAction {e ->
    getNextCell(classChoiceBox.selectionModel.getSelectedItem().toString(), classifications, pastCells)
}
skipButton.setMaxWidth(Double.MAX_VALUE)
skipButton.setDisable(true) //start disabled until a class is fetched the first time

//back up button for if user made a mistake with a cell
Button backButton = new Button("Back (B)")
backButton.setOnAction {e ->
    if(pastCells.size() > 1){
        tempAnnot=pastCells[pastCells.size()-2]
        QPEx.getCurrentHierarchy().selectionModel.setSelectedObject(tempAnnot,false)
        pastCells.remove(pastCells[pastCells.size()-1])
    }
}
backButton.setMaxWidth(Double.MAX_VALUE)
backButton.setDisable(true) //start disabled until a class is fetched the first time

//highlight button in case you "un-select" a cell and forget which one is being shown
Button highlightButton = new Button("Highlight Cell (H)")
highlightButton.setOnAction {e ->
    getCurrentHierarchy().selectionModel.setSelectedObject(tempAnnot,true)
    QPEx.getCurrentViewer().setCenterPixelLocation(tempAnnot.getROI().getCentroidX(),tempAnnot.getROI().getCentroidY())
}
highlightButton.setMaxWidth(Double.MAX_VALUE)
highlightButton.setDisable(true)

//when a class is chosen, fetch a cell and enable all the rest of the buttons
classChoiceBox.getSelectionModel().selectedItemProperty().addListener({v,o,n->
    getNextCell(n.toString(), classifications, pastCells)
    assignButton.setDisable(false)
    skipButton.setDisable(false)
    highlightButton.setDisable(false)
    backButton.setDisable(false)

} as ChangeListener)

//put all buttons into a grid pane
GridPane gridPane = new GridPane();
gridPane.setMinSize(100, 120);
gridPane.setPadding(new Insets(10, 10, 10, 10));
gridPane.setVgap(5);
gridPane.setHgap(10);
gridPane.setAlignment(Pos.CENTER);

//gridPane.add is read (object,Column,Row) and is 0-based
gridPane.add(fetchLabel,0,0)
gridPane.setHalignment(fetchLabel, HPos.RIGHT)
gridPane.add(classChoiceBox,1,0)
gridPane.add(assignmentLabel,0,1)
gridPane.setHalignment(assignmentLabel, HPos.CENTER)
gridPane.add(helpLink,1,1)
gridPane.setHalignment(helpLink, HPos.RIGHT)
gridPane.add(classListView,0,2,1,4)
gridPane.add(assignButton,1,2)
gridPane.add(skipButton,1,3)
gridPane.add(backButton,1,4)
gridPane.add(highlightButton,1,5)
gridPane.add(saveButton,0,6,2,1)
gridPane.add(reloadButton,0,8,2,1) 

gridPane.setOnKeyPressed{event ->
    KeyCode key = event.getCode()
    if(key.equals(KeyCode.S)){
        skipButton.fire();
    }
    if(key.equals(KeyCode.A)){
        assignButton.fire();
    }
    if(key.equals(KeyCode.H)){
        highlightButton.fire();
    }
    if(key.equals(KeyCode.B)){
        backButton.fire();
    }
}
//show the GUI
Platform.runLater { //something about threading I do not understand. Copied from Pete.
    def stage = new Stage()
    stage.initOwner(QuPathGUI.getInstance().getStage())
    stage.setScene(new Scene(gridPane))
    stage.setTitle("Fetch single cells to annotate")
    stage.show()
}
4 Likes

Dawwwww. Also, it was great to help out on this, and I learned quite a bit!

3 Likes

How can we accommodate people who prefer cells medium or medium-well? :crazy_face: :laughing: Great work guys! :grinning:

3 Likes

The important thing is to make sure the classifications are “well done!”

Ack… cough hack wheeeeeze

1 Like

A small variant that I will definitely not call CellRoulette, which allows the user to search through cells within classified annotations (multiplexing, etc.)