StarDist error Unable to load bundle:null

Hi,
I’m trying to use Stardist in Qupath on a linux HPC cluster and have asked the cluster admin to build Qupath with tensorflow from source as (https://qupath.readthedocs.io/en/latest/docs/advanced/stardist.html#building-qupath-with-tensorflow). I am getting the following error when running the script shown. I load the Qupath and tensorflow modules, are there other modules I need to make available for Qupath to call Stardist?

setImageType('FLUORESCENCE');
clearAllObjects();
createAnnotationsFromPixelClassifier("Tissue", 0.0, 0.0);

selectObjectsByClassification("Tissue");
createAnnotationsFromPixelClassifier("Tumour_Stroma", 0.0, 0.0, "SPLIT")

//Select ROI
def pathObjects = getAnnotationObjects().findAll{it.getPathClass() != getPathClass("Tissue")}


import qupath.tensorflow.stardist.StarDist2D

// Specify the model directory (you will need to change this!)
def pathModel = '/home/monkmanj/data/dsb2018_heavy_augment/'

def stardist = StarDist2D.builder(pathModel)
        .threshold(0.5)              // Probability (detection) threshold
        .channels('DAPI (DAPI)')            // Select detection channel
        .normalizePercentiles(1, 99) // Percentile normalization
        .pixelSize(0.49)              // Resolution for detection
        .cellExpansion(5.0)          // Approximate cells based upon nucleus expansion
        .cellConstrainScale(1.5)     // Constrain cell expansion using nucleus size
        .measureShape()              // Add shape measurements
        .measureIntensity()          // Add cell measurements (in all compartments)
        .includeProbability(true)    // Add probability as a measurement (enables later filtering)
        .build()

// Run detection for the selected objects
def imageData = getCurrentImageData()
//def pathObjects = getSelectedObjects()
if (pathObjects.isEmpty()) {
    Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
    return
}
stardist.detectObjects(imageData, pathObjects)

I get loops of the following error:

2020-11-17 09:24:40.310107: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310113: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310177: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310180: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310304: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310318: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310333: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310444: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310468: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310472: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310502: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310509: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310480: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310540: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310562: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310612: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310635: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310646: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310654: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310662: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310671: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310696: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310702: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310717: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310698: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310745: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310768: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310801: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310832: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.310863: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /home/monkmanj/data/dsb2018_heavy_augment/
2020-11-17 09:24:40.312067: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312134: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312265: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312428: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312435: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312488: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312505: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.312613: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.314243: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX2 AVX512F FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-17 09:24:40.316208: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316215: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316281: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316303: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316338: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316408: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.316531: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.317294: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.318015: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.318121: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.318439: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.319226: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.319299: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.320087: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.320177: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.320224: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.320464: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.322658: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.326266: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.330007: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.333428: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.336531: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-11-17 09:24:40.353153: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2020-11-17 09:24:40.385294: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385277: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385277: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385351: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385353: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385277: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385345: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385408: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385277: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385277: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385415: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385833: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.385921: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.386397: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.387128: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.387171: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.387734: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.392556: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.392594: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.393301: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.394064: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.394322: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.394375: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395089: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395250: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395263: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395265: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395692: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395981: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.395992: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-11-17 09:24:40.548181: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 237619 microseconds.
2020-11-17 09:24:40.550482: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 240150 microseconds.
2020-11-17 09:24:40.554203: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 243486 microseconds.
2020-11-17 09:24:40.555138: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 244635 microseconds.
2020-11-17 09:24:40.559848: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 249213 microseconds.
2020-11-17 09:24:40.560865: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 250163 microseconds.
2020-11-17 09:24:40.564968: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 254490 microseconds.
09:24:40.564 [ForkJoinPool.commonPool-worker-51] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.564 [ForkJoinPool.commonPool-worker-17] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.564 [ForkJoinPool.commonPool-worker-35] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.564 [ForkJoinPool.commonPool-worker-45] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.564 [ForkJoinPool.commonPool-worker-19] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.564 [ForkJoinPool.commonPool-worker-23] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.565 [ForkJoinPool.commonPool-worker-49] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
2020-11-17 09:24:40.566097: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 255425 microseconds.
09:24:40.566 [ForkJoinPool.commonPool-worker-41] [INFO ] q.t.TensorFlowOp$TensorFlowBundle - Loaded TensorFlow bundle: /home/monkmanj/data/dsb2018_heavy_augment/, (inputinput:0 [-1,-1,-1,1], output=concatenate_4/concat:0 [-1,-1,-1,33])
09:24:40.569 [ForkJoinPool.commonPool-worker-23] [ERROR] qupath.tensorflow.TensorFlowOp - Unable to load bundle: null
java.util.ConcurrentModificationException: null
        at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
        at qupath.tensorflow.TensorFlowOp.loadBundle(TensorFlowOp.java:151)
        at qupath.tensorflow.TensorFlowOp.getBundle(TensorFlowOp.java:99)
        at qupath.tensorflow.TensorFlowOp.getChannels(TensorFlowOp.java:134)
        at qupath.opencv.ops.ImageOps$Core$SequentialMultiOp.getChannels(ImageOps.java:1534)
        at qupath.opencv.ops.ImageOps$ChannelImageDataOp.getChannels(ImageOps.java:297)
        at qupath.opencv.ops.ImageOpServer.<init>(ImageOpServer.java:64)
        at qupath.opencv.ops.ImageOps.buildServer(ImageOps.java:158)
        at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:606)
        at qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:583)
        at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$1(StarDist2D.java:522)
        at qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
        at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:522)
        at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjectsImpl$4(StarDist2D.java:565)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
        at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
09:24:40.569 [ForkJoinPool.commonPool-worker-45] [ERROR] qupath.tensorflow.TensorFlowOp - Unable to load bundle: null

Interesting. I know nothing of Linux, and hopefully others will know more, but can you confirm which version of QuPath exactly you are using, and that the Tensorflow extension shows up in the Help->Installed extensions ?

I imagine this is all fine, but I want double check first.

That looks like it’ll be an unfortunate intermittent bug on the QuPath side.

There would be few ways to fix it, but they require updating the code. The easiest way is likely

where you can change

private static TensorFlowBundle loadBundle(String path) {

to become

private static synchronized TensorFlowBundle loadBundle(String path) {

Alternatively, in your script that sets the StarDist2D.builder you can include nThreads(1)… although presumably it could make things slower, so doesn’t sound very appealing for an HPC cluster.

1 Like

(post withdrawn by author, will be automatically deleted in 72 hours unless flagged)