Error message when loading my pretained model

I have just finished training my model via the example notebook 2_training.pynb via

if not quick_demo:
    model.export_TF()

now, when I try to load the model in my detection script:

import qupath.tensorflow.stardist.StarDist2D

// Specify the model directory (you will need to change this!)
def pathModel = '/Users/thiemomoellenkamp/Documents/models/stardist/TF_SavedModel'

def stardist = StarDist2D.builder(pathModel)
      .threshold(0.3)              // Prediction threshold
      .normalizePercentiles(1, 99) // Percentile normalization
      .pixelSize(0.5)              // Resolution for detection
      .build()

// Run detection for the selected objects
def imageData = getCurrentImageData()
def pathObjects = getAnnotationObjects()
if (pathObjects.isEmpty()) {
    Dialogs.showErrorMessage("StarDist", "Please select a parent object!")
    return
}s
stardist.detectObjects(imageData, pathObjects)
println 'Done!'

it gives me the following error message:

ERROR: Unable to load bundle: NodeDef mentions attr 'allowed_devices' not in Op<name=VarHandleOp; signature= -> resource:resource; attr=container:string,default=""; attr=shared_name:string,default=""; attr=dtype:type; attr=shape:shape; is_stateful=true>; NodeDef: {{node up_level_0_no_0/bias}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[up_level_0_no_0/bias]]
    at qupath.tensorflow.TensorFlowOp$TensorFlowBundle.<init>(TensorFlowOp.java:202)
    at qupath.tensorflow.TensorFlowOp.lambda$loadBundle$2(TensorFlowOp.java:151)
    at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
    at qupath.tensorflow.TensorFlowOp.loadBundle(TensorFlowOp.java:151)
    at qupath.tensorflow.TensorFlowOp.getBundle(TensorFlowOp.java:99)
    at qupath.tensorflow.TensorFlowOp.getChannels(TensorFlowOp.java:134)
    at qupath.opencv.ops.ImageOps$Core$SequentialMultiOp.getChannels(ImageOps.java:1534)
    at qupath.opencv.ops.ImageOps$DefaultImageDataOp.getChannels(ImageOps.java:215)
    at qupath.opencv.ops.ImageOpServer.<init>(ImageOpServer.java:64)
    at qupath.opencv.ops.ImageOps.buildServer(ImageOps.java:158)
    at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:606)
    at qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:583)
    at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$1(StarDist2D.java:522)
    at qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
    at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:522)
    at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjectsImpl$4(StarDist2D.java:565)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
    at qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:565)
    at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$0(StarDist2D.java:511)
    at qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
    at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:511)
    at qupath.tensorflow.stardist.StarDist2D$detectObjects$0.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
    at Script2.run(Script2.groovy:20)
    at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
    at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155)
    at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:926)
    at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:859)
    at qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:782)
    at qupath.lib.gui.scripting.DefaultScriptEditor$2.run(DefaultScriptEditor.java:1271)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
ERROR: Unable to load bundle: NodeDef mentions attr 'allowed_devices' not in Op<name=VarHandleOp; signature= -> resource:resource; attr=container:string,default=""; attr=shared_name:string,default=""; attr=dtype:type; attr=shape:shape; is_stateful=true>; NodeDef: {{node up_level_0_no_0/bias}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[up_level_0_no_0/bias]]
    at qupath.tensorflow.TensorFlowOp$TensorFlowBundle.<init>(TensorFlowOp.java:202)
    at qupath.tensorflow.TensorFlowOp.lambda$loadBundle$2(TensorFlowOp.java:151)
    at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
    at qupath.tensorflow.TensorFlowOp.loadBundle(TensorFlowOp.java:151)
    at qupath.tensorflow.TensorFlowOp.getBundle(TensorFlowOp.java:99)
    at qupath.tensorflow.TensorFlowOp.getChannels(TensorFlowOp.java:134)
    at qupath.opencv.ops.ImageOps$Core$SequentialMultiOp.getChannels(ImageOps.java:1534)
    at qupath.opencv.ops.ImageOps$DefaultImageDataOp.getChannels(ImageOps.java:215)
    at qupath.opencv.ops.ImageOpServer.<init>(ImageOpServer.java:64)
    at qupath.opencv.ops.ImageOps.buildServer(ImageOps.java:158)
    at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:606)
    at qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:583)
    at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$1(StarDist2D.java:522)
    at qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
    at qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:522)
    at qupath.tensorflow.stardist.StarDist2D.lambda$detectObjectsImpl$4(StarDist2D.java:565)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
ERROR: NullPointerException at line 19: null

ERROR: qupath.tensorflow.TensorFlowOp.getChannels(TensorFlowOp.java:134)
    qupath.opencv.ops.ImageOps$Core$SequentialMultiOp.getChannels(ImageOps.java:1534)
    qupath.opencv.ops.ImageOps$DefaultImageDataOp.getChannels(ImageOps.java:215)
    qupath.opencv.ops.ImageOpServer.<init>(ImageOpServer.java:64)
    qupath.opencv.ops.ImageOps.buildServer(ImageOps.java:158)
    qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:606)
    qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:583)
    qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$1(StarDist2D.java:522)
    qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
    qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:522)
    qupath.tensorflow.stardist.StarDist2D.lambda$detectObjectsImpl$4(StarDist2D.java:565)
    java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
    java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    java.base/java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
    java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
    java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
    java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
    java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
    java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
    qupath.tensorflow.stardist.StarDist2D.detectObjectsImpl(StarDist2D.java:565)
    qupath.tensorflow.stardist.StarDist2D.lambda$detectObjects$0(StarDist2D.java:511)
    qupath.tensorflow.stardist.StarDist2D.runInPool(StarDist2D.java:548)
    qupath.tensorflow.stardist.StarDist2D.detectObjects(StarDist2D.java:511)
    qupath.tensorflow.stardist.StarDist2D$detectObjects$0.call(Unknown Source)
    org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
    Script2.run(Script2.groovy:20)
    org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
    org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155)
    qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:926)
    qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:859)
    qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:782)
    qupath.lib.gui.scripting.DefaultScriptEditor$2.run(DefaultScriptEditor.java:1271)
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base/java.lang.Thread.run(Unknown Source)

What can I do, can I change my allowed_devices somewhere? I have repeated the training now three times with the same result.

This an error I get in the notebook, but that should not be an issue, right?:

Apparently I need to “check whether my GraphDef-interpreting binary is up to date with my GraphDef-generating binary”

I have reinstalled TensorFlow - can I update the version of my QuPath TensorFlow as well?

Not easily, as far as I know TensorFlow 1.5 is the most recent stable release for Java. I hope that 2.3 might be available soon; you can see a lot of activity on the repo: https://github.com/tensorflow/java

However see this post: TensorFlow 2.0 migration

1 Like

okay, so that is the problem then, right?

Thanks, I will dive into that over the weekend!

1 Like

I have rebuilt everything and have now come to the point where I get the following error in my StarDist notebook:

ModuleNotFoundError                       Traceback (most recent call last)
/usr/local/lib/python3.8/site-packages/csbdeep/models/__init__.py in <module>
      4 try:
----> 5     import tensorflow
      6     del tensorflow

ModuleNotFoundError: No module named 'tensorflow'
...
RuntimeError: Please install TensorFlow: https://www.tensorflow.org/install/

So next I will install Tensorflow for running the notebook.

When I played with my new QuPath, it gave me the message

INFO: Loaded TensorFlow bundle: /Users/thiemomoellenkamp/Documents/StarDist_model_HE_heavy_augment, (inputinput:0 [-1,-1,-1,3], output=concatenate_4/concat:0 [-1,-1,-1,33])
INFO: Done!

So Tensorflow is already installed, right?
Can I change the repository of the qupath-extension-tensorflow so that my Jupiter notebook accesses the same version?
Or @petebankhead which version should I now download? Thanks so much for the help!

When you built QuPath with -Ptensorflow-cpu=true or -Ptensorflow-gpu=true then it will download a Java-friendly version of TensorFlow (currently 1.15) that it can use but it is completely separate from the TensorFlow you’ll need to install for training in Python.

See also some of the answers at

If you train a model that works in Fiji then it should also work in QuPath, since both are using Java (and therefore subject to the same limitations).

Edit: Fixed typo in TensorFlow version number (previously written 1.5).

How is this version (1.5) related to the “official” version of TensorFlow (e.g. 1.14) as used by the Python and Java (Maven) libraries?

Sorry, typo, I meant 1.15!

It should be essentially the same, although if you’re using the QuPath master branch it’s the JavaCPP presets distribution (which contains both the standard Java API and an alternative that uses JavaCPP-specific features).

I figured that would aid interoperability with other libraries (e.g. OpenCV) but in the end it didn’t, so I’ve since switched to the ‘official’ TensorFlow distribution for the next QuPath release. However it’s unclear to me what the future of that is, since the official TensorFlow repo is now starting to incorporate JavaCPP at https://github.com/tensorflow/java

But all that is handled internally within QuPath; from a user’s point of view, compatibility issues should be the same as using TensorFlow 1.15 generally.

1 Like