DeepCell - Tiling

Hi @petebankhead and @Noah_Greenwald!

I promptly took the new script for a spin, and since there are some size limitations, I went ahead and tiled a large area of an image:
image
And tried DeepCell on it using Pete’s code from Twitter (no changes except including channel IDs).
It ran for… quite a while. I want to say nearly an hour, and eventually failed and created no cells.

ERROR: IOException: Unexpected code Response{protocol=h2, code=502, message=, url=https://deepcell.org/api/status/}

ERROR: org.vanvalenlab.KioskHttpClient.sendHttpRequest(KioskHttpClient.java:83)
    org.vanvalenlab.KioskHttpClient.getStatus(KioskHttpClient.java:192)
    org.vanvalenlab.KioskJob.updateStatus(KioskJob.java:121)
    org.vanvalenlab.KioskJob.waitForFinalStatus(KioskJob.java:54)
    org.vanvalenlab.KioskJob$waitForFinalStatus$0.call(Unknown Source)
    Script5.runJob(Script5.groovy:159)
    Script5$runJob$4.callStatic(Unknown Source)
    Script5.detectCells(Script5.groovy:124)
    Script5$detectCells$0.callStatic(Unknown Source)
    Script5$_run_closure1.doCall(Script5.groovy:79)
    jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
    java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    java.base/java.lang.reflect.Method.invoke(Unknown Source)
    org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
    groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1029)
    groovy.lang.Closure.call(Closure.java:412)
    groovy.lang.Closure.call(Closure.java:406)
    java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.base/java.lang.Thread.run(Unknown Source)

Did all of the tiles trigger a problem with the submission process?

The output prior to the error was 119 “Returned: ##” statements, and prior to that, 1582 lines of
"ROIs: ## "
“Detections:##”

Total size was only 7.6mm^2, so not that big compared to what I would want to submit in the future. I suspect for something like this I would want the model to be local, though, right?

The server can handle that volume of images, we’ve directly submitted large tiles before without problems. However, we haven’t tried through QuPath. Can you try submitting just a couple crops first to make sure the script works as expected?

1 Like

Yep, few crops is fine.

Okay, we’ll look into it https://github.com/vanvalenlab/kiosk-imageJ-plugin/issues/33

2 Likes

Something that @smcardle brought up is that it sounds like there is a limit to the number of ROI/Overlays the QuPath->ImageJ and back transfer can handle. It might not be so much the number of regions or the size, but the total number of “things” the bridge is sending back and forth. In that case, it might make sense to split the submissions a bit more.
I’ll look at it a bit more tonight once I get back home.

I’m not sure what that refers to – I can’t think of a reason why it would be a problem on the QuPath side unless there’s a RAM limit, or too many labels on a labelled image for the bit-depth.

In any case, the script isn’t really intended for this kind of use but really just as a demo to show how the results can be handled on the QuPath side. For serious use could/should be made more robust, using try/catch blocks to recover more gracefully (and/or try again) when a single region fails… and preferably also apply automatic tiling and give more meaningful progress monitoring.

(I did see errors on a couple of individual/small manually-drawn regions myself when testing, but I didn’t record what the errors were… and if I shifted the regions slightly they worked fine).

If I send a downsampled image with ~1000 annotations and ~16000 detections (25 classes) to ImageJ with “Include overlay” checked, I consistently get this error:

ERROR: QuPath exception
    at qupath.imagej.gui.IJExtension.extractOverlay(IJExtension.java:411)
    at qupath.imagej.gui.IJExtension.extractROIWithOverlay(IJExtension.java:366)
    at qupath.imagej.gui.ExtractRegionCommand.run(ExtractRegionCommand.java:254)
    at qupath.imagej.gui.IJExtension$IJExtensionCommands.lambda$new$1(IJExtension.java:533)
    at qupath.lib.gui.QuPathGUI.lambda$createImageDataAction$0(QuPathGUI.java:360)
    at org.controlsfx.control.action.Action.handle(Action.java:419)
    at org.controlsfx.control.action.Action.handle(Action.java:64)
    at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:234)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
    at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
    at javafx.event.Event.fireEvent(Event.java:198)
    at javafx.scene.control.MenuItem.fire(MenuItem.java:459)
    at com.sun.javafx.scene.control.ContextMenuContent$MenuItemContainer.doSelect(ContextMenuContent.java:1380)
    at com.sun.javafx.scene.control.ContextMenuContent$MenuItemContainer.lambda$createChildren$12(ContextMenuContent.java:1333)
    at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:247)
    at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:234)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
    at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
    at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
    at javafx.event.Event.fireEvent(Event.java:198)
    at javafx.scene.Scene$MouseHandler.process(Scene.java:3890)
    at javafx.scene.Scene.processMouseEvent(Scene.java:1885)
    at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2618)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:409)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:299)
    at java.base/java.security.AccessController.doPrivileged(Unknown Source)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:447)
    at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:412)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:446)
    at com.sun.glass.ui.View.handleMouseEvent(View.java:556)
    at com.sun.glass.ui.View.notifyMouse(View.java:942)
    at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    at java.base/java.lang.Thread.run(Unknown Source)

I haven’t tested it extensively. But, sending only the 1000 annotations as an overlay works fine. Maybe the problem isn’t the number of objects but their shape or size or amount of overlap? Some of these objects will be smaller than 1 pixel when downsampled and the detections come from intersections between multiple rounds of pixel classification.

(I rearranged my workflow to simply avoid this issue, and I didn’t report this as a bug because who, other than me and @Mike_Nelson would even try to do this?)

2 Likes

Yep, and I was originally thinking the issue was mostly on the DeepCell side since all of the errors I was getting were server related. I have poked the bear a few times and it looks more like it is the total number of objects (vertices?) being submitted at once, regardless of how many tiles I break things up into. I think I could adjust the script logic to loop per annotation and it would be fine, but I haven’t gotten back to it just yet.

Hmmm, roi.setStrokeColor seems a weird place for such an exception… although ImageJ Rois are a lot more heavyweight than QuPath ROIs, so storing them may require a fairly huge amount more overhead. I’m not too surprised it doesn’t work, although I don’t know what exactly is going awry.

I think that’s a fair assessment :wink:

Just restrict the number of parallel threads?

Re. vertices, I don’t think any ROIs are being passed… just the TIFF image (written as a temp file), and the result is a TIFF labeled image that is used to generate ROIs later on the QuPath side.

Separately, I can get it to fail with this error:

KioskJobFailedException: Traceback (most recent call last):
  File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 202, in consume
    status = self._consume(redis_hash)
  File "/usr/src/app/redis_consumer/consumers/multiplex_consumer.py", line 122, in _consume
    image = self.predict(image, model_name, model_version)
  File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 605, in predict
    model_dtype)
  File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 525, in _predict_small_image
    padded_img = np.pad(image, pad_width, 'reflect')
  File "<__array_function__ internals>", line 6, in pad
  File "/usr/local/lib/python3.6/site-packages/numpy/lib/arraypad.py", line 746, in pad
    pad_width = _as_pairs(pad_width, array.ndim, as_index=True)
  File "/usr/local/lib/python3.6/site-packages/numpy/lib/arraypad.py", line 517, in _as_pairs
    raise ValueError("index can't contain negative values")
ValueError: index can't contain negative values

This happened when I passed an image that is 287 x 215 pixels in size (including via the ImageJ Kiosk plugin installed separately in Fiji, so with no QuPath link). Various other sizes fail similarly.

@Noah_Greenwald is there a fixed size / range of sizes that is expected?

Hi @petebankhead ,
That’s a great catch. We automatically pad images that are too small. However, our current padding assumes that an image which is too small will be too small in both dimensions, not just one. I’ll get that updated today.

In the meantime, submitting images where both dimensions are greater than 256 or both dimensions are smaller than 256 should serve as a workaround.

3 Likes