Detection measurements table can't be saved after subcellular detection of 2D whole image_QuPath exception error

Hi,

I am trying to analyse a .ndpi 2D whole image (1GB), using a 48GB RAM computer. The QuPath version I am using is 0.2.0.
The idea is to do cell detection of the whole image first, to then do a subcellular detection to identify DAB dots from a RNAscope image. This took about 1 hour and a half, not sure if this is the usual time. The problem comes when i try to open and save the detection measurements table. I can see the table, but it won’t allow me to save it or copy it, as it’s too big and an error message pops up:

ERROR: Reached end of file…
ERROR: Error

  • at qupath.lib.objects.hierarchy.PathObjectHierarchy.setHierarchy(PathObjectHierarchy.java:563)*
  • at qupath.lib.io.PathIO.readImageDataSerialized(PathIO.java:201)*
  • at qupath.lib.io.PathIO.readImageDataSerialized(PathIO.java:95)*
  • at qupath.lib.io.PathIO.readImageData(PathIO.java:292)*
  • at qupath.lib.projects.LegacyProject$LegacyProjectImageEntry.readImageData(LegacyProject.java:750)*
  • at qupath.lib.gui.QuPathGUI.openImageEntry(QuPathGUI.java:2293)*
  • at qupath.lib.gui.panels.ProjectBrowser.lambda$new$2(ProjectBrowser.java:170)*
  • at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)*
  • at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)*
  • at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)*
  • at javafx.event.Event.fireEvent(Event.java:198)*
  • at javafx.scene.Scene$ClickGenerator.postProcess(Scene.java:3564)*
  • at javafx.scene.Scene$ClickGenerator.access$8200(Scene.java:3492)*
  • at javafx.scene.Scene$MouseHandler.process(Scene.java:3860)*
  • at javafx.scene.Scene$MouseHandler.access$1200(Scene.java:3579)*
  • at javafx.scene.Scene.processMouseEvent(Scene.java:1849)*
  • at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2588)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:397)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:295)*
  • at java.base/java.security.AccessController.doPrivileged(Native Method)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:434)*
  • at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:390)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:433)*
  • at com.sun.glass.ui.View.handleMouseEvent(View.java:556)*
  • at com.sun.glass.ui.View.notifyMouse(View.java:942)*
  • at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)*
  • at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)*
  • at java.base/java.lang.Thread.run(Thread.java:834)*
    ERROR: Load ImageData: QuPath has encountered a problem, sorry.
    If you can replicate it, please notify a developer.

In addition, once i close the program and try to re-open the project again to see the analysed image, the image won’t open and an error message shows again (java.lang.NullPointerException):

ERROR: Reached end of file…
ERROR: Error

  • at qupath.lib.objects.hierarchy.PathObjectHierarchy.setHierarchy(PathObjectHierarchy.java:563)*
  • at qupath.lib.io.PathIO.readImageDataSerialized(PathIO.java:201)*
  • at qupath.lib.io.PathIO.readImageDataSerialized(PathIO.java:95)*
  • at qupath.lib.io.PathIO.readImageData(PathIO.java:292)*
  • at qupath.lib.projects.LegacyProject$LegacyProjectImageEntry.readImageData(LegacyProject.java:750)*
  • at qupath.lib.gui.QuPathGUI.openImageEntry(QuPathGUI.java:2293)*
  • at qupath.lib.gui.panels.ProjectBrowser.lambda$new$2(ProjectBrowser.java:170)*
  • at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)*
  • at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)*
  • at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)*
  • at javafx.event.Event.fireEvent(Event.java:198)*
  • at javafx.scene.Scene$ClickGenerator.postProcess(Scene.java:3564)*
  • at javafx.scene.Scene$ClickGenerator.access$8200(Scene.java:3492)*
  • at javafx.scene.Scene$MouseHandler.process(Scene.java:3860)*
  • at javafx.scene.Scene$MouseHandler.access$1200(Scene.java:3579)*
  • at javafx.scene.Scene.processMouseEvent(Scene.java:1849)*
  • at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2588)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:397)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:295)*
  • at java.base/java.security.AccessController.doPrivileged(Native Method)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:434)*
  • at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:390)*
  • at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:433)*
  • at com.sun.glass.ui.View.handleMouseEvent(View.java:556)*
  • at com.sun.glass.ui.View.notifyMouse(View.java:942)*
  • at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)*
  • at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)*
  • at java.base/java.lang.Thread.run(Thread.java:834)*
    ERROR: Load ImageData: QuPath has encountered a problem, sorry.
    If you can replicate it, please notify a developer.

Does anyone have any suggestions to solve this problem?

Cheers.

It looks like you are using v0.2.0-m1/2 with a project written by v0.1.2. The message when you start up the milestone version warns that this isn’t a good idea: https://github.com/qupath/qupath/blob/master/STARTUP.md

It also looks like those two error messages you posted are the same…

The first one doesn’t look relevant to the problem. My guess is that the actual error arises from the table just being much too big to write to a text file. This is a known limitation/bug. As far as I recall, QuPath uses the same code to write to the file as to copy to the clipboard, which limits it to the maximum length of a String in Java… which is around 231 characters.

This needs addressed, but in the meantime there are script-based ways to export larger amounts of data with a lot more control.

The second error might be connected to mixing versions. See also https://github.com/qupath/qupath/issues/58 and Java.lang.null Error

Regarding performance, I’m not terribly happy with the subcellar detection command generally (although I wrote it… with the help of ImageJ’s Find Maxima) but it’s all QuPath has for now. It’s somewhere on my todo list but not something I’m actively working on, so the best option in the meantime is to apply it in restricted regions rather than across the full slide, or to develop a custom detection (either as an extension or script).

One important consideration is that it’s likely RNAscope analysis will always require analysis at a very high resolution. While your compressed image may be 1 GB, I would guess the raw image could be 40-50 GB at full resolution, and the full resolution information is required to identify the smallest features.

Thanks so much Peter for your quick reply.

I guess the main problem is that the table is too big to write to a text file, so i’ll apply the detection to smaller regions rather than the whole section.

Regarding the different versions, i’m a bit confused, as this is a new project and i never used the old version, not sure how this could happen.

Thanks so much and i look forward to seeing the optimised version for subcellular detection although i am quite happy with the subcellular detection so far.

Thanks Laura, I was thinking it’s an older project because of this line in the error:

qupath.lib.projects.LegacyProject$LegacyProjectImageEntry.readImageData(LegacyProject.java:750)

I could be wrong though. Projects are still undergoing a revision for the next milestone so I’m not often using v0.2.0-m2 myself…

Hi Peter,

Sorry for bothering you again. I have now tried doing the subcellular detection in a smaller region, and I still get an error message:

qupath.imagej.detect.cells.SubcellularDetection {“detection[DAB]”: 0.05, “doSmoothing”: true, “splitByIntensity”: true, “splitByShape”: true, “spotSizeMicrons”: 1.0, “minSpotSizeMicrons”: 0.5, “maxSpotSizeMicrons”: 2.0, “includeClusters”: true}
ERROR: QuPath exception

  • at java.base/java.util.Arrays.copyOf(Arrays.java:3745)*
  • at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)*
  • at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)*
  • at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)*
  • at qupath.lib.gui.commands.SummaryMeasurementTableCommand.getTableModelString(SummaryMeasurementTableCommand.java:741)*
  • at qupath.lib.gui.commands.SummaryMeasurementTableCommand.saveTableModel(SummaryMeasurementTableCommand.java:777)*
  • at qupath.lib.gui.commands.SummaryMeasurementTableCommand.lambda$run$10(SummaryMeasurementTableCommand.java:308)*
  • at qupath.lib.gui.commands.SummaryMeasurementTableCommand$$Lambda$1616/0x0000000800912040.handle(Unknown Source)*
  • at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)*
  • at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)*
  • at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)*
  • at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)*
  • at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)*
  • at javafx.event.Event.fireEvent(Event.java:198)*
  • at javafx.scene.Node.fireEvent(Node.java:8879)*
  • at javafx.scene.control.Button.fire(Button.java:200)*
  • at com.sun.javafx.scene.control.behavior.ButtonBehavior.mouseReleased(ButtonBehavior.java:206)*
  • at com.sun.javafx.scene.control.behavior.ButtonBehavior$$Lambda$824/0x00000008004e8840.handle(Unknown Source)*
  • at com.sun.javafx.scene.control.inputmap.InputMap.handle(InputMap.java:274)*
  • at com.sun.javafx.scene.control.inputmap.InputMap$$Lambda$490/0x000000080045cc40.handle(Unknown Source)*
  • at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:218)*
  • at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)*
  • at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)*
  • at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)*

Is it because the region is still quite large even if smaller than the original one?

Cheers.

If the text file is too large, you could also try removing measurements from the cells, subcellular detections, or remove the subcellular detections entirely. I usually opt for the third option first, as I mostly want the estimated spot counts for a given stain/channel.

So I would copy that value to a new measurement, then run subcellular detection again with -1 in the threshold(s).

Also, have you tried just running it on a very small region?