Draw an overlay programmatically

[jrjf2 (Jacob Francis)]

Hi All,

I have a paticular issue that was hoping someone could help me with. It goes as follows:

• We have a custom java application using ImageJ2, graphically embeded within this is ImageJ’s main toolbars.
• We are using dataset and display window (from uiService.defaultUI.createDisplayWindow()) to create an image
• The image comes from a camera. Each time a new image is availabe, we call setPlane(0, theImage)
• When I draw an overlay (rectangle, oval, etc…) using the standard drawing tool, the overlay persists on the image despite the frequent setPlane() calls.
• When I draw an overlay on the same image using OverlayService.drawOverlay() the overlay is drawn but there are two issues:
  • Firstly, the overlay disappears after the first call to setPlane(). I can call drawOverlay() each time I call setPlane() to make it “persist” but that involves creating a new overlay object on each new image.
  • Secondly, the overlay that is drawn using drawOverlay() just has a black outline and none of the graphical appearance (blue outline, yellow shading etc…) of an overlay that was drawn using the drawing tool.

Is OverlayService.drawOverlay() fully implemented or is it supposed to act differently to using the overlay drawing tool?

Related post:

Draw overlay on a dataset (or iterate through pixels of the contour of an overlay)

Hello @stelfrich, thanks for the tips but I already used this service but the problem is that it doesn’t take account the “lineWidth” of the overlay. I would need an iterator that allows me to iterate through the neighbours of pixel position for instance or an iterator that takes the this into account which would allow me to draw on a dataset directly. [image] (On the picture here, drawing is done using the JavaFX nodes. I want to reach the same result but at a dataset level). Also, I’m also…

I would appreciate any advice on this or any possible work arounds

Thanks in advance,

Jacob

[imagejan (Jan Eglinger)]

When I draw an overlay on the same image using OverlayService.drawOverlay() the overlay is drawn but there are two issues:

• Firstly, the overlay disappears after the first call to setPlane() . I can call drawOverlay() each time I call setPlane() to make it “persist” but that involves creating a new overlay object on each new image.
• Secondly, the overlay that is drawn using drawOverlay() just has a black outline and none of the graphical appearance (blue outline, yellow shading etc…) of an overlay that was drawn using the drawing tool.

I’m not sure exactly how OverlayService::drawOverlay() is supposed to work, but please note there’s a difference between ij.gui.Overlay (the ImageJ 1.x overlay) and net.imagej.overlay.Overlay (the ImageJ2 overlay).

If you say “graphically embedded within this is ImageJ’s main toolbars”, you most likely mean the legacy UI of ImageJ 1.x, together with the preferences (appearance of ROIs etc.) of ImageJ1. These are not automatically transferred to net.imagej.overlay.Overlay objects (yet).

In general, I fear that @ctrueden’s comment from the topic you linked is still valid today:

Draw overlay on a dataset (or iterate through pixels of the contour of an overlay)

ImageJ2’s Overlay classes are highly incomplete compared to ImageJ 1.x. Furthermore, the drawing of ImageJ2’s Swing UI cheated by using JHotDraw. We definitely want to move away from this in favor of algorithms in ImageJ Ops that render ROIs in a UI-agnostic way. This is a highly unexplored & unimplemented area of ImageJ2, which many people want and need, but no one has time to work on.

However, since then there has been the development of a new, powerful ROI model in imglib2-roi, together with converters from and to ImageJ1 ROIs in imagej-legacy.

I would appreciate any advice on this or any possible work arounds

If in the end you’re actually using ImageJ 1.x infrastructure (ImageWindow, ImagePlus etc.) to display your images in your custom GUI, it might be the safest to also stay with ImageJ 1.x overlays for now.

On the other hand, if you’re using #bigdataviewer , #sciview or some other (ImageJ2-based) visualization UI, you might want to post a link to your source code, so some experts can give more specific advice.

[jrjf2 (Jacob Francis)]

Hi Jan,

Thank you for your detailed reply. To clarify a few questions you had:

• Everything we are doing is using ImageJ2 infrastructure wherever possibe. I am using net.imagej.overlay.Overlay
• See the screenshot of how we’ve embedded ImageJ. You will see we’ve also added some new buttons to one of the toolbars.

• I was reluctant to share code at first as we are using Kotlin (100% interoperable with Java) and other technologies. However here is the code (please note it is in Kotlin, but you should be able to understand it):

Create the camera display window from a dataset:

display = displayService.createDisplayQuietly(dataset)

view = ((display as DefaultImageDisplay).activeView as DefaultDatasetView)

displayWindow = uiService.defaultUI.createDisplayWindow(display)

uiService.viewerPlugins
	.map { pluginService.createInstance(it) }
	.find { it != null && it.canView(display) && it.isCompatible(uiService.defaultUI) }
	?.let {
		GUIMain.threadService.queue {
			(displayWindow as SwingDisplayWindow).apply {
				uiService.addDisplayViewer(it)
				it.view(this, display)
				pack()
				windowPanel.add((displayWindow as SwingDisplayWindow).rootPane)
			}
		}
	}

Then later on, using a custom right click menu get the drawn and selected overlay:

activeDisplay = imageDisplayService.activeImageDisplay
selectedROI = overlayService.getActiveOverlay(activeDisplay)

In the same block of code that calls setPlane(0, theImage) we have:

newOverlay = RectangleOverlay()
newOverlay.setOrigin(200.0, 0)
newOverlay.setOrigin(200.0, 1)
newOverlay.setExtent(200.0, 0)
newOverlay.setExtent(200.0, 1)
newOverlay.name = "TestOverlay"
ds = GUIMain.overlayService.defaultSettings
newOverlay.alpha = ds.alpha
newOverlay.fillColor = ds.fillColor
newOverlay.lineColor = ds.lineColor
newOverlay.lineStyle = ds.lineStyle
newOverlay.lineWidth = ds.lineWidth
	
threadService.queue {
	overlayService.drawOverlay(newOverlay, activeDisplay, ChannelCollection(listOf(0.0)))
	activeDisplay.update()
}

I have managed to find a possible workaround to this: using Java’s Robot() object - programmatically click on the overlay tool and then programmatically click and drag over the camera image. This does work but has pitfalls such as other custom build UI elements being in the way and its a workaround I want to avoid if I can.

Thanks,

Jacob

[ctrueden (Curtis Rueden)]

@jrjf2 I’m afraid that your project has collided with some of ImageJ2’s rough edges.

1. You are using the Swing UI, which was originally written to replace ImageJ’s AWT user interface. In recent years, development of the Swing UI has been shelved in favor of continuing to support the AWT interface via the imagej-legacy adapter layer.

2. The ImageJ2 net.imagej.overlay.Overlay machinery is now deprecated in favor of the imglib2-roi API. However, the ImgLib2 ROI types have not yet been integrated into the ImageJ user interface—and no one is actively working on doing this.

Keeping in mind both of the above points: what you are trying to do should be possible fundamentally, but there may be bugs. If you have the time and energy to debug into the code and fix them, then pull requests are welcome. Alternately, if you are able to create a minimal example demonstrating the issue, we can file an issue in the imagej-ui-swing component and I can try to dig a bit myself on what might be going wrong.

[jrjf2 (Jacob Francis)]

@ctrueden

Please see the attached minimal examples created in IntelliJ IDEA. “DrawingOverlayMinimal” is truly minimal, “DrawingOverlay” is almost minimal but uses some more UI code that we use in our program. Both examples can reproduce the issue. I have also attached an example .tif as a dataset that will work with the examples provided.

I think we will end up making a pull request and look at fixing this ourselves but I don’t know the time frame on this.

Thanks,

JacobDrawingOverlay.zip (74.5 KB) DrawingOverlayMinimal.zip (71.3 KB) example.tif (6.3 MB)

[ctrueden (Curtis Rueden)]

I think we will end up making a pull request and look at fixing this ourselves but I don’t know the time frame on this.

Sounds great, and sorry I don’t have more time to dig in myself!