Conversation with the OpenJDK developers about JavaFX and Unsafe

java
memory
javafx
unsafe
graphics
scenery
paintera

#1

Hi @axtimwalde @hanslovsky @kephale @skalarproduktraum @tpietzsch and everyone,

I have spoken in the recent past with some of you about reaching out to the developers of OpenJDK and/or JavaFX 3D about two very major issues for our developer community:

  1. Lack of performant / low-level access to the graphics context of a JavaFX scene. This affects latency in Paintera and Scenery, if I understand correctly.

  2. Lack of an Unsafe-like API for fine-grained memory control in Java 11. This affects imglib2-cache/BDV(?) and imglib2-unsafe, and therefore imglyb and pyimagej.

I would like to create a technical writeup of these issues, so that we can submit it jointly to the relevant people at Oracle et al., asking for advice on how best to proceed.

  • @skalarproduktraum As discussed in December, specific words from you regarding how Scenery is impacted, and how you would like to see the API be improved, would be super helpful.

  • @axtimwalde @hanslovsky Similarly, a concise but technical explanation of the issues with JavaFX+Paintera would be very helpful.

  • @tpietzsch Would you be able to comment on the issues with Java 11 + imglib2-cache and/or BDV?

There is no hard deadline for us to put together this writeup per se, but the sooner we get the conversation rolling, the sooner we might see technical solutions emerge. The pressure to switch to Java 11 will grow over the next 2 years, and I’d like to be on top of these impending obstacles.

Thanks so much for any time you invest in this direction!


#2

@ctrueden you mentioned imglyb but explicitly asked for Paintera/JavaFX, so I will focus on the latter. If you would also like to have a quick summary on imglyb, please let me know.

In Paintera we visualize orthogonal 2D-cross-sections through a 3D volume (three of the four viewer panes) and 3D-renderings polygon meshes of three-dimensional objects of interest in a fourth viewer pane. We also render the 2D-cross sections as textured meshes into the 3D-viewer:

JavaFX provides a very nice and abstracted API that is very convenient to build UIs and scene graphs. This abstraction, however, prevents us from directly accessing the OpenGL context. As a result, we cannot share textures between the 2D-cross-sections and their embeddings in the 3D-viewer. Instead, we

  1. Render the 2D-cross-sections on the cpu (like BDV)
  2. Update an ImageView with the newly rendered image
  3. Copy the texture onto the graphics card (JavaFX does that for us)
  4. Copy the texture for the 3D embedding onto the graphics card (JavaFX does that for us)

The copy to the graphics card seems to be the bottleneck for us, and it also seems that this copy happens on the application thread, i.e. the entire application has to wait until the copy is finished. With high-res monitors (many pixels) and a lot of screen updates (e.g. scrolling through a 3D volume), this becomes very slow. We circumvent this by

  1. Downscaling the images before embedding in the 3D-viewer (step 4 in previous list)
  2. Delaying the updates of the textured meshes in the 3D-viewer

Ideally, we would create an OpenGL texture on a separate thread (i.e. not the application thread), and then just pass the texture id instead of steps (3) and (4).


#3

I would also like to provide an example of a case in which the use of Unsafe can be easily replaced by the Java Buffer interface. In ByteUtils in our label multiset repository, we use Unsafe not to create native memory (array sizes will not exceed Integer.MAX_VALUE, no other need to look into native memory, either) but instead we want to create heterogeneous lists of primitive types, e.g. {int, int, long, long, int, int, int, long}, hence ByteUtils.putLong etc. The same (at least for this use-case) functionality is provided by ByteBuffer.

For this and similiar use cases, we could easily replace Unsafe with ByteBuffer.


#4

We are also not the only ones who are interested in accessing OpenGL directly in a JavaFX application:

And while looking at the java-gaming link above, I also saw this very interesting blog post:

DriftFX allows you to directly embed content rendered by native pipelines into JavaFX WITHOUT going through the Main-Memory. The rendered artifacts stay at the GPU all time!

I haven’t looked into it, yet, though.


#5

I was also investigating a bit, which replacements are there, and which replacements will likely come.
I think the general situation can be summarized as:
They have started removing stuff from Unsafe in Java 11, but only things for which there is already a replacement. There still are things for which there is no replacement. I think it is fine to keep relying on Unsafe for those, until there is a better alternative.

For the memory access, a more recent alternative to ByteBuffer is VarHandles, which also can do volatile writes for example (possibly CAS, I don’t remember…). I watched this very interesting talk
https://youtu.be/_bVcHCt-J6Y about various stuff that Unsafe is used for and replacements (I recommend it).

What is missing (and I think we might need that for interfacing python) is allocating >2G and wrapping specific addresses (we could wrap in new DirectByteBuffer through JNI, but still only <2G). Replacement for that will come with Project Panama and might look like this

public long memory(long index) {
     // The scope where the memory region is available
     // Implements AutoClosable but `close` can be called manually as well
     try (Scope scope = new NativeScope()) {
        // Allocate the actual memory area, in this case in the style of a "long-array"
        Pointer<Long> ptr = scope.allocate(
                     NativeLibrary.createLayout(long.class), numElements);

        // Get the reference to a certain element
        Reference<Long> ref = ptr.offset(index).deref();

        // Set a value to this element through the reference
        ref.set(Long.MAX_VALUE);

         // Read the value of an element
         return ref.get();
    }
}

(copied from https://forum.byte-welt.net/t/project-panama-bekommt-memory-regions/10792 by the presenter of the above talk)

So I see no problems with any of our usage of Unsafe going forward.


#6

The problem with imglib2-cache has nothing to do with Unsafe.
There, the problem is that the semantics of PhantomReference slightly changed between java 8 and 9
(See the difference between the javadocs:
https://docs.oracle.com/javase/8/docs/api/java/lang/ref/PhantomReference.html
https://docs.oracle.com/javase/9/docs/api/java/lang/ref/PhantomReference.html)

Let me first explain the task that needs to be solved in imglib2-cache (and then why we use PhantomReferences to do it, and where that breaks now):

In imglib2-cache, when an entry (lets say a Cell from a CellImg) is thrown out of the Cache, a CacheRemover is notified – so that it can write the Cell out to disk for example. The problem is that we do not want to throw anything out of the cache while it is still in use. If a RandomAccess holds Cell A and we remove it from the cache (and write it to disk), then changes to Cell A after that are lost. Worse: If another RandomAccess now comes and wants Cell A again, it will get a different object, and now two competing versions of reality exist.
In most “normal” caching scenarios, this is a non-issue, because the entries in the cache are immutable things (with equals() equality). But we have modifiable entries (with ‘==’ equality).
Anyway, so unless we want to do reference counting (let clients check entries out and back in to the cache), we need to rely on the garbage collector.

The GC tells us when the entry can be collected, and then we can be sure that it is no longer in use. However, before it is really garbage collected, we need to get to the entry and write it to disk. So basically we need to bring a dead object back to life. This is obviously evil and we need to be careful to really only use the “zombie Cell” to write it to disk and not let it get back out to a RandomAccess.

The way imglib2-cache does it, is by keeping PhantomReferences to the entries. When an entry is (about to be) garbage collected, its PhantomReference is enqueued in a reference queue where we retrieve it.
The javadoc says

In order to ensure that a reclaimable object remains so, the referent of a phantom reference may not be retrieved

However that was not really true: you still could get to the referent via reflection.
And the referent never was null, because the javadoc also said

Unlike soft and weak references, phantom references are not automatically cleared by the garbage collector as they are enqueued. An object that is reachable via phantom references will remain so until all such references are cleared or themselves become unreachable.

So basically, until we process the queue, the stuff will stay around.

That has changed, the java 9 javadoc no longer has that last paragraph. We still can get to the referent via reflection, but now its null, always.

It’s fine that this changed, what we did was a bit fishy anyway.
Now we need to find a workaround.
I see no way to do it without changing the imglib2-cache API.

Maybe I should say something about how PhantomReferences are typically used in this scenario: Assume you have object A that you want to do something with (write to disk…) when its garbage collected. But actually, all that you care about is some data contained in A (lets say a field B b). Now you derive a PhantomReference<A> subclass that also has the b field. So even after A is gone for good, you still have the data required to write “A”.

The nice thing about current imglib2-cache API is, that it works (worked) without putting any restrictions on what T you could put in the cache. But to do the above trick, now we probably need some T extends Cacheable that gives us access to the Internals so that we can put those in our PhantomReference. The writer for T now either needs to be able to write Internal, or we need a way to recreate a T from Internal. Clients of the cache have to promise to never hold a reference to Internal when they don’t also hold a reference to T, etc. That all doesn’t sound so nice…

I don’t really think we need anything changed in the JDK or something like that. We just need a good idea. Basically we need a “special” reference to T and be notified if only “special” references to T are left.


#7

@tpietzsch: I am not sure if this could be done by caching Reference<Reference<T>> instead of Reference<T>?


#8

@tpietzsch Dumb question: what about overriding finalize()?

Edit: I guess it was deprecated in Java 9; but what about Cleaner?


#9

This is the same problem basically. It requires clients to know more about caching, i.e., classes that should be cached need to provide a finalize() implementation or a Cleaner that integrates with the CacheRemover correctly. (which is probably fine but requires API changes).

Judging from the javadoc, Cleaner is probably internally based on a PhantomReference queue, basically implementing the same thing I described above.


#10

I don’t see how this could help.
Holding a Reference<T> means that T can be collected if no longer externally used.
Holding a Reference<Reference<T>> means that the Reference<T> can be collected if no longer externally used. And the Reference<T> would never be externally used?


#11

True—never mind :frowning: