Using LambdaMetafactory to create a Consumer

Hi all,

I’m trying to create a Consumer in jython for use in Iterator.forEachRemaining, a default interface method available in a net.imglib2.Cursor.

While I can do it with a simple jython class, the performance is horrifying. I can also do it using ASM, but it’s tedious–I could abstract it and then using String substitutions use it quite flexibly, but frankly I am looking for something easier–a concise idiom that, ironically, modern java has, and jython lacks.

So now I’ve explored java 8 LambdaMetafactory, and my current attempt looks like this:

from java.lang.invoke import MethodHandles, MethodType, LambdaMetafactory
from java.lang import Void
from java.util.function import Consumer
from net.imglib2.type.numeric.integer import UnsignedByteType
from net.imglib2.type.operators import SetOne

caller = MethodHandles.lookup()
mt = MethodType.methodType(Void)
target = caller.findVirtual(SetOne, "setOne", mt)
#target = caller.findSpecial(SetOne, "setOne", mt, UnsignedByteType)
fn = target.type()
invokedType = MethodType.methodType(Consumer)
mt2 = MethodType.methodType(Void, UnsignedByteType, Void)
site = LambdaMetafactory.metafactory(caller, "accept", invokedType, fn.generic(), target, mt2)

consumer = site.getTarget().invokeExact()

t = UnsignedByteType(0)
consumer.accept(t)

print t

The above results in this surprising error:

Started test_consumer_from_Lambdametafactory.py at Sun May 17 22:51:28 BST 2020
Traceback (most recent call last):
  File "/home/albert/lab/scripts/python/imagej/tests/test_consumer_from_Lambdametafactory.py", line 9, in <module>
    target = caller.findVirtual(SetOne, "setOne", mt)
	at java.lang.invoke.MemberName.makeAccessException(MemberName.java:871)
	at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1003)
	at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1386)
	at java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:861)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
java.lang.NoSuchMethodException: java.lang.NoSuchMethodException: no such method: net.imglib2.type.operators.SetOne.setOne()Void/invokeInterface

Surprising, because what it says isn’t true: the setOne method as described exists.

If, instead, at line 10 I use the findSpecial approach (commented out above) to invoke the setOne interface method in the implementing class UnsignedByteType:

target = caller.findSpecial(SetOne, "setOne", mt, UnsignedByteType)

… then I get a different, also surprising error:

Started test_consumer_from_Lambdametafactory.py at Sun May 17 23:11:52 BST 2020
Traceback (most recent call last):
  File "/home/albert/lab/scripts/python/imagej/tests/test_consumer_from_Lambdametafactory.py", line 10, in <module>
    #target = caller.findSpecial(SetOne, "setOne", mt, UnsignedByteType)
	at java.lang.invoke.MemberName.makeAccessException(MemberName.java:850)
	at java.lang.invoke.MethodHandles$Lookup.checkSpecialCaller(MethodHandles.java:1572)
	at java.lang.invoke.MethodHandles$Lookup.findSpecial(MethodHandles.java:1002)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
java.lang.IllegalAccessException: java.lang.IllegalAccessException: no private access for invokespecial: class net.imglib2.type.numeric.integer.UnsignedByteType, from org.python.core.PyReflectedFunction

The lack of “private access” puzzles me.

In the end, what I am looking to writing is something like:

img = ... # an IterableInterval
cursor = img.cursor()
t = cursor.next()
cursor.forEachRemaining(t::setOne) # INVALID code, this is a mockup

Any comments appreciated.

Some progress. The surprising error was of my own confusion: turns out, that, of course, Void is the class, and it is Void.TYPE that is the primitive void type (same as e.g. double and Double.TYPE).

Also I had the wrong number of arguments in the mt2 ,method arguments signature.

And the call to generic() was misplaced (so I removed it), as has nothing to do with Consumer.accept having generic type arguments (see https://stackoverflow.com/a/30547064 ).

from java.lang.invoke import MethodHandles, MethodType, LambdaMetafactory
from java.lang import Void
from java.util.function import Consumer
from net.imglib2.type.numeric.integer import UnsignedByteType
from net.imglib2.type.operators import SetOne

caller = MethodHandles.lookup()
mt = MethodType.methodType(Void.TYPE)
target = caller.findVirtual(SetOne, "setOne", mt)
#target = caller.findSpecial(SetOne, "setOne", mt, UnsignedByteType)
fn = target.type()
invokedType = MethodType.methodType(Consumer)
mt2 = MethodType.methodType(Void.TYPE, UnsignedByteType)
site = LambdaMetafactory.metafactory(caller, "accept", invokedType, fn, target, mt2)

consumer = site.getTarget().invokeExact()

t = UnsignedByteType(0)
consumer.accept(t)

print t

Now the error I’m getting is, at line 16 where it says consumer = ...

Traceback (most recent call last):
  File "/home/albert/lab/scripts/python/imagej/tests/test_consumer_from_Lambdametafactory.py", line 16, in <module>
    consumer = site.getTarget().invoke() # Exact()
	at java.lang.invoke.MethodHandle.invoke(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
java.lang.UnsupportedOperationException: java.lang.UnsupportedOperationException: MethodHandle.invoke cannot be invoked reflectively

Turns out this error may have to do with the scripting environment, as shown by googling the error and retrieving similarly reported issues with Groovy, or in debugging environments (e.g. https://stackoverflow.com/a/46773433).

The workaround is to use invokeWithArguments instead of invokeTarget, which works, but now I am getting the following error:

Traceback (most recent call last):
  File "/home/albert/lab/scripts/python/imagej/tests/test_consumer_from_Lambdametafactory.py", line 19, in <module>
    img.cursor().forEachRemaining(site.getTarget().invokeWithArguments())
	at org.python.core.PyReflectedFunction$$Lambda$132/2022280633.accept(Unknown Source)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
java.lang.AbstractMethodError: java.lang.AbstractMethodError: Method org/python/core/PyReflectedFunction$$Lambda$132.accept(Ljava/lang/Object;)V is abstract

… which tells me there’s something fundamental about java 8 lambdas that I haven’t understood yet. Back to reading some more.

Still not there, down to this now:

from java.lang.invoke import MethodHandles, MethodType, LambdaMetafactory
from java.lang import Void, Object
from java.util.function import Consumer
from net.imglib2.type.numeric.integer import UnsignedByteType
from net.imglib2.type.operators import SetOne
from net.imglib2.img.array import ArrayImgs

caller = MethodHandles.lookup()

site = LambdaMetafactory.metafactory(caller, # execution context
                                     "accept", # name of method to implement in Consumer interface
                                     MethodType.methodType(Consumer, SetOne), # return type is the interface to implement, then type of capture variable
                                     MethodType.methodType(Void.TYPE), # return type signature of Consumer.accept, plus no arguments
                                     caller.findVirtual(SetOne, "setOne", MethodType.methodType(Void.TYPE)), # method to invoke on argument of Consumer.accept
                                     MethodType.methodType(Void.TYPE)) # signature of return type of Consumer.accept

consumer = site.getTarget().invokeWithArguments()

t = UnsignedByteType(0)
consumer.accept(t)

… with this error:

Traceback (most recent call last):
  File "/home/albert/lab/scripts/python/imagej/tests/test_consumer_from_Lambdametafactory.py", line 17, in <module>
    consumer = site.getTarget().invokeWithArguments()
	at java.lang.invoke.MethodHandle.asTypeUncached(MethodHandle.java:775)
	at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
java.lang.invoke.WrongMethodTypeException: java.lang.invoke.WrongMethodTypeException: cannot convert MethodHandle(SetOne)Consumer to ()Object

@ctrueden @tpietzsch have you touched java.lang.invoke.MethodHandle and the LambdaMetafactory? Any hints as to what am I missing above?

@albertcardona Everything I know about LambdaMetafactory is written here:

I don’t know how relevant it is to what you’re trying to do, but I hope it might help!

Thanks very much @ctrueden, that makes a wonderful read. but I see that:

… which is the case for the Consumer.accept method. I see that @gselzer addressed your particular issue:

… but I didn’t quite get it above. The irony of it all is that it’s clearer (yet verbose) to write the bytecode directly in ASM:

from net.imglib2.img.array import ArrayImgs
from time import time
from net.imglib2.type.numeric.integer import UnsignedByteType
from java.util.function import Predicate, Consumer, Function
import sys
from org.objectweb.asm import ClassWriter, Opcodes
from java.lang import Object, ClassLoader, String, Class, Integer

class CustomClassLoader(ClassLoader):
  def defineClass(self, name, bytes):
    """
       name: the fully qualified (i.e. with packages) name of the class to load.
       bytes: a byte[] (a byte array) containing the class bytecode.
       Invokes the defineClass of the parent ClassLoader, which is a protected method.
    """
    # Inheritance of protected methods is complicated in jython
    m = super(ClassLoader, self).__thisclass__.getDeclaredMethod("defineClass", String, Class.forName("[B"), Integer.TYPE, Integer.TYPE)
    m.setAccessible(True)
    try:
      return m.invoke(self, name.replace("/", "."), bytes, 0, len(bytes))
    except:
      print sys.exc_info()


def createConsumerInvokeSetOne():
  cw = ClassWriter(0)

  cw.visit(52, Opcodes.ACC_PUBLIC + Opcodes.ACC_FINAL + Opcodes.ACC_SUPER, "my/InvokeSetOne", "<T::Lnet/imglib2/type/operators/SetOne>Ljava/lang/ObjectLjava/util/function/Consumer<TT>;", "java/lang/Object", ["java/util/function/Consumer"])

  mv = cw.visitMethod(Opcodes.ACC_PUBLIC, "<init>", "()V", None, None)
  mv.visitCode()
  mv.visitVarInsn(Opcodes.ALOAD, 0)
  mv.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/Object", "<init>", "()V", False)
  mv.visitInsn(Opcodes.RETURN)
  mv.visitMaxs(1, 1)
  mv.visitEnd()

  mv = cw.visitMethod(Opcodes.ACC_PUBLIC + Opcodes.ACC_FINAL, "accept", "(Lnet/imglib2/type/operators/SetOne;)V", "(TT)V", None)
  mv.visitCode()
  mv.visitVarInsn(Opcodes.ALOAD, 1)
  mv.visitMethodInsn(Opcodes.INVOKEINTERFACE, "net/imglib2/type/operators/SetOne", "setOne", "()V", True)
  mv.visitInsn(Opcodes.RETURN)
  mv.visitMaxs(1, 2)
  mv.visitEnd()

  mv = cw.visitMethod(Opcodes.ACC_PUBLIC + Opcodes.ACC_BRIDGE + Opcodes.ACC_SYNTHETIC, "accept", "(Ljava/lang/Object;)V", None, None)
  mv.visitCode()
  mv.visitVarInsn(Opcodes.ALOAD, 0)
  mv.visitVarInsn(Opcodes.ALOAD, 1)
  mv.visitTypeInsn(Opcodes.CHECKCAST, "net/imglib2/type/operators/SetOne")
  mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, "my/InvokeSetOne", "accept", "(Lnet/imglib2/type/operators/SetOne;)V", False)
  mv.visitInsn(Opcodes.RETURN)
  mv.visitMaxs(2, 2)
  mv.visitEnd()

  cw.visitEnd()

  # Load class
  return CustomClassLoader().defineClass("my.InvokeSetOne", cw.toByteArray())


# TEST:
img = ArrayImgs.unsignedBytes([512, 512, 5])
n_iterations = 5

for i in xrange(n_iterations):
  c = createConsumerInvokeSetOne()
  t0 = time()
  img.cursor().forEachRemaining(c())
  t1 = time()
  print "stream ASM:", t1 - t0 # 10 to 20 milliseconds: ~5 times slower than a clojure loop, and far more verbose

The performance is good, close to a plain clojure or java loop, and 500x faster than e.g. a jython’s imap. What I’m after is avoiding the verbose ASM and using a one-liner using the MethodHandle and LambdaMetafactory, but I can’t get it right so far.

1 Like

Finally, I got it: a java.util.function.Consumer created for net.imglib2.type.operators.SetOne setOne method. The key was the call to generic modifed as @gselzer had indicated to make the void return type work, as well as nailing each of the multiple MethodType argument lists required in the call to LambdaMetafactory.metafactory:

from java.lang.invoke import MethodHandles, MethodType, LambdaMetafactory
from java.lang import Void
from java.util.function import Consumer
from net.imglib2.type.numeric.integer import UnsignedByteType
from net.imglib2.type.operators import SetOne
from net.imglib2.img.array import ArrayImgs


caller = MethodHandles.lookup()
target = caller.findVirtual(SetOne, "setOne", MethodType.methodType(Void.TYPE))
mh = MethodType.methodType(Void.TYPE, SetOne)
site = LambdaMetafactory.metafactory(caller, # execution context
                                     "accept", # name of method to implement in Consumer interface
                                     MethodType.methodType(Consumer), # return type is the interface to implement, then type of capture variable (none in this case)
                                     MethodType.methodType(Void.TYPE, mh.generic().parameterList()), # return type signature of Consumer.accept, plus argument list as Object entries (that's what "generic()" means)
                                     target, # method to invoke on argument of Consumer.accept
                                     mh) # signature of return type of Consumer.accept

setOne = site.getTarget().invokeWithArguments()

img = ArrayImgs.unsignedBytes([512, 512, 5])

for i in xrange(n_iterations):
  t0 = time()
  img.cursor().forEachRemaining(setOne)
  t1 = time()
  print "steam method:", t1 - t0 # 8 ms -- comparable to a clojure loop

The performace is the same, or very close, to a plain clojure loop.

2 Likes