Writing ops in Jython

radoslaw.ejsmont · June 29, 2017, 2:31pm

Hi,

has anyone tried to actually write an ImageJ op in Jython? Since it appears that ImageJ Jython interpreter can handle Java annotations, do I need to wrap my python class in Java class?

I have tried the following, but it did not work:

# @PluginService plugin
# @OpService op

from net.imagej.ops import AbstractOp
from net.imagej.ops import Op
from org.scijava import ItemIO
from org.scijava.plugin import Parameter
from org.scijava.plugin import Plugin
from org.scijava.plugin import PluginInfo


# @Plugin(type = Op.class, name = "narf")
class Narf(AbstractOp):

    # @Parameter input
    # @Parameter(type = ItemIO.OUTPUT) output

    def run(self):
        output = "Egads! " + input.upper()

narfInfo = PluginInfo(Narf.__class__, Op.__class__)
plugin.addPlugin(narfInfo)
result = op.run("narf", "Put some trousers on")

Traceback (most recent call last):
  File "/Users/radoslaw.ejsmont/Desktop/op.py", line 23, in <module>
    result = op.run("narf", "Put some trousers on")
	at net.imagej.ops.DefaultOpMatchingService.assertCandidates(DefaultOpMatchingService.java:235)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:93)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:84)
	at net.imagej.ops.OpEnvironment.module(OpEnvironment.java:252)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:135)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: No candidate 'narf' ops

	at org.python.core.Py.JavaError(Py.java:546)
	at org.python.core.Py.JavaError(Py.java:537)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:188)
	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
	at org.python.core.PyObject.__call__(PyObject.java:496)
	at org.python.core.PyObject.__call__(PyObject.java:500)
	at org.python.core.PyMethod.__call__(PyMethod.java:156)
	at org.python.pycode._pyx8.f$0(/Users/radoslaw.ejsmont/Desktop/op.py:23)
	at org.python.pycode._pyx8.call_function(/Users/radoslaw.ejsmont/Desktop/op.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:167)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1386)
	at org.python.core.__builtin__.eval(__builtin__.java:497)
	at org.python.core.__builtin__.eval(__builtin__.java:501)
	at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at org.scijava.script.ScriptModule.run(ScriptModule.java:159)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:167)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:126)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:65)
	at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:237)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Is this even possible? Should I rather port my script to Java?

Cheers,

Radek

imagejan · June 29, 2017, 2:43pm

Not in Jython, but in Groovy :

#@OpService ops
#@PluginService ps

import net.imagej.ops.Op
import net.imagej.ops.AbstractOp
import org.scijava.log.LogService
import org.scijava.plugin.Parameter
import org.scijava.plugin.Plugin
import org.scijava.plugin.PluginInfo

@Plugin(type = Op.class, name = "myop")
class MyOp extends AbstractOp {
    @Parameter
    String text

    @Parameter
    LogService log

    @Override
    void run() {
        log.error(text)
    }
}

ps.addPlugin(new PluginInfo(MyOp, Op))

ops.run(MyOp, "Hello")

You can write an Op using any scripting lanugage, but I’m not sure about the parameter annotation for Python.

The goal is to make them first class citizens so you can put them into your scripts directory and they will be discovered at startup. See this issue:

https://github.com/scijava/scijava-common/issues/261

and the linked discussion on gitter:

radoslaw.ejsmont · June 29, 2017, 3:39pm

Well, the thing seems to be that

@Plugin(type = Op.class, name = "myop")

annotation seems to be ignored in Jython. I know it works with Groovy. But if I have to write in Groovy, I would just go ahead and write it in Java

Probably because there is no API to import annotations i Jython

imagejan · June 29, 2017, 4:06pm

Hm, there’s an interesting post about this here:

Well, Groovy is weakly typed and has a lot of syntactic sugar similar to Python. I would consider converting a Python script to Groovy much simpler (and resulting in shorter code) than to Java

But as mentioned by @ctrueden, it’s possible that at some point in the future we can simply write:

#@op(name = "narf")
#@String input
#@OUTPUT String output

def run():
     output = "Egads! " + input.upper()

… and this Op will be runnable by the OpService.

If you describe your use case in more detail, I’m sure it would help guiding the future development such that it’ll help you achieve your goals.

radoslaw.ejsmont · June 30, 2017, 1:33pm

Dear Jan,

Thanks for info. I am not strictly bound to jython, so I guess a switch to groovy would be an option. I insisted to use Jython because I plan to use numpy/scipy for further data analysis. And with stuff like http://jyni.org coming on the horizon I would love to have an all-in-one solution.

Regarding my application, in big picture I am working on a new iteration of this https://github.com/HassanLab/nuclearP but now with imageJ2 API (cause my imaging setup has changed and I need to change some things, so why not rewrite it better).

Currently, due to changes in imaging setup I actually have multiple (5) separate stacks of the same volume, each in a different channel. One stack (with nuclear marker) is 2xXYZ resolution than the others, for better segmentation. What I wrote now is a pipeline to take multiple stacks of the same volume, rescale all of them to the same size, taking into account user-supplied channel Z-offsets (cause I do have Z-shifts between channels ), merge them, set lookup tables and correct histogram. And while the script works, I do have certain operations that would make nice and useful ops IMHO, so I wanted to opify them.

Cheers,

R.

radoslaw.ejsmont · June 30, 2017, 2:19pm

Actually, it might work even in Jython. Java annotations are kind of interfaces. And Jython can implement them! So what I am trying to do now is to write Jython classes that one could use instead of annotations to register Jython ops. won’t work automagically, but should work in script context. I am able to supply jythonic @Plugin annotation to PluginService.addPlugin:

from net.imagej.ops import AbstractOp
from net.imagej.ops import Op
from org.scijava.plugin import Plugin
from org.scijava.plugin import PluginInfo
from org.scijava import UIDetails;
from org.scijava import Priority;

class NarfPlugin(Plugin):

    def type(self):
        return Op.__class__    
    def name(self):
        return "narf"
    def label(self):
        return ""
    def description(self):
        return ""
    def menuPath(self):
        return "";
    def menu(self):
        return [];
    def menuRoot(self):
        return UIDetails.APPLICATION_MENU_ROOT;
    def iconPath(self):
        return "";
    def priority(self):
        return Priority.NORMAL_PRIORITY;
    def selectable(self):
        return False;
    def selectionGroup(self):
        return "";
    def enabled(self):
        return True;
    def visible(self):
        return True;
    def headless(self):
        return False;
    def attrs(self):
        return [];

class Narf(AbstractOp):

    input = "test"
    output = ""

    def run(self):
        output = "Egads! " + image.toUpperCase()
        return output

narfInfo = PluginInfo(Narf.__class__, Op.__class__, NarfPlugin())
plugin.addPlugin(narfInfo)

Now need to figure out how to handle IO, cause this is not working yet!

tarted op.py at Fri Jun 30 16:17:17 CEST 2017
Traceback (most recent call last):
  File "/Users/radoslaw.ejsmont/Desktop/op.py", line 60, in <module>
    result = op.run("narf", "Put some trousers on")

Request:
-	narf(
		String)

Candidates:
1. 	org.python.proxies.__builtin__$Narf$1()
	Too many arguments: 1 > 0

	at net.imagej.ops.DefaultOpMatchingService.singleMatch(DefaultOpMatchingService.java:433)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:98)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:84)
	at net.imagej.ops.OpEnvironment.module(OpEnvironment.java:252)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: No matching 'narf' op

Request:
-	narf(
		String)

ctrueden · June 30, 2017, 6:01pm

@radoslaw.ejsmont I very much appreciate your enthusiasm. But I think an easier way to go here will be a way to express scripts as ops in the framework. Needing to implement the annotation interfaces as you have above is pretty unfortunate. And then you would need to have something similar in place for @Parameter fields, since ops determines the inputs and outputs via that annotation. We already have script parsing that generates ModuleInfo objects from scripts without the need for actual Java annotation objects; let’s try to extend that so that scripts can also be ops, rather than only modules.

What is the timeline for your work? Can it wait a couple of months? We will be hacking heavily on improving the ops infrastructure in September.

radoslaw.ejsmont · July 3, 2017, 3:31pm

Hi Curtis,

Thank for your comments. Unfortunately, I most likely won’t be able to wait until September. In this case, I will probably migrate to groovy then. But I would like to make a few comments / suggestions about the Op framework.

It seems, from several threads I found on the web, that Jython is a second-class citizen on the JVM. Groovy, Scala and Clojure support annotations natively. It is a pity that Jython lacks this support - but this hopefully will get resolved upstream. I have found yesterday some Jython packages that claims to add annotation support - https://pypi.python.org/pypi/Jynx/0.4.2-jython or https://github.com/rotsenmarcello/jython-annotation-tools. I have not tested it but will give it a try and come back to you with results.

Regarding scripting framework. I love the work you did with the parses and generation of ModuleInfo from pseudo-annotations for DI and IO setup. The problem I see with this framework in case of ops is that AFAIK those annotations are not contextual and are applied on the script scope. This limits Jython Ops to one op per file. And also excludes combining ops and bussiness-logic in a single file. This is hardly Pythonic. Normally I would put a bunch of ops in a single python file. On the other hand, I am aware that contextual parser would need to be language-specific and require much more work to implement. Though if you were thinking of that - I suggest Doctrine Annotations PHP package (https://github.com/doctrine/annotations) as an inspiration. I do not know how do you intend to implement Op discovery - would you search for user op files in a defined folder? Would you rather scan the whole classpath for files annotated with @Op?

Cheers,

Radek

ctrueden · July 10, 2017, 8:52pm

Thanks for your comments, @radoslaw.ejsmont. Much appreciated.

Great. It is very helpful when people track technologies which interest them, and suggest improvements and additions to ImageJ with respect to them. It is too much for me to track everything.

The ScriptProcessor framework in general is a simple preprocessing framework which works line by line. In general it is possible to implement annotations which are contextual—i.e., which behave differently based on which lines come before or after. It’s just that none of the current script processor plugins needed to do it until now.

So I think defining multiple ops sequentially in the same file will be feasible, if we tweak how the #@ parameter script processor behaves when multiple sets of #@ parameters are defined in different partitions of the code (as divided by e.g. #@op or something). Definitely merits further thought and consideration.

radoslaw.ejsmont · July 20, 2017, 4:17pm

Good news - I have tested them - bad news - it seems that either I was doing something very wrong or annotations were ignored. I am judging by the fact that annotated Jython Op module does not get registered…

ctrueden · July 20, 2017, 4:41pm

It may be tricky in this case, because the annotation needs to be noticed by javac at compile time, so that a META-INF/json/org.scijava.plugin.Plugin file can be generated and lumped into the resultant JAR archive. I am guessing that while the Jython annotation thingie lets you declare the right annotation, the code is still not run through APT (Java’s Annotation Processing Tool). So no metadata file is generated, and thus no plugin magic.