Jython Imagej coding advice

jython
imagej

#1

@albertcardona @haesleinhuepf @Alex_H
We have a couple of students here at our institute that started writing jython code for Fiji. One of them is using Fiji script editor, another is using Atom, and yet another is using Spyder. We decided to have a Imagej jython coding session together next week and I was wondering whether there are any recommendations that one could give, e.g., in terms of IDE (or, in fact, anything else)?
Personally, I am an IntelliJ Java person, so not much hands-on jython experience…

I know about this:


#2

i usually use Eclipse with pydev for jython plugins.
but really a lot of time the script editor could have been enough.

here something more about it

hth
Emanuele


#3

I very much recommend the dais Learnathon material by @haesleinhuepf here:


#4

Hi @Christian_Tischer,
Naturally, I use Fiji’s Script Editor. The version that I am using is a bit different–I hacked it together during the EMBL ImageJ & Friends conference–, and IIRC it hasn’t been released it yet.

It looks like this:

Notice the file system tree on the left (with “Options - Save preferences” it will remember your choice of top-level folders), and also the scripting interpreter prompt to the bottom right, enabled by ticking the “persistent” checkbox. What the “persistent” does is to not discard the interpreter used to run the script, and to reuse it when executing commands from the prompt, or from re-running selected text in the script, or the whole script.

The one caveat is that if your dependencies have changed, e.g. other python files, then you’ll have to reset the interpreter by unticking and ticking again the “persistent” checkbox, or manually resetting jython’s cache. The upside is that the result of executing parts or the whole script is there in the interpreter for further use or inspection, which is wonderful.

If you want to try it (adjust paths as needed):

$ git clone https://github.com/scijava/script-editor
$ mvn -Dimagej.app.directory=~/Fiji.app/ -Dmaven.test.skip=true clean install

Notice though this critical issue that I fixed in this pull request: https://github.com/scijava/script-editor/pull/30 Hopefully @ctrueden will merge it soon – UPDATE: @ctrueden has merged it. Should be released soon to the Fiji updater.


#5

Looks great! Does is have auto-completion?


#6

@haesleinhuepf @imagejan @ctrueden @frauzufall @albertcardona
We just did a whole morning of very constructive jython hacking and it was a lot of fun and everybody learned a lot. However, in the end people were afraid that jython is maybe not really the way to go, because it is not very well supported (e.g. still python version < 3 ). What would be your general advice on this? Maybe actually rather use Groovy?


#7

Hey @Christian_Tischer,

I’ve gone through some transition over the years. I started with macro, switched to Jython. Then I started programming Java plugins. Then it became a messy mixture of languages. I had strong issues with Groovy because I failed in very basic things such as creating maps, arrays and lists. Maybe because of my simple mind.

Today, I program re-usable things as Java-Plugins, distribute them via update sites and the project-dependent top-layer is written in macro. So I basically gave up Jython; just because of the lack of good auto-completion. I cannot work without. It’s wasting time. Only microscope control rapid prototyping I still do in Jython using a customized version of the auto-completion shown here. But as soon as possible, I translate the code to Java and wrap it in a plugin.

I hope that helps.

Cheers,
Robert


#8

I personally use a lot the Jython macro language even to make full plugin because I am used to python and its light syntax. I am not a Java guy but I also think of turning my good Jython stuff into Java plugin in the end because it fits better with the rest and the potential for reusability is maybe higher.

My opinion is that data scientist like me are not familiar to Java but very much with Python and with the Jython option you can almost do everything like Java-based plugin.
However there are a number of pitfalls compared to normal python that might really discourage some people from pushing further like when you have modified an imported home-made module, or some bug like this one (almost fixed), but I think there is a real potential for improvement if we raise interest about it.


#9

Major benefit of jython is that, if you already know python, it makes life infinitely easier for basics like reading and writing files, sorting through directories, etc.

While the new (and the old) Script Editor lack auto-completion, I hear that IDEs like pycharm and eclipse can do auto-completion. Building auto-completion for python is hard. For jython, perhaps it is easier when attempting auto-completion on java classes imported within the jython file. Should be easy to hack together, but may end up being not satisfying.

The way I discover APIs is via the jython interpreter (now built into the Script Editor) by using the “print dir(object)” function. Although, most of the time, I also read the source code of the libraries that I use. I never or almost never use javadocs, which are incomplete at best, and often plain wrong.

In addition, a lot of useful API is hidden under “protected” and “private” keywords in many libraries. I understand that’s meant to be for a reason, but often it isn’t a good reason. So I use a lot this too:

object = ...
f = type(object).getDeclaredField("<name-of-field>")
f.setAccessible(True)
f.set(object, <new-value>)

… which from jython is trivial to do. works even for final fields (the JVM doesn’t have the concept of “const” present in other languages; the “final” keyword is only a hint). I even have a utility method in my libraries to abstract this stuff, looking for the <name-of-field> in the hierarchy of classes of the object.

A final point: we built support for many languages in Fiji precisely because one size doesn’t fit all, and the expertise of any individual should be leveraged. If you already know python, using jython is natural. Otherwise, use something else. It doesn’t matter.

Which leads me to another ending thought: if you are building code for others to use, you should use java, or at least package it all into a jar and distribute via an update site. If you use code for yourself or maybe for another lab member who knows e.g. python, then jython (or the scripting language of choice ) will do great.


#10

I’m heavily biased towards Groovy:

  • Groovy runs natively in the JVM, while Jython suffers from slow startup time upon first run
  • Jython can sometimes fail when calling overloaded Java methods, or when matching inputs for ops. In Groovy, you can still cast if necessary.
  • Groovy has almost all the “syntactic sugar” that Python has

In the end you’re mostly calling Java API anyways, so I prefer using Groovy.

If you know Python and want to benefit of all the Python 3 and Numpy/Scipy functionality, use Jupyter notebooks with #pyimagej.


EDIT:

  • Don’t know whether it’s a benefit or a danger, but: in Groovy, you can access private fields of Java objects :slightly_smiling_face:

#11

The sensible thing to do would be to ask upstream. But the quality of support from the key Jython people is not very good, so you might not get an answer: I was on the jython mailing list and IRC channel for a long time, and the activity on those channels was extremely low. However, I know that Jython has/had corporate sponsorship, and that people were working updating to Python 3 for many years, but the Jython 3 git repository looks dead/dormant now for a couple of years.

I :sparkling_heart: Groovy. Don’t get me wrong—Python is also great—but Groovy feels like much of the best parts of Java and Python together in one amazing language.

myList = []
myStringArray = [] as String[]
myMap = [:]
mySet = [] as Set
println(myList.getClass().getName())
println(myStringArray.getClass().getName())
println(myMap.getClass().getName())
println(mySet.getClass().getName())

prints:

java.util.ArrayList
[Ljava.lang.String;
java.util.LinkedHashMap
java.util.LinkedHashSet

Which is really nice: map and set keys remain in the the same order they were initially added, which sometimes makes code much more elegant.

In Groovy, you can access private and protected members without using reflection. Just pretend it’s public, and Groovy does the work under the hood. I find this “Pythonic” in that the Python language has no explicit conception of reduced visibility, only the convention of using _ or __ for members that are internal and/or subject to change. As such, I thought it might work in Jython too, but unfortunately not:

AttributeError: 'my.package.MyClass' object has no attribute 'myPrivateMember'

Groovy also supports many of the most elegant functional programming approaches possible in Python. Here are a couple of examples:

animals = ['lion', 'tiger', 'bear']
animals.each { animal -> println(animal.toUpperCase()) }

or even:

animals.each { println(it.toUpperCase()) }

Combining with the Java 8 stream API works well and is shorter than Java:

iAnimals = animals.stream().map{it.toUpperCase()}.filter{it.contains('I')}.collect()

Which in Java would be:

List<String> iAnimals = animals.stream().map(it -> it.toUpperCase()).filter(it -> it.contains("I")).collect(Collectors.toList());

In Python it would be:

i_animals = list(filter(lambda it: 'I' in it, [it.upper() for it in animals]))

Python “wins” by 6 characters on that one, although really both are very nice and I’m sure some further optimizations are possible depending on your tastes.

One place where I think Groovy wins over Jython is multiple method dispatch. Java supports method overloading—lots of methods with the same name but different argument types—whereas Python does not. Jython tries hard to accommodate that, but sometimes gets confused when deciding which method you intended to call. We have seen this rear its head when invoking methods of the OpService, for example. Whereas Groovy seems to do a better job more often choosing the right method based on the actual argument types given. If people want me to substantiate that with evidence/examples, just ask and I can do some experiments.

Last time I tried it (more than one year ago), PyDev made my Eclipse very slow and crashy. Is it working well for you, @emartini?

My goal is to release it by week’s end. It is a big change, so once it is released, please give it a spin and report any issues. (Tag @albertcardona and @ctrueden here on the forum, or @acardona and @ctrueden on the script-editor issue tracker. Relatedly, Albert, I finally added you as a developer to the script-editor POM. But technically that means you won’t answer support questions, since you don’t have support role. Is that what you wanted?)

Another caveat is that “persistent” mode does not work with script parameters. If anyone has time to track down why not and submit a fix, it would be much appreciated!

I agree that having auto-completion would be super valuable. Some day I want to implement auto-completion at least for script parameter variables at least for Jython and Groovy…


#12

Wow! Thanks sooooo much everybody for their super informative and detailed answers!!!
Probably this is a somewhat important discussion, as it could be good to kind of agree what to teach the next-gen image analysts :slight_smile: !


#13

From the perspective of teaching people what will be useful in the future, I agree with @LThomas that there is good synergy between Jython-in-ImageJ and Python in general. The looming issue is that Python 2 is increasingly going away, so there will be increasing frustration as people only familiar with Python 3 try doing stuff in Jython and find it doesn’t work. I hope this creates pressure for some developers to complete the Python 3 version of Jython, but I don’t know the politics of that; if no one steps up then Jython may simply molder instead. Even if a Python 3 version of Jython is created, there is still the issue of native extensions like NumPy not working; hopefully JyNI moves forward to address that, but it’s not clear at this point.

I have been working on the use ImageJ from actual Python angle lately to avoid this issue. My hope is that some day, Fiji comes bundled with a Python interpreter—and/or is able to link with an existing Python installation—such that you can script real Python from inside the Script Editor, and use all the goodies data scientists have come to expect, combined with the goodies provided by ImageJ and the entire Java ecosystem.

The real solution is for everyone to just learn every language. Right? :stuck_out_tongue_winking_eye:


#14

In fact, considering exactly this, and reading through all the arguments I am leaning towards giving this advice:

If you want write scripts for Java, you should use Groovy, because that is the best scripting language for Java. By doing this properly, you will learn how write good Groovy code. This should make it quite be easy for you to, later, also learn how to write good Python code. Because, in the end, writing good code shares many similar concepts, across languages.


#15

Also extremely great (in different ways) is Clojure. But IMHO less approachable for new people than Groovy and Python are. And then there’s Kotlin, which is really making waves right now. And let’s not forget Scala, another very powerful and well-architected language!

Not trying to muddy the waters further here—just pointing out that it’s probably advisable not to call any scripting language the “best” but rather just stress Groovy’s strengths: easy (easier than Java) to learn, elegant, powerful, strongly-but-dynamically typed, able to load and import dependencies dynamically at runtime, and maybe most importantly, WELL TESTED in an ImageJ context!

:+1: :+1:


#16

I did add a half-broken implementation of autocompletion to the Jython Interpreter years ago.

I will try now to do a better one for the Script Editor. It should be fun–lots of regex and String processing, given the inability to parse the code without executing it (at least, I wasn’t able to just yet).


#17

On the subject of autocompletion in jython: it could be done quite well if one was to provide type hints for function arguments and function return types, a la mypy: https://mypy.readthedocs.io/en/latest/python2.html#type-checking-python-2-code

Writing python code in that way is done sometimes anyway for documentation, so one might just as well take advantage of it to enable code autocompletion.

By the way @ctrueden, I have a half-baked python autocompletion system created, based exclusively on parsing the jython source text. It has limitations, e.g. function arguments and their return types, that could be autocompleted for method names using the above approach (not yet done). In any case, the question is where to place such a file? I’d add it to the Script Editor, but it’s all abstracted into services so perhaps you’d want this abstracted into an autocompletion service of sorts? I can’t think in terms of services, but of compile-time dependencies, so I don’t know how to fit this language-specific feature into the Script Editor in a way that you’d want it.


#18

How about making it part of https://github.com/scijava/scripting-jython? Does it require any new dependencies?


#19

Hi @ctrueden, the autocompletion for jython is self-contained (and rather simple and limited). If I put it in scijava/scripting-jython, how do I use it from scijava/script-editor without a compile-time dependency?


#20

Hi @haesleinhuepf,
Turns out a code completion library exists for jython, written in java: https://github.com/Yaqiang/editorjy/tree/master/src/main/java/org/meteothink/editor/codecomplete

I haven’t tested it, but it seems to be part of a larger library that includes a code editor. Looks like it should be possible to provide a proper python+java autocomplete in a jython editor, and that’s wonderful.

Of course, for function/method parameters, one would have to annotated them with e.g. mypy-style comments or hints, which as far as I know is not (yet) supported by the above library.


PlotJy library for Jython