Hey @ctrueden,
sure! I’m happy to join the hackathon. I’m afraid that good auto-completion will be hard to achieve without influencing variable naming - just because of the type-less nature of jython. But maybe there is a way and I just don’t see it 
Otherwise, let’s think about that:
IMHO, it is a de-facto standard that people call variables something like ...Imp but any improvement is welcome.
Who here in the forum has an opinion of how to encourage good variable naming?
Help! I have opinions! 
If we do decide to use variable naming to hint at autocompletion, this could be fine. I’d suggest we model it after names used very commonly in the wild: imp for ImagePlus, rm for RoiManager, img for Img, and maybe in general fooBar for FooBar with adjusted camel case. I find suffixes like foo_imp much less nice (reminds me of Hungarian naming, which some substantial fraction of programmers hate). But I could probably be convinced, especially if we cannot find any other viable solution.
That said, I think @imagejan’s idea of keying off the script parameters is right on, because at least for script parameters, we also do have the types as a starting point. It will be much easier to start with those.
@haesleinhuepf I finally got the chance to test this out. This is very helpful. Thanks for linking the code.
I don’t have a great solution for the naming question beyond when I already posted. At the moment, I defined some names that were convenient. But already ij is really helpful.
Another question came up through. Is it possible to have javadoc show up? In the macro autocompletion I guess you pull the documentation directly from the web. Can you think of an easy way to do that with javadoc?
Hey @karlduderstadt,
I’m pretty sure that there are libraries for parsing Java-doc. We could also think of automatically reading the javadoc online. However, at the moment this project is kind of stuck. We’re looking for volunteers how would be interested in investing time (See message above). If you’re interested and found a library which can parse javadoc, I’m happy to give you directions how to build it into the auto-completion. But keep in mind: python doesn’t play well with types, so it might be hard to guess the right type (and thus Java-class)
Let me know if you need support!
Cheers,
Robert
Hi @haesleinhuepf,
Great initiative, and sorry that I missed it. Thanks for pointing this thread out to me.
I have opinions, strong ones, but as often happens, who does the work gets to choose.
On variable naming and using #@ params: I’d do neither. Instead, I’d prefer light parsing of the code.
Example 1:
from ij import IJ # IJ is class for which we call getDeclaredFields and getDeclaredMethods behind the scenes.
imp = IJ.getImage() # we know that getImage returns an ImagePlus
imp. # autoexpand knows that to do
Example 2:
from ij import ImagePlus # is class for which we call getDeclaredFields and getDeclaredMethods behind the scenes.
from ij.process import ByteProcessor # idem
imp = ImagePlus("one", ByteProcessor(256, 256))
imp. # autoexpand knows what to do
In other words: my first approach to autoexpand would be to scan the script for import statements, load data from the classes found, and then use that as shown above.
The above would be already very useful, and it’s feasible, as it doesn’t try to do everything but already removes most of the pain.
To note that it’s feasible but it isn’t trivial: one has to parse code scopes to find out which of the many “imp” is meant. Which in python code is straightforward, given the use of white space, and it would be acceptable to “give up” when it’s too convoluted and not offer autocompletion then. The point here is to cater to the trivial case, which are the majority.
Another painful point is writing the imports themselves. So for that:
Example 3:
from ij import # here, autoexpand would list all classes in package ij
For the above to work, all it needs is the list of classes in all jar files in Fiji. Finding this out takes 4 seconds on my system, and once cached, it’s essentially instantaneous. See new class ClassUtil in the Script Editor repository.
Unlike examples 1 and 2, example 3 is trivial to implement.
In addition, consider this example as well:
Example 4:
imp = ImagePlus # here, autoexpand would suggest both expanding to class names starting with ImagePlus and auto-adding the import to it at the top of the file
Above, a choice among possible classes to auto-import would be offered, and upon acceptance the import would be added at the top.
It’s also trivial to implement, given the strong convention for upper case in the first letter of a class name. And for static methods, ditto: trivial to load. This helps with e.g. ImgLib2’s Intervals, Converters, Views, RealViews, ImageJFunctions which appear in many scripts.
None of the above examples expand on declared variables, including classes and their methods defined in the python script itself. This could be added later, as these are far less of an issue: the script author just wrote them and knows them. A possibility would be to tokenize the whole script by words, then pattern-match on the last word-like string typed: would not be class-aware (would be more like a plain text editor autoexpand), but would do the job almost always. These could be colored differently in the autoexpand listing.
The key issue is really discovery of classes whose name one only vaguely remembers (e.g. was it VolatileShortArray or ShortArrayVolatile?) and to facilitate their auto-import (remembering the package this.that.then.whoa.long.isn.t.it is even harder).
In other words, solving Examples 3 and 4 would already go a long way.
My 2 cents. I really would like to see the above happen. With my limited time available, it may take me a year, but I’m slowly crawling towards it. The ClassUtil which was useful to fix the “Open Help for Class” in the Script Editor is my first step towards it.
I am sorry if I am interrupting your discussion but I thought it maybe is relevant to point out that since few days ImageJ1 should now also has a Python option in the Recorder: MacroRecorder ImageJ language
Hi @haesleinhuepf et al.,
A first pass at implementing Examples 3 and 4: https://github.com/scijava/script-editor/pull/47
In other words, autocompletion for java classes is on in the Script Editor.
Lots more can be done. It’s relatively straightforward, for now.
Hey @albertcardona,
awesome, that’s so cool. It was maybe mentioned earlier, just in case: Icys Script Editor is also very advanced and it might make sense to take a look at their implementation:
@MarionLouveaux @Stephane Do you know by chance how much customization of the Script Editor was necessary on the Icy side in order to make it work so smoothly? Can you maybe point us to where this is implemented?
Thanks!
Cheers,
Robert
Goodness that looks great. If someone was to pick up the RSyntaxTextArea autocompletion system from Icy and put into Fiji’s Script Editor, that’d be super.
In the meanwhile, I’ll clean up a bit the partial implementation that I’ve created and go with that, which is far better than nothing.
Hi everyone (many people involved in the discussion ^^)
Icy’s script editor heavily relies on libraries (rsyntaxarea, autocomplete and javadoc parser as far i know).
I’m not the original author of this plugin and to be honest it’s quite hard to maintain as the sources are a real mess :-/ But if you really want to look into it, i think the relevant packages are plugins.tprovoost.scripteditor.completion and a bit in plugins.tprovoost.scripteditor.javasource.
I know that the internal script engine does parse plugins JAR files as well than core icy.jar file. The advantage is that we almost always include the sources files into these JAR file (plugins and core) so it can access and parse the javadoc out of it.
Cheers,
– Stephane
On topic, the scijava-commons package offers a DefaultAutoCompleter for programming languages, but Fiji’s Script Editor doesn’t seem to use it. At least, this command:
$ grep -rn "getAutoCompleter" *
returns nothing in the script-editor repository.
at ScriptLanguage.java#L93 we see:
/**
* Gets a helper object capable of generating autocomplete suggestions for a
* code fragment.
*/
default AutoCompleter getAutoCompleter() {
return new DefaultAutoCompleter(this);
}
@MarcH @ctrueden were these auto-completers never implemented, or am I looking into the wrong place?
For the record, I managed to get an autocompletion system for jython (python) in the Script Editor, currently limited to java classes and their static methods.
But the infrastructure is there to include python code AST parsing for variable and python-declared class imports to also autocomplete. Uses the LanguageSupportPlugin approach.
See also: Help wanted to understand the service discovery model in scijava
After some exploration, I found out that the jython jar includes classes such as org.python.core.ParserFacade that enable parsing jython text without executing it. I have an example here:
The above illustrates how one can collect “Assign”-type entries (and others) from the code tree, which could then be used to provide additional autocompletions (beyond the ones currently existing for java classes and their static methods) via the new AutocompletionListener interface (not yet merged: see https://github.com/scijava/script-editor/pull/50 ).
These additional autocompletions would include not only variables within scope as parsed, but also python modules and their functions as included in the jython jar, which is straightfrward to parse (as they express a module as a java package, and its functions each as a lowercase-named class).
One limitation will be that for variables derived from e.g. method calls on a python-defined function arguments one can’t really find out the underlying java class, at least not at all easily (which is why modern python, unlike jython which is at 2.7, facilitates type annotations).
In other words, progress.
@albertcardona, @haesleinhuepf,
Looking into this has been on my wish list for years. So sincere thank you for working on this!
@haesleinhuepf: I looked at your approach in clij. It is great. I tried it, and it was awesome: in minutes I had auto-completion working without much effort using your classes, But, in real-world-usage, i had a hard-time making it useful. The hard-wired keywords, don’t really allow for for much flexibility, and after extending it to all classes in the search path, the auto-completion list became too large to be usable. I think the whole approach is great for IJM, but not sure it adapts well for Jython/Groovy, etc.
@albertcardona, your approach of restricting completions to imports was the most useful for me, so I extended it to support references to instantiated classes, and also to include parameters in the description of completion entries (this part inspired by clij)
Please see: https://github.com/scijava/script-editor/pull/51
What I am a bit confused is about breaking changes described in here: https://github.com/scijava/script-editor/issues/48. I don’t understand how they come about. I would need help trying to understand how to fix them!
Assume you have code like
from net.haesleinhuepf.clij2 import CLIJ2
clij2 = CLIJ2.getInstance()
clij2...
If the auto-completion opens behind clij2 and shows its operations, you fixed the issue 
Maybe you did already?
The answer is maybe?
this currently does not work:
from net.haesleinhuepf.clij2 import CLIJ2
from ij import ImagePlus
clij2 = CLIJ2.getInstance()
clij2.
But that is just because, I only predicted the case of classes being instantiated from a constructor 
. One just needs to support objects being returned from static calls. I added that to a running todo list here.
That apart, completion works well with CLIJ2 (as any other detected class). Example:
The problem is that it is no longer aware of your ‘injection’ mechanism. It uses @albertcardona’s code, and your customizations are nowhere to be seen~?. Is that an issue for you? I have a hard time troubleshooting it, as I must be honest, although I used it, and read your code. I don’t understand ‘the magic’ in the background, and so I’m not really able to fix much there.
Not really. I think the .getInstance() issue is more important. But that should be fixable, no?
I will try to fix it.
Unrelated question for those monitoring this thread:
Should the replacement text in auto-completion include the type of parameters? While I think it they are useful, sometimes they end-up becoming an annoyance, so I have mix feelings about this.
TL;DR: I am asking if for e.g.,: ImagePlus#draw(int x, int y, int width, int height), the auto-completion should be just: 1) imp.draw( or 2) imp.draw(int, int, int, int).
NB: before someone asks: I don’t think it will ever be possible with the current approach to propose ImagePlus#draw(x, y, width, height)
I tried earlier to read out parameter names and never managed. Reason was that the java compiler throws away parameter names. Do you know how to overcome this? Would be amazing!
