Who populates SciJava's ObjectIndex?

I couldn’t find where and when scijava’s ObjectIndex is getting populated and what the available mechanisms are. Any hints that could bring me back on track? May be I am looking for the wrong thing, so here is what I am doing: I want to discover all classes that have a specific annotation and then set up a pool of class-specific handlers. Currently I am using this library which works great https://github.com/lukehutch/fast-classpath-scanner but the first thing it does is a full sweep over the entire class-path collecting everything possible. Of course this is happening only once or rarely and so may be just fine. But I had the impression that Opsmust be doing something similar and tried to find out what. Digging into it, I ended up at ObjectIndex which is just a Collection and I cannot find who populates it.

One of the core features of SciJava Common is its annotation indexer, which was originally forked from the SezPoz library.

The process is as follows:

  • You put a special annotation (e.g., @Plugin) on a class.
  • A special annotation processor notices this annotation and generates a bundle of JSON containing the attributes of that annotation, which it stores into META-INF/json/org.scijava.plugin.Plugin, where org.scijava.plugin.Plugin is the fully qualified name of the annotation used.
  • At runtime, these JSON bundles can be quickly picked up and indexed without needing to eagerly load the annotated classes.

Code links:

Ops does nothing special with respect to plugins—it just uses SJC’s plugin mechanism. This mechanism was designed to be very fast: you can discover all available plugins on the classpath, including all metadata in the @Plugin annotations, without even loading the classes. You can also invent your own annotations with arbitrary metadata (as long as it is expressible in Java annotation form, which is admittedly a bit limited), and they will be generated at compile time, and accessible quickly at run time without loading the annotated classes.

I want to discover all classes that have a specific annotation and then set up a pool of class-specific handlers.

Would you please consider using SciJava Common’s plugin framework and/or annotation processor for this? That’s why we invented it.

Thanks @ctrueden! I will of course use SciJava Common’s plugin framework if it turns out to be the best available solution. That is why I was asking here and why I was trying to find out how it works in the first place. I generally like it much better to index annotations at compile time than at runtime (which in my case would be somewhere near JVM start time). What I do not understand and what I cannot find is how and where the indices of independent jar files are getting merged. Could you give me a pointer?

There is a lot at stake here. The entire ImageJ2 ecosystem of plugins is built around this mechanism. If you find any better existing solution, let’s discuss whether it could be adopted under the hood to improve the entire software stack. Or if the SciJava plugin mechanism is the “least bad” but has certain deficiencies, let’s discuss how to address those deficiencies.

I understand that it is tempting to simply cherry-pick all the most technically suited tools for each individual project, but there are serious social consequences to the inconsistency that doing so creates. We need to consider not just the in-the-moment technical issues, but what developers new to the ecosystem see when they start exploring the available tools. ImgLib2 and Java in general already have a reputation for being “too complex”, especially compared to the Python numpy/scipy/scikit-image stack. As insiders, you and I know the added value that ImgLib2, BDV, etc., offer which justifies that complexity. But most developers will not have such insight, and seeing e.g. multiple dependency injection frameworks in use across different components will certainly (and rightfully so, IMHO) add to that perception.

Each JAR file contains its own plugin metadata. These get “merged” only insofar as the indexer calls ClassLoader#getResources and iterates on all results over the whole classpath.

The only time the JSON files get merged explicitly is when creating an uber-JAR. SciJava Common provides an AnnotationCombiner class that can be used to aggregate all the JSON files into one. See also this issue for some relevant discussion.

1 Like

@ctrueden Unfortunately, I haven’t yet been able to get this to work. Probably missing some crucial understanding. Here is the diff between what I am doing with https://github.com/lukehutch/fast-classpath-scanner and my unsuccessful attempt to do the same with scijava:

I am not seeing a JSON file in the target jar which I assume explains that the index remains empty. How do I get this JSON index? Simply calling mvn package or mvn install does not do it.

Thanks as always,

Found it, I was missing the @Indexable annotation in my custom annotation. Now it works :). In case that you have time to have a look, is this the expected solution?


Sorry for the delay in reply. Yes, the @Indexable annotation is needed. Glad you figured it out.

As an aside: If you want the @Plugin annotation be erased (so it’s not present in the bytecode, but only the metadata is generated at compile time), you can put @Retention(RetentionPolicy.SOURCE) instead. This is useful e.g. if you want to eliminate runtime dependence on the component housing the CompressionType class.