How to modularize ImageJ, BoneJ and Fiji

The following conversation is transplanted from bonej-org/LocalThickness@6289c5b4:

@mdoube commented on 6289c5b 8 days ago

On 25/11/15 18:20, @ctrueden wrote:

My suggestion right now would be for BoneJ to be an update site on top of ImageJ—and maybe even on top of Fiji. In which case, there is no disagreement: LocalThickness is part of the Fiji layer. And BoneJ-specific plugins not present in Fiji can live use org.bonej for their package prefix.

This seems eminently sensible. The main thing for BoneJ is that if someone has vanilla ImageJ2 and adds only the BoneJ update site, all the dependencies, no matter what their namespace, are pulled in. And, if someone has a very full installation with all the trimmings, adding BoneJ doesn’t cause dependency hell or namespace collisions.

So if it doesn’t really matter beyond a convention that helps us to organise the projects and keeps things unique and discoverable, then I’m OK with sc.fiji for general plugin code and with org.bonej for Bonej-specific code.

@ctrueden commented on 6289c5b 23 hours ago

The main thing for BoneJ is that if someone has vanilla ImageJ2 and adds only the BoneJ update site, all the dependencies, no matter what their namespace, are pulled in.

To accomplish that, you’ll have to upload all of BoneJ’s components to the BoneJ update site directly. In other words: there will then be duplicate(ish) stuff on the Fiji and BoneJ update sites. This is necessary because there is currently no mechanism for update sites to declare dependency on other update sites.

How important is it to you that BoneJ be usable independent from Fiji? I.e., that it not be “built on top of Fiji”?

If the answer is “very important” then we might want to create some shared org for “stuff built on top of ImageJ2 that is not core ImageJ2 but is common to multiple distributions of ImageJ” (in this case BoneJ and Fiji). And then it can have its own package prefix etc.

Thoughts?

@mdoube commented on 6289c5b 4 hours ago

The main thing for BoneJ is that if someone has vanilla ImageJ2 and adds /only/ the BoneJ update site, all the dependencies, no matter what their namespace, are pulled in.

To accomplish that, you’ll have to upload all of BoneJ’s components to the BoneJ update site directly. In other words: there will then be duplicate(ish) stuff on the Fiji and BoneJ update sites.

OK, that’s awkward. The whole point of the BoneJ2 project is to avoid duplication and maximise reuse.

This is necessary because there is currently no mechanism for update sites to declare dependency on other update sites.

Perhaps then such a mechanism should be implemented? I can volunteer our services to help with that. I still have in mind an APT/YUM-ish view of dependency management, whereby the actual libraries are devolved downwards to being shared resources that any project can declare a dependency on. Then the responsibility becomes making sure that the shared resource (let’s say, a linear algebra library) and its clients don’t diverge or create some conflicts among themselves, by requiring different versions of the library for example.

How important is it to you that BoneJ be usable independent from Fiji? I.e., that it not be “built on top of Fiji”?

Very important. Fiji is a bit bloated, insufficiently granular in its modularity and very much targeted towards life sciences. It has a pretty poor track record of producing major breakages for users, evidenced by loud squeals on the mailing lists usually after an update has been applied. I don’t want BoneJ users to experience that level of risk (a risk they are exposed to by having installed a whole lot of stuff they don’t use, but which has the potential to take down their system). Many (most?) of BoneJ’s users are not doing light microscopy - usually X-ray based techniques. And many of them are using images of soil, rock and engineering materials. So tying these users into a ‘life-sciences distribution’ of ImageJ, with tons of irrelevant stuff, is not ideal.

If the answer is “very important”

It is.

then we might want to create some shared org for “stuff built on top of ImageJ2 that is not core ImageJ2 but is common to multiple distributions of ImageJ” (in this case BoneJ and Fiji). And then it can have its own package prefix etc.

That’s an elegant way to deal with it. Users having the freedom to easily select groups of functionality (gimme all the segmentation tools), or from particular groups (gimme a couple of segmentation tools) could be helpful. As well, pre-selected / curated collections of functionality (as Fiji is, essentially), could be selected - gimme the massive life sciences meta-package (Fiji) AND gimme special bone tools (BoneJ). Or, just gimme the bone-specific tools, and a few of those awesome segmentation tools, but not the rest of the stuff that comes with Fiji.

What this means is that all of the resources can be freely combined into novel packages suiting different user domains. The challenge is to make sure all the shared resources and their clients play nicely together as they all develop. One of our longer-range objectives is to help re-wrap some of BoneJ’s functionality into bundles that are suitable for plant and soil scientists, for example. So, we’d need the functionality to be able to be mixed and matched and wrapped according to the needs of the particular user community.

I would rather switch to an industry standard update mechanism. But it is a lot of work. We do have an issue for it though: imagej/imagej-ui-swing#47.

Also related:

That would be absolutely wonderful. But I urge extreme caution pursuing this—it is a TON of work. There is no way the LOCI team has time to rework the update mechanism any time soon, but if @rimadoma wants to tackle it, then that would be amazing. Personally, I think your best chance for success (long-term) is to adopt an existing industry-standard update scheme like Eclipse update sites… but those may have issues which make them inappropriate for the ImageJ developer community, which has historically demanded extreme flexibility—and smallest possible learning curve—in how it develops and distributes extensions.

The good thing is that we already require SemVer for all core ImageJ and Fiji components, so that is a huge step in the right direction. But as @axtimwalde is fond of pointing out: SemVer is not a panacea for version skew issues, even when developers exercise perfect diligence (which they don’t).

While I understand where you are coming from, I think these are critiques are rather unfair:

  • Bloated. Yes, Fiji comes bundled with a lot of stuff. But the point is to avoid exactly the version clashes you describe above, while making it as easy as possible for users to get up and running. As long as the menus are well organized, why is it such a large problem for there to be extra “parts on the table”?

  • Insufficiently granular. Really now. I have worked very hard for the past 6 years to make ImageJ and Fiji modular enough to meet developer needs as the community moves forward. Fiji has more than 130 components, with each plugin (for the most part) in its own git repository in the fiji, bigdataviewer, trakem2 and other orgs. If there are cases where code is not well-separated enough (e.g., VIB), we would be happy to separate the components out further—just speak up.

  • Very much targeted towards life sciences. Actually, more and more, we are leaning toward dropping the “targeting the life sciences” stated goal of Fiji in favor of just “batteries included distribution of ImageJ.” I see little point in the life sciences “focus”—when in actuality most of the plugins have almost nothing to do with life sciences specifically. The central benefit of Fiji is actually technical/convenience: it is a single download of many useful components that are all tested against each other, to avoid version skew breakages.

    At its heart, I see Fiji as a “flagship” distribution of ImageJ, providing a central integration point for the community to contribute their work in a way that easily reaches a large user base. Check out the Fiji contribution requirements page if you haven’t already.

  • Major breakages. In its defense, Fiji has been insanely ambitious in trying to supporting updates from the inception of the distribution in 2008, all the way through now. It is an incredibly hard problem to allow arbitrary direct upgrading from any point A in the past to the current point in time, and it has made it very difficult to remove any old code in core components, ever. Many of those past breakages have been my fault personally as I push to continue iterating on ImageJ2 core components.

    In contrast, look at how Eclipse does things: they release a new major version every year which you must download again from scratch—you cannot upgrade e.g. 4.4 Luna to 4.5 Mars. This is a much less crazy way of doing things, since each year you get a “fresh start” so to speak when it comes to plugins and the updater. And their updater is much, much more robust than ImageJ2’s.

The bulk of the risk is not posted by the Fiji components—most of those are simple IJ1 plugins that have no impact on the application dynamics. Rather, much of the risk comes from the SciJava core itself, and from extensions which modify that core behavior. The SciJava application context startup is too fragile right now and needs improvement—see in particular imagej/imagej#97. Addressing that issue would be a massive step toward reducing future startup breakages after update.

Also, as stated above, the Updater is quite fragile. Overhauling it would be another major step forward.

I absolutely agree with this vision. I love how Icy does it, where you just cherry-pick the plugins you want, with none (or at least less) of this “update site” stuff. Add in metapackages (i.e., groups/collections of plugins) and you have exactly what you describe above.

Actually, the current updater’s file-based scheme already allows that—you just have to know the filename of the JAR you want, and flag it for installation. By default, when you tick an update site, the updater tries to install all the stuff from it. But you can instead cherry-pick only the things you want. It’s just that the UI for doing it is not the friendliest. So another less development-heavy option here would be to simply enhance the updater’s UI to look prettier and appear more “plugin-centric” and less “file-centric”—even though those two things are largely the same in the vast majority of cases. If each plugin had a shiny icon, which the updater displayed in a grid or something, it might go a long way toward user happiness.

The Mavenization of ImageJ and Fiji was designed to allow exactly that. On the developer side, it’s already done. It’s just a question of how easy you want to make it for users to cherry pick using a friendly point-and-click UI.

1 Like

We’re already making fast progress in unbundling BoneJ1’s old plugins so this is something we can keep an eye on and perhaps could be a topic for discussion when @rimadoma comes to visit LOCI next year.

Thank you for clarifying. There has certainly been a lot of progress since the early days when it was simply a bundle of stuff to make TrakEM go. By granularity I meant from the user perspective - you download a huge thing (213 MB w/o JRE) and get it all. On the developer side - a lot to be happy about, for sure. We wouldn’t be embarking on BoneJ2 without all your work which allows it.

Fiji’s landing page says, 2nd paragraph, “the main focus of Fiji is to assist research in life sciences”, and I know that a lot of the core developers are in, or associated closely with, groups doing biology with light (and TEM) microscopes. So maybe the docs are getting out of date? @rimadoma - do you have a WIki account to help with documentation?

If they were just passively there, under the table tidily away in a drawer, it wouldn’t be a problem at all. But they’re not - the updater will dutifully update everything, including all the stuff you don’t use. And sometimes, that has been known to break your system. It usually takes a few minutes to checksum and download the new versions of the things you don’t use and install them, which is time and bandwidth wasted. You can turn off updates, but then you don’t get updates of the things you do use.

Not such a bad idea, perhaps? Maybe it would help with reproducibility too, because projects could be attached to specific versions of software more readily.

If all it would take is a bit of rejiggling of the Updater UI then that would be fantastic in terms of utility for users, and we are definitely willing to contribute effort to make that work.

As easy and flexible and robust as possible - so that metapackage maintainers can easily make a nice bundle of things with some sensible wrappers and specialised custom stuff (e.g. BoneJ’s domain specific column headings that use standardised nomenclature), and users can easily choose from what is on offer. My personal experience here is strongly influenced by the UIs and backends employed by Fedora and Ubuntu in the last 15 or so years for package selection. You can select individual libraries, but usually you select some higher-level thing like GIMP, which then makes sure that the image handling libraries it needs are present and correct.

@mdoube

Yes, I have a wiki account. I’ll start adding knowledge as soon as I’ll have some.

1 Like

And probably best to start small, rather than the 2nd paragraph of the landing page.