Split BOMs from parent configuration

maven
scijava
imagej
pom

#1

As I mentioned in the thread Maven 3.4.0 may break ImageJ and Fiji project builds, our current POM structure may not work as intended in future versions of Maven. We are relying on a subtlety of the POM hierarchy which was never documented: the fact that (in Maven 3.3.9 and earlier) managed versions from inheritance trump those from import scope in dependencyManagement. Maven 3.4.0 may change this behavior. And regardless: this fact was never officially documented nor guaranteed. It would be best to change our POM structure to avoid this uncertainty.

Therefore, I think we need to split out the <dependencyManagement> of each organization’s POM parent into a dedicated BOM which can be imported downstream. But as usual, there are complications and open questions.

Requirements

  1. A common “community base POM” which non-core SciJava-based projects extend, which provides useful plugin configuration.

  2. A common “project POM” for each layer (i.e.: scijava, imglib, imagej, scifio, fiji) which core projects of that layer extend, which provides not only the plugin configuration, but also metadata common to many/all components of that layer. This typically includes sections such as <organization>, <licenses>, <mailingLists> and <developers>. The important thing to note is that third party (i.e., non-core) projects built on the layer in question will probably not want to inherit this configuration. (Maybe these project POMs extend the community base POM, but maybe not; see options below.)

  3. For each layer, a BOM defining compatible component versions of that layer.

  4. An uber-BOM—a.k.a. “BOM of BOMs”—defining which versions of all layers play nicely together. Without this, there is no way to know e.g. which versions of net.imagej:imagej and net.imglib2:imglib2-algorithm-gpl should be used together, since neither depends on the other.

  5. NEW: No dependencyManagement inherited from a parent clashing with dependencyManagement obtained via import scope. This probably means that either: A) the community base POM (1) above cannot have any dependencyManagement whatsoever; or B) the community base POM imports the uber-BOM (4) to provide all the management, without requiring consumers to import anything. (My proposal below opts for the latter, to make things as easy as possible for consumers.)

The current structure

As described on the Architecture page, the current structure:

  • Has one base POM per layer, which fulfill requirements (1), (2) and (3) above.
  • Meets requirement (4) using a rather hacky approach: pom-fiji defines pom-bigdataviewer.version and pom-trakem2.version constants which identify the recommended versions of those respective BOMs to import. Then the Fiji project itself (sc.fiji:fiji) extends pom-fiji and imports pom-bigdataviewer and pom-trakem2.

With this scheme, any other component besides Fiji which wants to consume a mixture of BDV and TrakEM2 components (and/or any other future Fiji sub-layers with their own GitHub org + BOM) needs to repeat the boilerplate XML of importing multiple BOMs.

New proposed structure (option A)

The current scheme has issues:

  • It does not fulfill our new requirement (5), because it has overlap across layers regarding managed component versions—which allows potential “version skew” to creep in.
  • It does not fulfill (4) in a very elegant way, as explained above.
  • It conflates version management of the imagej, imglib and scifio layer components all in pom-imagej. This violates PoLA, with pom-imglib2 and pom-scifio both extending pom-imagej, despite the fact that ImgLib2 is a separate project from ImageJ.

To address these issues, I propose the following new structure:

  • Individual BOMs (3)
    • bom-scijava - version management for components in the scijava organization
    • bom-imagej - version management for components in the imagej organization
    • bom-imglib - version management for components in the imglib organization
    • bom-scifio - version management for components in the scifio organization
    • bom-fiji - version management for components in the fiji organization
    • bom-bigdataviewer - version management for components in the bigdataviewer organization
    • bom-trakem2 - version management for components in the trakem2 organization
    • etc.
  • BOM of BOMs (4)
    • bom-umbrella - combined version management for all components of the managed SciJava ecosystem; community can request to have their BOM included here; imports all the individual BOMs above
  • Community base POM (1)
    • pom-base - the One True POM ™ that all community projects extend; extends bom-umbrella (NB: it needs to extend, not import, so that overridding of version properties in consumer POMs keeps working as desired.)
  • Core project POMs (2)
    • pom-scijava - for projects in the scijava organization; extends pom-base
    • pom-imagej - for projects in the imagej organization; extends pom-base
    • pom-imglib - for projects in the imglib organization; extends pom-base
    • pom-scifio - for projects in the scifio organization; extends pom-base
    • pom-fiji - for projects in the fiji organization; extends pom-base
    • pom-bigdataviewer - for projects in the bigdataviewer organization; extends pom-base
    • pom-trakem2 - for projects in the trakem2 organization; extends pom-base
    • etc.

This design:

  • Allows components at any managed layer to depend on components at any other managed layer freely, without worrying about version skew, because all core project POMs extend the community base POM, which manages the known universe.
  • Allows third party components to import only those BOMs which it needs, as well as third party BOMs outside the known universe.
  • Eliminates the current conflation between the imagej, imglib and scifio organizations.
  • Limits the number of components which must be released when low-level components are updated (se next section).

Release cascades

Another issue the current scheme has is that releasing components creates a chain of releases—or “release cascade”—which can be obnoxiously long.

Let’s suppose that the spim_data component, currently part of the bigdataviewer organization, needs to consume a bug-fix from scijava-common.

How it works now

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within pom-scijava. Cut new release of pom-scijava.
  3. Bump version of pom-scijava parent in pom-imagej. Cut new release of pom-imagej.
  4. Bump version of pom-imagej parent in pom-fiji. Cut new release of pom-fiji.
  5. Bump version of pom-fiji parent in pom-bigdataviewer. Cut new release of pom-bigdataviewer.
  6. Bump version of pom-bigdataviewer parent in spim_data.

Total: 6 commits, 5 releases

How it works with the new proposal

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within bom-scijava. Cut new release of bom-scijava.
  3. Bump version of bom-scijava import in bom-umbrella. Cut new release of bom-umbrella.
  4. Bump version of bom-umbrella import in pom-base. Cut new release of pom-base.
  5. Bump version of pom-base parent in pom-bigdataviewer. Cut new release of pom-bigdataviewer.
  6. Bump version of pom-bigdataviewer parent in spim_data.

Total: 6 commits, 5 releases

Does this change make releases easier?

Either way, we are cutting five releases in order to push a new component version to another component. Intuition tells us there must be a simpler structure which meets all the requirements outlined above. But I personally do not see it.

The advantage of the new way is that it scales. It is consistently flat, always requiring five releases, regardless of the “depth” of the component, since all layers of the stack are now on equal footing. E.g.: a new bdv3d layer which extends the bigdataviewer layer would require 6 releases with the old scheme, but still only 5 releases with the new one since pom-bdv3d would extend pom-base directly like every layer does.

I also believe that the new structure will make it easier for us to create release tooling that automatically performs these changes in sequence. While we have taken steps toward creating auto-release-cascade tooling in the past, I always get hung up on challenges caused by details of the current structure. My intuition is that the new structure will be less difficult to automate.

Short circuiting

Just to clarify one thing: it is of course possible to “short circuit” the process above, by having spim_data directly override its version of scijava-common. This is known as “version pinning”. The issue with this is that the BOMs become inaccurate, no longer reflecting what we actually ship in applications like Fiji. That is: we want consumers who import the bom-umbrella and/or extend the pom-base to end up with the same versions our various component dependency trees use. Hence, version pinning should only be done very transiently, with the new versions pushed back into BOMs which are then released, as soon as possible.

Other possible structures

No imports (option B)

Consider a new structure as described above, except that pom-base does not import bom-umbrella. In other words, none of the parent POMs provide any version management. With this structure, our release cascade process for the aforementioned example becomes:

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within bom-scijava. Cut new release of bom-scijava.
  3. Bump version of bom-scijava import in bom-umbrella. Cut new release of bom-umbrella.
  4. Bump version of bom-umbrella import in spim_data.

Total: 4 commits, 3 releases

This is simpler and (IMHO) more intuitive: of course we need to cut a new release of the SciJava BOM, as well as the umbrella BOM, every time a SciJava component changes version, right?

There are two disadvantages to this approach, however:

  1. Every consumer (e.g., spim_data) must now both extend pom-base and import bom-umbrella. This adds 11+ lines of XML boilerplate to every single project POM.
  2. Perhaps more importantly: the linkage between a particular version of the base POM and a particular version of the umbrella BOM is lost. While the interaction between the two is likely to be negligible, I am not certain. For example, what if the dependency version of a particular managed component is somehow needed within the base POM’s plugin configuration? (E.g.: we decide to embed the version of SciJava Common in every JAR manifest? A contrived example to be sure, but an illustration of what I mean here.)

Deferred imports (option C)

Suppose that like option B, pom-base does not import bom-umbrella, but each layer’s POM parent does. Our release cascade process then becomes:

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within bom-scijava. Cut new release of bom-scijava.
  3. Bump version of bom-scijava import in bom-umbrella. Cut new release of bom-umbrella.
  4. Bump version of bom-umbrella import in pom-bigdataviewer. Cut new release of pom-bigdataviewer.
  5. Bump version of pom-bigdataviewer parent in spim_data.

Total: 5 commits, 4 releases

This is a compromise between options A and B above. I do not like this scheme, and would prefer either option A or B, but I list it here for completeness.

Some disadvantages to this approach are:

  1. We are somehow not eating our own dog food: every non-core consumer in the community will need to import the bom-umbrella, or create their own parent POM which does, or (erroneously IMO—we should discourage this) extend a parent POM of one of the core layers.
  2. We still have to do 5 releases every time—may as well do 6 at that point, for downstream consumers’ maximum convenience.

A great idea I missed?

Do you see an option not discussed here, which would be superior to any of the above? Now is the time to raise it! (I am looking in your direction, @axtimwalde—and BTW: :birthday: :champagne: :smile:)

For any who read this far: my sincere thanks. And please post a comment with your thoughts!


Comprehensive Fiji updates
TrackMate - Getting data from non-tracking channel
#2

There is a wrinkle I failed to address in my writeup above: whether to keep supporting the ability to override a version property in order to override the dependency version itself.

Right now, you can write e.g.:

<properties>
  <imagej.version>2.0.0-rc-50</imagej.version>
</properties>

And if your component depends on net.imagej:imagej, it will then magically depend on 2.0.0-rc-50 instead of the normal managed version.

However, this feature only works if your managed version came in through inheritance, not import.

My new proposal above brings in all the dependency management via imports, and thus you cannot override the version anymore in this manner.

One way to keep supporting this feature is Option D: eliminate all the layered bom-<foo> components in favor of only bom-umbrella. So we have ONLY ONE BOM for the entire SciJava ecosystem. The more I think about this, the more I like it—it is simpler, and cuts down by 1 on the number of components we must release for each cascade.

What do others think? (If this needs clarification, I can expound in more detail, but we are already dealing with a mountain of text here.)


#3

I decided to write up these new simplified options, for clarity and completeness. Here they are:

Simplified proposed structure (option D)

  • A single Bill of Materials (3, 4)
    • bill-of-materials - combined version management for all components of the managed SciJava ecosystem; community can request to have their component(s) included here
  • Community base POM (1)
    • pom-base - the One True POM ™ that all community projects extend; extends bill-of-materials (NB: it needs to extend, not import, so that overridding of version properties in consumer POMs keeps working as desired.)
  • Core project POMs (2)
    • pom-scijava - for projects in the scijava organization; extends pom-base
    • pom-imagej - for projects in the imagej organization; extends pom-base
    • pom-imglib - for projects in the imglib organization; extends pom-base
    • pom-scifio - for projects in the scifio organization; extends pom-base
    • pom-fiji - for projects in the fiji organization; extends pom-base
    • pom-bigdataviewer - for projects in the bigdataviewer organization; extends pom-base
    • pom-trakem2 - for projects in the trakem2 organization; extends pom-base
    • etc.

This design:

  • Allows components at any managed layer to depend on components at any other managed layer freely, without worrying about version skew, because all core project POMs extend the community base POM, which manages the known universe.
  • Allows third party components to import the BOM if it needs to extend some other parent.
  • Either way, third party BOMs outside the known universe can be imported freely.
  • Eliminates the current conflation between the imagej, imglib and scifio organizations.
  • Limits the number of components which must be released when low-level components are updated (se next section).
  • Preserves the ability to override a component’s associated version property to change the dependency version used. NB: Options A, B, C and E (below) do not have this quality, because the Bill of Materials POM imports the component BOMs, thus resolving the versions which “locks them in.”

Release cascade procedure

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within bill-of-materials. Cut new release of bill-of-materials.
  3. Bump version of bill-of-materials parent in pom-base. Cut new release of pom-base.
  4. Bump version of pom-base parent in pom-bigdataviewer. Cut new release of pom-bigdataviewer.
  5. Bump version of pom-bigdataviewer parent in spim_data.

Total: 5 commits, 4 releases

Simplified “no imports” option E (D+B)

Same as D, but pom-base does not extend bill-of-materials. Each component must extend pom-base and import bill-of-materials itself.

This has a simpler release cascade procedure:

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within bill-of-materials. Cut new release of bill-of-materials.
  3. Bump version of bill-of-materials import in spim_data.

Total: 3 commits, 2 releases

Very nice to have things this simple, however, we have the same downsides as option B:

  • Every consumer (e.g., spim_data) must now both extend pom-base and import bill-of-materials. This adds 11+ lines of XML boilerplate to every single project POM.
  • Perhaps more importantly: the linkage between a particular version of the base POM and a particular version of the BOM is lost. While the interaction between the two is likely to be negligible, I am not certain. For example, what if the dependency version of a particular managed component is somehow needed within the base POM’s plugin configuration? (E.g.: we decide to embed the version of SciJava Common in every JAR manifest? A contrived example to be sure, but an illustration of what I mean here.)
  • As with all options besides D, we lose the ability to override version properties, as discussed above.

So in conclusion, my preferred solution is option D right now—what do others think?

In particular, it would be helpful to know whether anyone actually cares about being able to override the version properties (vs. simply declaring the version explicitly in your <dependencies> section, which works regardless). Because if not, option E looks mighty tempting.


#4

One more option, and then I’m letting this thread lie for a few days:

Ultimate unified base (Option F)

  • A single parent + Bill of Materials (1, 2, 3, 4)
    • pom-scijava - a single parent + BOM for the entire SciJava ecosystem.

Upsides

  • Allows components at any managed layer to depend on components at any other managed layer freely, without worrying about version skew, because all components extend pom-scijava directly.
  • Allows third party components to import pom-scijava as a BOM if they need to extend some other parent.
  • Eliminates the need to cut releases of layer-specific parent POMs, since all components extend pom-scijava directly.
  • Would use properties of the form scijava.<foo> to specify layer-specific configuration more succinctly, to reduce boilerplate. E.g.: <scijava.license.name>Simplified BSD License</scijava.license.name> in each component POM.
  • Preserves the ability to override a component’s associated version property to change the dependency version used, as long as dependency management was obtained via inheritance rather than import.
  • Vastly simplifies the field of parent POMs etc., making things much simpler for everyone, especially newcomers. No more bom- this and pom- that. Everything is just pom-scijava.

Release cascade procedure

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within pom-scijava. Cut new release of pom-scijava.
  3. Bump version of pom-scijava parent in spim_data.

Total: 3 commits, 2 releases

Downsides

  • Some boilerplate repetition across components of a given layer; e.g., blocks such as mailingLists would need to be redeclared in every ImageJ component’s POM.

But I think the extreme reduction of complexity in general makes this repetition worth it. There are not very many sections of XML which would need to be repeated. And for several of them, we can use properties to mitigate the issue, as described above.

Now someone tell me why this won’t work! :wink:


Migrating: Ant -> Maven
#5

(copied from gitter)
I’m just now reading through the SciJava Maven POM structure thread.
I fast-forwarded to the final proposal (still need to work through the rest).
My concern with that is that it is no longer possible to work on a sub-project (bigdataviewer or imglib2) in isolation from the rest. With the old solution, I can work on a new “version” of a sub-project before unleashing it into the bigger context, bumping versions on e.g. spim_data, bdv-core, bdv-fiji that depend on each other. I could manage those versions in pom-bdv and only after a new “stable state” is reached bump the pom-bdv version in pom-fiji.
Instead of locally moving forward using pom-bdv, I would now have to make incremental changes to pom-scijava. Every change is immediately visible (and forced upon) other sub-projects that move forward in the same way. Assume that SPIM_Registration would also be made of several interdependent components. Then for any of these to move forward, it has to use whatever is the latest incremental bdv-core version, SPIM_Registration no longer can decide when to upgrade it’s BDV version.
Sub-projects will not want this, so I predict that we will effectively see a lot of short-circuiting to do this sub-project development.
I could imagine that to not lose overview in the short-circuiting mess I would still make a pom-bigdataviewer parent that does the (temporary) short-circuiting for the sub-project.
Also, versions of pom-scijava will be virtually meaningless. They say nothing about when imglib moved forward for example.


#6

I think your major concern here could be addressed by one of the other options discussed above. Let me know what you think after you finish reading through everything above—including the “wrinkle” about version properties.

Either way, I will reply back again to this thread with more detailed comments later.


#7

I have to admit that I do not understand all the details…

Could you clarify the following: Assume there is pom A that imports version declarations from pom B, and pom C that has pom A as a parent. Can pom C override versions defined in pom B? Because they are technically not imported now (from B) but inherited (from A)?

(By override I mean by setting the <xxx.version> properties)

If that was the case, then we could work around the “wrinkle” by introducing an additional inheritance level


#8

Nope. The import of B into A “bakes in” B’s version properties. Those properties do not exist in pom A—and hence they do not exist in pom C—because they were not inherited.

That said, you can still override the versions… but you have to do it by writing <version>x.y.z</version> in the <dependency> block of the dependency whose version you want to override. Just setting the property will do nothing.

You can already see this wrinkle affecting our current structure in some cases. If you clone https://github.com/fiji/fiji and set the bigdataviewer-core.version property in its POM, you will notice that it has no effect. This is because the version of bigdataviewer-core comes in via import scope, not inheritance.


#9

Ok, I read through everything now. I favour something like option D currently. I would definitely keep the intermediate pom-imglib etc for the core projects, because that allows them to move forward independently. Is there a way to incorporate this into option F? For example, pom-scijava could define all the imglib-algorithm.version etc, but imglib2-algorithm extends pom-imglib. Then we can override imglib-xxx.version in pom-imglib and release new version of that etc, promoting a new set of versions to pom-scijava less frequently. ImgLib2 could move forward independently, and if someone else wants to depend on a state that has not been promoted to pom-scijava yet, they can import that pom-imglib to override the versions from their current pom-scijava parent. Does that make sense?


#10

In that scenario, promoting a set of versions to pom-scijava would roughly coincide with uploading them to the updater.


#11

Unified base + layer-specific POM parents (option G)

This is basically option D, but merging the pom-base together with the bill-of-materials to reduce the number of POMs in play.

  • Community base POM (1, 4)
    • pom-base - the One True POM ™ that all community projects extend; provides plugin configuration and a unified bill of materials for the entire universe
  • Core project POMs (2, 3)
    • pom-scijava - for projects in the scijava organization; extends pom-base; overrides scijava-*.version properties as needed.
    • pom-imagej - for projects in the imagej organization; extends pom-base; overrides imagej-*.version properties as needed.
    • pom-imglib - for projects in the imglib organization; extends pom-base; overrides imglib2-*.version properties as needed.
    • pom-scifio - for projects in the scifio organization; extends pom-base; overrides scifio-*.version properties as needed.
    • pom-fiji - for projects in the fiji organization; extends pom-base; overrides Fiji component version properties as needed.
    • pom-bigdataviewer - for projects in the bigdataviewer organization; extends pom-base; overrides bigdataviewer-*.version properties as needed.
    • pom-trakem2 - for projects in the trakem2 organization; extends pom-base; overrides TrakEM2 component version properties as needed.
    • etc.

This design:

  • Allows components at any managed layer to depend on components at any other managed layer freely, without worrying about version skew, because all core project POMs extend the community base POM, which manages the known universe.
  • Provides a universal BOM defining intended compatible versions across the universe
  • Provides each layer with its own base POM + BOM for that layer.
  • Allows third party components to import the universal BOM, or layer-specific BOM(s)—if it needs to extend some other parent.
  • Either way, third party BOMs outside the known universe can be imported freely.
  • Eliminates the current conflation between the imagej, imglib and scifio organizations.
  • Limits the number of components which must be released when low-level components are updated (see next section).
  • Preserves the ability to override a component’s associated version property to change the dependency version used, as long as pom-base or a layer-specific POM are used as parent.

Release cascade procedure

  1. Bug-fix to scijava-common merges to master. Cut a new release of scijava-common.
  2. Bump version of scijava-common within pom-base. Cut new release of pom-base.
  3. Bump version of pom-base parent in pom-bigdataviewer. Cut new release of pom-bigdataviewer.
  4. Bump version of pom-bigdataviewer parent in spim_data.

Total: 4 commits, 3 releases

Actually, the total number of releases depends on which layers need the new SJC. In the scenario above, we assume it is a component of the bigdataviewer layer that needs it. But if, say, scifio also needs it, then we also have:

  1. Bump version of pom-base parent in pom-scifio. Cut new release of pom-scifio.
  2. Bump version of pom-scifio parent in scifio.

So the generalized totals are:

Total: L+C+2 commits, L+2 releases

Where L is the number of downstream layers needing access to the bug-fix, and C is the number of downstream components needing it.

Variations

If we combine pom-base with pom-scijava (assuming we never want to manage versions of core SciJava components independently of the umbrella BOM, which is definitely arguable), we get the SciJava layer for free.

And of course, as with all the options above, we can combine multiple component updates into a single release of pom-base and/or the layer POMs.

I admit to being disappointed that we need to keep the layer-specific POMs around, since they greatly increase amount of manual effort, and/or increase the complexity of the tooling. But I think this structure addresses your concern, @tpietzsch?

Anyone else have objections, concerns, etc.?


#12

Yes, that would address my concerns. I would prefer this model over option F, although more component releases are involved.

I could also live with option F if that is what others prefer, provided that it is ok to release artifacts that locally override versions. In this way it would still be possible to move sub-projects forward locally. They simply have to override versions. We could even offer a bom-imglib (independently of the hierarchy) that defines the “releases” of the imglib sub-project (whether they immediately make it into pom-scijava or not). My gut-feeling is, that this would involve more discipline and bureaucracy than the other option although it requires fewer releases. But I might be wrong.


#13

I gave it some further thought. My current thinking is that option G will cause us problems. What follows is my attempt to articulate why.

I think it is definitely OK to release artifacts that locally override versions. (See below for discussion.)

My main concern is this: The more diverging dependency graphs we have, the more different configurations we need to test, and the more possibilities there are for something to go wrong due to "skew."

For each option, I think it is important to consider how many different configurations (i.e.: dependency graphs) there can be.

Suppose we have a third party component foo which depends on spim_data via some mechanism, and therefore also indirectly on other things such as scijava-common. Let’s consider the possible mechanisms for resolving the dependency versions:

  1. foo has no parent, and manually specifies the version of spim_data, inheriting the scijava-common version from spim_data's usual dependency graph.
  2. foo extends or imports pom-base, inheriting the spim_data and scijava-common versions.
  3. foo extends or imports pom-bigdataviewer, inheriting the spim_data and scijava-common versions.

All three of these mechanisms are supposed to offer working combinations of components.

What I like about option F is that it minimizes the recommended paths for combining components: you can do (1) or (2) above, but (3) is off the table.

If we provide pom-bigdataviewer as an alternative (potentially more “bleeding edge”) BOM, foo now has a third possible mechanism for defining its dependency graph. Unfortunately, your stated use case implies that versions of components from other layers (e.g., TrakEM2) are not guaranteed/tested to be compatible anymore. After all: the point of pom-bigdataviewer is to have the freedom to advance BDV component versions without worrying about the implications for the software stack as a whole, correct?

So we now have something that looks like a BOM covering all layers, but which actually is only intended for use internally at the BigDataViewer layer (and possibly “deeper” layers—but not “sibling” layers such as TrakEM2). Of course, we can document in the pom-bigdataviewer description that this POM is “for internal use only” as the parent of BDV components. But my question is: do we really need it then? Can we instead get away with simply overriding version properties as needed in each individual BDV component?

Option F does not let foo import e.g. bigdataviewer-server and bigdataviewer_fiji at compatible versions unless it uses the umbrella BOM, since only that BOM defines compatible versions of such independent components. But while option G does offer foo an alternative way to import such components by consuming pom-bigdataviewer, it contains this potentially major pitfall that certain other managed component versions may actually be wrong. Hence, do we actually want to recommend that people can use pom-bigdataviewer in this way?

Maven does not have any such thing as “visibility” for POMs. Continuing to publish pom-bigdataviewer means that anyone can consume it. The easiest way to avoid this pitfall is to stop publishing it, in favor of specific per-component version overrides.

My initial intuition was similar, that every layer should have its own independent BOM. But the more I think about it, the more I think it is not actually very useful, and in fact likely to be harmful overall. Many of these layers are interconnected (particularly imagej, imglib and scifio), and there is not really any such thing as “independent BOM” here.

If my concerns above are unconvincing about the dangers of layer-specific BOMs, then my next question is: what are some concrete use cases for e.g. bom-imglib? Because the more I think about it, the more trouble I have coming up with anything.


#14

I think the main use case is for defining sets versions of the imglib2 projects that are expected to work together.
The same way that pom-scijava is supposed to bundle versions of the whole scijava/imglib2/scifio/imagej… ecosystem.

My feeling is that with putting all definitions into pom-scijava there comes a shift in responsibility? The developers of subprojects need to make sure that the whole ecosystem works for every single released artifact that they want to communicate to consumers in other (or even the same) subproject – unless they want to explicitly override the version in the consumer?

Anyway, nobody else objects. So, I’m fine with option F.
If real problems come up with that they can always be fixed using explicitly defined versions anyway.


#15

With this scheme, would you have bom-imglib extend pom-base? Or would you have it stand alone as only a BOM, which must be imported? I’m trying to understand the logistics here:

  • If it’s only a BOM, we lose the ability to use & override version properties.
  • If it extends pom-base, it inherits the dependency management of the entire software stack, but then overrides the ImgLib portion of it, leaving a potentially invalid state w.r.t. the other layers.
  • Other structures are possible, but I don’t see one that cleanly meets all requirements here… do you?

In the common case of wanting to release a new version of some component, e.g. bigdataviewer-core, and then use it in some other component of the same layer, e.g. bigdataviewer_fiji, I really think it is fine to just cut the release, and then pin it in the other component directly. No need to update any POMs or BOMs or whatever.

The only time you would update the POM/BOM(s) is after the versions have all been tested together. We can have Jenkins do this, using the melting pot I wrote last year. Once Jenkins gives the green light, we can push the new version numbers into the umbrella BOM all at once.

Regarding responsibility: I think the responsibility was already there for component developers to do integration testing. When you change bigdataviewer-core, you will/must eventually test it in Fiji. And once you do, and want the component to be part of Fiji, you bump the pom-bigdataviewer.version property in pom-fiji. This is not something I have taken the liberty of doing myself as Fiji maintainer, because I do not know the status of your latest BDV releases—whether you consider them “ready” for distribution with Fiji yet. I rely on your judgment and wishes for when to “throw new releases into the mix.”

I think generalizing this process to an umbrella BOM for all layers of the ecosystem helps to move us away from the “Fiji-specific” aspects of our current structure, in favor of a more inclusive scheme where anyone can join the “core SciJava ecosystem” as long as they are willing to fix “version skew” with other layers when the melting pot flags it.

OK. And actually, if using explicitly defined versions is still somehow insufficient, you can always reintroduce pom-bigdataviewer again just for the BDV layer, and/or pom-imglib2 just for the ImgLib2 layer, etc. There is no rule that we have to have a pom-<foo> at every single layer—just the layers where it somehow helps the maintainer to continue development effectively.

Later this week, I will restructure everything according to Option F above. To make sure there is no grievous flaw in the approach, I will first do it locally with snapshots across various layers, before deploying any releases.


#16

Hi, sorry for the late response but I came back from leisure time only yesterday night. This is a lot of text, and I am sure I forgot something at the top when I reached the end. But here are some thoughts that may help us relax about this:

  • BOMs and parent poms not only manage dependencies but also specify useful build system magic such as enforcer rues, java versions, and alike.
  • BOMs or parent poms that manage compatible sets of dependency versions are helpers for two distinct crowds: (1) component developers, and (2) ecosystem distributors.
  • For component development, usually smaller BOMs or parent poms are more convenient because they allow to focus on local interaction.
  • For ecosystem distribution, such as Fiji, larger BOMs or parent poms are more convenient because they describe entire ecosystems, i.e. the distribution.
  • Component developers often want to cut releases of smaller ecosystems deferring adoption in dependent projects in larger ecosystems to their respective authors. For this scenario, large ecosystem BOMs or parent poms are not helpful and/ or make potentially false promises about the consistency of the larger ecosystem.

I currently think that I like the build system magic and whatever comes with it, but I would like to be able to care for Fiji only after I am done with my work on some smaller project. So I would like to be able to have small local parent poms specific to individual components that can consume the helpful build magic. This local component management is a useful hint for dependent projects that want to use my component, they can copy the managed dependencies or consume the BOM. If I then feel that I want to release my component in Fiji, I will have to update and test pom-fiji (or whatever it will be called) updating all the versions of my components and potentially fixing dependent projects, this BOM could also import the component managing poms, skew is possible and has to be addressed manually or by automatic testing. In this scheme, the distribution centric poms are at the end of the food chain instead of at the beginning, and I like this better. I currently cannot see how any of the above schemes would prevent version skew reliably, but importing/ inheriting managed dependencies from large Uber-poms makes false promises about the compatibility of dependent projects.

And I am sure that I am totally missing something, sorry for that already.


#17

Indeed. And there is a third crowd: (3) ecosystem consumers: people in the community who want to use the components—and hence the BOMs+parents—to build their own components outside the “core” set of components.

I see the umbrella BOM as describing compatibility across the entire ecosystem. This actually goes beyond Fiji, since there are components which do not ship with Fiji, but which nonetheless must be managed at compatible versions. But one great consequence of the umbrella BOM is that we get version management for distributions such as Fiji “for free” on top.

Can you please describe a proposed structure for your “smaller BOMs”? I tried very hard above, with many different options. It is vital, if you are voting against Option F above, to discuss what would work for your desired workflows.

As discussed, this can be achieved by cutting new releases of your components in question, and using version property overrides to pin to newer dependency versions as needed, until those new releases can be tested in the full core SciJava ecosystem—at which point those new released version numbers all get propagated into the umbrella BOM at once, and the pins can be removed.

I am trying to understand how a separate/smaller pom-imglib2 actually helps solve anything here.

And how would you structure our POMs to achieve this?

As discussed above, if the umbrella BOM imports other BOMs, then we lose the ability to override versions using properties. This will make the melting pot I wrote last year non-functional. If we are OK with this, we can consider it as an option. Just keep in mind that we only get functional version properties via inheritance, not via import.

In theory, that is the design that makes sense. But in practice, there are several obstacles, as discussed above. In particular: 1) the version properties issue; and 2) the complexity/pain/burden of nested parent POM structures. Regarding (2), it is confusing for developers to be faced with many possible POM parents, and it is a lot of manual time and effort to be cutting releases of these POM trees all the time.

To quote my earlier post:

If we have one shared BOM, then the only possible skew is between the managed version of a dependency declared in that BOM, and the default one inherited when depending on a component.

If we have multiple different BOMs (e.g., one per layer of the ecosystem), then there are many possible different versions of lower-level components such as imglib2, depending on exactly how one structures one’s POM downstream (which POM is extended, which are imported, in which order, etc.).

Again, I think we need a concrete alternative proposal here to compare against the options above, and against the stated requirements, if we are going to move forward.


#18

@ctrueden in my naive view of the world, I think that it makes sense for a component to inherit a parent-pom that manages dependencies that the component is using for building, testing, or running. The umbrella-pom that you suggested above, if I get it right, manages the entire eco-system of components relevant for a variety of distributions. If a component inherits this umbrella-pom without locally overriding any version properties, I assume that the expectation is that the component is compatible with all managed components in the umbrella-pom. This, however, is not guaranteed at maven build time. Therefore I think that using large umbrella-poms as parents is not useful. I believe that umbrella-poms (distribution-poms) should exist independently of the management-poms for components. The currrent parent-poms are a mix of the above, e.g. pom-imglib2 manages the versions of several imglib2 components and thus is a distribution-pom, through inheritance from pom-imagej, it is even more so, but it also specifies build instructions, java versions, enforcers. I think that umbrella-poms (distribution-poms) should not act as parents of components because that leads to false promises of compatibility that are not discovered at local build and test time. I think that umbrella-poms (distribution-poms) should be generated by aggregators such as your melting pot, testing compatibility of all included components.


#19

@axtimwalde Please describe—specifically—your preferred POM structure that you think will solve all of the requirements.


#20

Since no other concrete structures have been proposed, I am going to move forward with my plan to unify all SciJava version management under a single BOM. Work started on the universal-pom branch.

However, I do appreciate the desire to have other BOMs which manage smaller subsets of the universe. As such, I offer the following choice to the maintainers of each layer of the SciJava stack:

  1. Keep a dedicated intermediate POM parent for the components at your layer (e.g., pom-bigdataviewer), where you manage desired components only; OR:
  2. Eliminate your layer’s intermediate POM parent in favor of the umbrella BOM only.

To make option 1 above feasible, I will create two ancestor POMs (names still tentative):

  • scijava-base - with the “build magic” (i.e. plugin configuration et. al) only—NO managed dependencies
  • pom-scijava - which extends scijava-base and adds ALL dependency management for the universe

So, for maintainers who wish to produce their own BOMs, they can extend scijava-base to avoid inheriting any potentially mismatched versions. The only catch is that maintaining these “BOM subsets” is outside my role as general SciJava maintainer.

As maintainer of the scijava, imagej, scifio and fiji layers of the software stack, I will be eliminating their respective layer-specific POM parents, in favor of those components directly extending pom-scijava. When I release new versions of those components, I will let the Jenkins melting pot verify that they work properly throughout the SciJava stack, then bump the managed versions in pom-scijava accordingly once the releases have been vetted.

@tpietzsch @axtimwalde I hope this satisfies your requirements and wishes here. Let me know what you decide for the imglib, bigdataviewer and TrakEM2 layers.

  • If you go with option 1, I will henceforth defer to you—I won’t ever push to master or cut releases anymore, but instead leave those actions to you as maintainers.
  • If you go with option 2, I am happy to do the work of updating each component to extend pom-scijava directly, and I will continue to act as a co-maintainer.

If I do not hear back from you, I will assume you want option 1.


fillHoles (imagej ops)