Gradle migration

Development
, , , , ,
#1

Hi,

long short story: I’m going to start by Casus next week (Scenery and Sciview) and I’d like to suggest Gradle as building tool and dependency manager to the sciJava community.

Why? I’ll be honest, simply because I’ve been using it all my career, I really never did something serious with Maven. The first thing, at the time, which put me off was the xml and since then I always sticked with Gradle.

But I do totally understand that this has to be seen and analyzed professionally in order to be taken seriously, and in hindsight, I can see I took, luckily, the right decision at the time.

The main reasons to choose Gradle over Maven are essentially: flexibility, performance, user experience, and dependency management.

Also worth mentioning, Gradle features and its performances vs Maven

This post follows a discussion on the Gradle Slack channel and I’m going to report here the most interesting replies since some of the sci people, who are interested in this, might not be interested joining the channel just to read them, like I was suggested on the scenery Gitter channel (definitely too many channels around :upside_down_face:).

Let’s start right away with the Curtis’ critic:

My take on Gradle in general is that it is a step backwards from Maven, because it allows people to move away from declarative and back to imperative in dangerous and fragile ways. But I appreciate that it certainly can be more powerful.

Which triggered the following meditations:

Alex Nordlund (github com deepy)

I agree 100%
But I’m also yet to see a fully declarative maven project in the wild
And whenever people ask me about maven they tend to try and want it more imperative, and the moment you got antrun in there you might as well go for gradle and save yourself the headache

Personally I think your best bet is to find a case where something is difficult to do in maven, then take the time to port it into gradle and use that as an example when pushing for improvements

@ctrueden, in this regard, could you list which are/were your pain points, if any?

Björn Kautler, (github com Vampire)

He is imho totally right with all, except for the “step backwards”, instead it should read “step forwards”.
In my opinion Gradle takes the best from Maven (convention over configuration) and Ant with Ivy (the power to easily write a build that does what you want instead of forcing you to jump through ugly hoops)
and adds some amazing more features like the Gradle wrapper, build output cache, Java compile avoidance, many others.
You can also do imperative work in dangerous and fragile ways with Maven by simply writing your own plugin that does bad things or by using antrun.
But well, you can abuse any system always.
So you should always consider what you are doing and whether you should do it that way, that’s simply how software development works.
Remember, it is a tool for software developers, not for managers.
There is no good reason to force software developers to bend into very tight ways a tool maker thinks makes most sense, when you can also provide good conventions and defaults but preserve for the user the power to bend the tool to his will which is how it should be imho.

Björn helped me a lot of times and he proved to be quite experienced. I trust him and I give his point of view a lot of confidence.

Daniel Lacasse (github com lacasseio)

As some mention, finding the bottleneck in term of performance, pain point and developer productivity is your key to make an accurate assessment of the validity for a migration to Gradle. The migration doesn’t have to be a all or nothing. Partial migration or even wrapping the maven build to allow Gradle to drive the build process is a good approach to the problem. Regardless, another good point to consider is the expertise. If most build engineers knows Maven but not Gradle, then it may be a bad idea to migrate given the lost in productivity this may cause. It’s a though call to make, you need to gather more data to support your case.

I totally agree especially with Daniel on the migration point, which can be totally partial of course. I’m trying first with Scenery (and Sciview), we could analyze these as showcases and then draw our conclusions. Although a full Gradle conversion has its benefits, such as aligning versions natively, kind of what already happens nowadays with the parent pom s.

Andrey Mischenko, (github com gildor)

I kinda agree with this comment, having declarative build script is nicer than imperative, but I would say that it completely abstract idea, which really doesn’t make Maven better or Gradle worse

I know Andrey the longest and trust him a lot, his concept is also shared by Vladimir, right after

Vladimir Sitnikovm (github com vlsi)

I migrated Apache Calcite build from Maven to Gradle, and here are my findings:

  • No-one ever raised “declarative vs non-declarative” thing
  • Everybody said “it will break everything”
  • Many said that Maven-based build evolved for very long time, so it would take ages to migrate

The pain-points for me were:

  • Project loading to IDEA. Calcite uses JavaCC to generate parser from Freemarker templates, so it was like “you need to execute mvn install before the project is importable to IDEA”. Of course, switching between branches was painful;.
  • Incremental compilation/jar was not really working with Maven. The project used lots of profiles like “skip this, skip that”
  • It was hard to “just build the jar” or “just run tests in core module”. Maven executes Checkstyle, JavaDoc, etc, etc, so building jar spent noticeable time on building irrelevant things.
  • Integration testing was hard. Maven doesn’t support declaring project version once, and it still requires to duplicate it all over the pom files. If you need to create a branch and publish it under different version, then you are out of luck. You edit lost of poms.

Now the migration to Gradle was done, and it was like, ok, it just works. There are contributors/committers who know nothing of Maven :scream_cat:, and now they know nothing of Gradle :slightly_smiling_face: . The build just works for them.

An (un-)expected consequence of the migration was that some validations were missing. For instance, the Maven-based build was configured to fail on javac warnings, and I missed this setting when migrating to Gradle. So the code did contain some violations by the time -Werror was re-added.

In general, I agree one needs to understand the pain-points and use-cases before doing the migration.

This is, imho, really a valuable reply. His experience shows like the “declarative vs imperative” dualism exists actually only “on paper”, that is it has no practical implementation in reality.

That’s it for the moment, I wanted to write this earlier but I didn’t have time.

I hope to ignite some interesting thoughts about.

Ps: corresponding github account not as link because I’m new and I can’t post more than 5 links…

3 Likes
#2

Thanks @gBarbieri for this detailed conversation starter!

For many years I was on the Maven mailing list, and consumed several conversations like this. I eventually had to leave to keep my sanity. I am reluctant to dive into a detailed discussion / opinion-fest / flamewar of declarative vs. imperative, dynamic vs. static typing, or any of those extremely complex and nuanced topics. Nonetheless, since you have raised the issue, I’ll start with some replies to a few of the points you quoted, and then summarize my take on the pros and cons of switching.

We have hundreds of them.

Sure, that seems fair. Out of ~300 projects in the Fiji software stack, 1 uses antrun: the ImageJ launcher. Switching that one to Gradle might be a good idea. :+1: Of the rest, 29 of them have a <build> section—the rest are 100% standard Maven builds built on the pom-scijava infrastructure.

:+1: Makes total sense!

I haven’t given Gradle a serious try yet. Just followed the progress and did some research over the past few years, when Gradle started getting popular. See my summary below for anticipated pain points.

No, these project management tools are useful for both developers and managers. SciJava uses Maven both technically and socially: e.g. POMs document developer roles, project status, web links, license details, and much more—in addition to only technical details of how to execute builds. We use that metadata to autogenerate sidebars on plugin web pages. Project management tools like Gradle and Maven with a broad range of purposes, useful to developers, managers, and others.

No, there are many good reasons to do so if that strictness enables useful guarantees. Taking this argument to an extreme, we should not have static typing, or perhaps programming languages at all because they restrict what programmers can express.

I dislike this common argument. “You can write bad code in any language, therefore choice of language doesn’t matter.” On the contrary: the easier and more natural it is to write “bad” (less maintainable, more fragile, harder to understand, etc.) code, the more people will do it, and the more problems will arise later as a consequence. One reason I love Maven is because it makes it hard to do such “bad” things by default. Please understand I’m not saying one should never use Gradle—it really depends on the use case.

I think comments like this illustrate a sort of “implicit bias” of the Gradle community. Of course projects doing this kind of thing will prefer Gradle, because it is a more flexible tool. And that’s good: I’m glad Gradle is helpful to projects like Calcite. But SciJava is not doing this kind of thing. And I don’t want our user community doing this kind of thing unless they really know what they are doing—in which case they can use Gradle already with their extension, since it’s Maven-compatible.

*nod* We make heavy use of the maven-enforcer-plugin to protect people from common mistakes in the POM. I’m glad to see that a Gradle enforcer plugin now exists too. :smile:

I strongly disagree. If you want to analyze anecdotal accounts, here’s a counterexample:

And associated Reddit discussion with others who had similar experiences, including this one:

To state that “[‘declarative vs imperative’ dualism] has no practical implementation in reality” based on one successful build system migration grossly overgeneralizes the incredible breadth of scenarios in the wild.

OK, I got a bit ranty there at the end, sorry. :sheep: Here’s my pros + cons list for migrating the SciJava ecosystem to Gradle:

Pros

  1. More succinct build script syntax would be friendlier for non-software-engineers. The same scientists who love Python would likely also appreciate the build.gradle DSL over Maven’s pom.xml. However, there’s also the takari polyglot extension for Maven which would similarly solve the problem. Either way, to ultimately deploy artifacts to a Maven repository, my understanding is that a pom.xml needs to be generated somehow… or else everyone downstream would also be locked into Gradle. No?

  2. More robust dependency management logic. I’m fully in support of a system enabling better dependency declaration metadata than Maven’s “closest wins” rule. Right now we use SemVer and I’d love better infrastructure in place to guide developers on which combinations of artifacts are safe, as well as better tools for maintainers (e.g. me) to diagnose and resolve potential breakages of dependency upgrades.

  3. Easier to mix in native builds. The Gradle native build support sounds super good, and we have a couple of projects (the ImageJ Launcher, and FLIMLib from my lab) that could potentially make use of it to have a nicer more unified build, rather than the current CMake+Maven approach we use now. Certainly it would be more appealing than how we used to do it using the NAR Maven plugin. :ghost:

  4. Active developer community with (I hope!) less toxic politics. I’ve found the politics surrounding Maven greatly concerning. A bunch of PRs were merged into the Maven 1.4.x development branch, but then the decision was made to burn that work forever, with contributors expected (unrealistically) to recontribute patches again—including patches fixing critical, long-standing dependency resolution bugs that still hurt the SciJava project today in some scenarios. There is also a complex and apparently cool (at best) relationship between the Apache Software Foundation, Sonatype, Takari.io, and the Eclipse Foundation, which appeared to me to have caused a number of casualties, including the NAR Maven plugin being abandoned a decade ago. This is probably a big part of why Maven development has slowed down in recent years. If the Gradle community is even somewhat less dysfunctional, I’d be happy to be there instead of staying in Maven world.

Cons

  1. Tooling breakages. As mentioned above, we consume the POMs downstream. Unlike Groovy, XML is cross-language and reliably parseable in every popular development environment. The status.scijava.org site would probably need TLC. The ImageJ website’s plugin sidebars would need TLC. The release-version.sh script would need TLC. All the CI builds would need TLC. Plugins such as the license-maven-plugin, formatter-maven-plugin, and many others, would need migration and/or TLC. The jgo launcher would need TLC. Other things I’m not thinking of off the top of my head would need TLC. This leads me to the next point, which is most important:

  2. The cost/benefit analysis of switching does not come out in Gradle’s favor right now. It’s hard to estimate much time it would take to switch, but it would be similar in scope to the switch I did of all 300 projects from Jenkins to Travis CI, and similar to Fiji’s switch from one Git repository to hundreds. Each of those took months of time. For the Gradle switch, of course I would write a script that converts each pom.xml to build.gradle and run it en masse. But I’d need to develop new pom-scijava-base and pom-scijava build.gradle parents (assuming Gradle works that way?) encapsulating all the SciJava conventions, so that downstream build.gradles are as slim as possible while ensuring they declare all the metadata our community needs. Anyway, I’d expect 100+ hours of my time needed—maybe as much as 500 hours, depending on all the wrinkles that emerge.

    What is the opportunity cost? If my time is spent migrating to Gradle, what wouldn’t I be doing with that time instead? Right now the answers are: 1) getting PyImageJ 1.0.0 ready in time for the I2K conference in <1 month; and 2) completing the ImageJ website migration; and 3) supporting the existing developer ecosystem. And what benefits would we gain for this update to Gradle? We have the pros above. Other potential advantages you listed include:

    • Build times. But Java builds are already incredibly fast, and our components are very modular. It’s simply not a bottleneck, and there’s no need to optimize it.
    • More flexible. But 99.9% of projects in this ecosystem would not benefit from it, and might potentially even be impacted negatively from the standpoint of leading users toward doing the best thing.

  3. Maven is not blocking people in the SciJava ecosystem from using Gradle. If you want to use Gradle with your SciJava project, go ahead. I was happy when scenery was using Gradle since it served as a test bed for potential future migration of additional projects—and honestly a little disappointed when @skalarproduktraum needed to switch to Maven instead. If scenery and/or sciview are now able to go back to Gradle successfully, great! If other plugins are able to be built with Gradle, great! As far as I’m concerned, the discussion here is only about en masse migration to Gradle of the core Fiji software stack.

So: what am I missing? What concrete benefits would we gain as a community from switching to Gradle now? Please help me to understand.

8 Likes
#3

I believe -Werror has nothing to do with maven-enforcer-plugin

This must be a joke :slight_smile:

I must repeat: “no-one from Apache Calcite community raised declarative vs non-declarative thing”. That is exactly what I said.
I do not say “declarative vs non-declarative” programming does not exist.
What I say is:
a) No-one cared (for a good reason, and I am extremely happy I could avoid those discussions because they, well, are useless)
b) Gradle build can be “declarative”
c) Gradle build provides better means to structure the build
d) I agree there are megatons of extremely poor Gradle builds in the wild

For instance, most of the time when I see Groovy-based build, and/or Groovy-based Gradle plugin it is a stong sign that the plugin was written long-ago when proper Gradle conversions did not exists. The rare exceptions are like “plugin written by a Gradle core developer”.
On the other hand, Kotlin-based scripts/plugins are typically better.

Are you sure 100% of Maven-based builds in the wild are good and shiny?
I am sure there are awful-based Maven builds.

That is why I believe “Gradle is not declarative, so Gradle is bad” is a very weak point.
The same goes for “Maven is declarative, so Maven is good”. That is weak again.

I believe Gradle-based build can be declarative enough with very little effort, so “declarative vs non-declarative” is one of the least important points to compare.

Could you clarify what do you mean by TLC?

Gradle produces and deploys the same (or better) pom.xml, so tools could still parse XML

Just in case, I’ve no idea :slight_smile:

One of the helpful Gradle features is composite builds where you can import multiple projects as if it was a single one. That enables to edit them at once, run tests, etc, etc, without changing versions in pom files.

#4

Nope. I just saw this announcement:

Third party I guess, but maybe useful.

Why did you think I was joking? Is there some reason Gradle doesn’t need this functionality?

I apologize for the offense. I was responding to @gBarbieri’s statements, not yours—specifically, his claim about “declarative vs imperative” dualism having “no practical implementation in reality.”

Right. Let’s not debate this point any further in this thread. :smile:

Sorry for the slang. It stands for “tender loving care.” It means I’d have to invest time on adapting that component to the updated system, to keep that component “happy” and functional.

Cool. How better?

Thanks, makes sense. I think that feature would make certain members of our developer community very happy.

3 Likes
#5

Andres is smart, however, I believe they have too Mavenish mindset, so lots of Andres’ decisions are against idiomatic Gradle of doing things.
Here’s a recent case right from Andres’ case with their enforcer: https://discuss.gradle.org/t/resolving-configurations-in-a-single-project-results-in-error/38080/5

That is insane. I don’t want my build script to resolve and download test dependencies when I ask the build script to check Java codestyle.

I more-or-less agree with that :slight_smile:

For instance, Gradle-generated poms are cleaner: they do not include “build instructions” which make no sense for the consumers.
Gradle-generated poms can inline properties.

Gradle-generated pom: https://repo1.maven.org/maven2/org/apache/calcite/calcite-core/1.26.0/calcite-core-1.26.0.pom
Maven-generated pom: https://repo1.maven.org/maven2/org/apache/calcite/calcite-core/1.21.0/calcite-core-1.21.0.pom

Gradle has way better metadata when publishing artifacts. For instance, Gradle metadata can convey information on features. For instance, if signing feature requires dependencies a, b, and c, then Gradle can publish that, and consumers do not need to know the exact set of dependencies. The best Maven has is “optional” dependencies which is like “go figure which ones you need” (see https://blog.gradle.org/optional-dependencies)

Gradle has API vs implementation separation, so the published artifacts are not pushed on the compilation classpath for the end-users. In Maven every transitive dependency ends up on the compilation classpath. That slows down compilation, and it creates possibility for end-users to accidentally depending on a dependency.

Gradle does not need to edit multiple pom.xml to change version. The version could be specified in a property file, or it could be taken from a command line. The version could be taken even from a Git tag. Frankly speaking, I hate that Maven’s “bump version” results in altering lots of files (it might be improved recently, however, it is still very limited there).

Gradle enables us to use the same language for automation scripts. For instance, I migrated Apache JMeter from Ant to Gradle, and, as a part of that, I migrated release.sh-kind-of script to Gradle’s Kotlin. That was non-trivial (I had to reverse-engineer JMeter’s Ant-based build), however, now the release is 42 times easier.
I hate that .sh scripts are very platform-specific, and there are subtle differences even in trivial commands like find , xargs, awk, sed for macOS vs Linux. Then, go figure how to run that script with Windows. I find that Kotlin enables to avoid fighting with platform specifics, and it integrates nicely with the rest part of the build.

Here’s the feedback from Milamber, who was the release manager for JMeter for a very long time, and it was the first time Milamber used Gradle https://lists.apache.org/thread.html/c08780952b7fbd71eccff717cb494562723e11dc99ea1b4bb39af461%40<dev.jmeter.apache.org>

That is it. A single command builds artifacts, pushes to Nexus, pushes to ASF SVN, prepares a draft mail text.

Creating Gradle plugins is WAY more enjoyable than creating Maven plugins. Plugins are easier to write, easier to test, easier to use, etc, etc.

That is true. It would take time.

#6

Here’s another case for “small automation”.
In JMeter we have ./gradlew runGui task which launches JMeter right from the source code.
That is it builds everything, and launches the application (as you might guess JMeter is very picky to the file layout since it was invented long ago and it did not change much for ages). I think Maven has no ways to tell "ok, take compileClasspath, put project jars here, other jars there, and launch this java process like that). Note: it does not execute unit tests. It just builds the needed jars, copies them, and launches the app.

If you alter dependency version (e.g. bump Jackson), a single ./gradlew runGui is enough to re-launch JMeter with the updated jar.

Here’s how runGui is declared: https://github.com/apache/jmeter/blob/150a2753b4bb055c678965a1ec9a66af3362345d/src/dist/build.gradle.kts#L592-L595

The same in Calcite. There we have ./sqlline script that builds Calcite and launches SQL shell to access it. Gradle-based variation is way easier, and it has less platform-specific issues that its previous Maven-based one.

#7

One more point, Gradle has way better help for those who would try running build tasks.

For instance, ./gradlew tasks in JMeter produces

> Task :tasks

------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------

Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles main classes.
clean - Deletes the build directory.
createDist - Copies JMeter jars and dependencies to projectRoot/lib/ folder
generatePom - Generate pom.xml files (see build/publications/core/pom*.xml)
jar - Assembles a jar archive containing the main classes.
renderLicenseForBinary - Generate LICENSE file for binary distribution
renderLicenseForSource - Generate LICENSE file for source distribution
testClasses - Assembles test classes.
testFixturesClasses - Assembles test fixtures classes.
testFixturesJar - Assembles a jar archive containing the classes of the 'testFixtures'.

Build Setup tasks
-----------------
wrapper - Generates Gradle wrapper files.

Coverage reports tasks
----------------------
jacocoReport - Generates an aggregate report from all subprojects

Development tasks
-----------------
runGui - Builds and starts JMeter GUI

Distribution tasks
------------------
distTar - Creates binary distribution with LF line endings for text files
distTarSource - Creates source distribution with LF line endings for text files
distZip - Creates binary distribution with CRLF line endings for text files
distZipSource - Creates source distribution with CRLF line endings for text files

Documentation tasks
-------------------
groovydoc - Generates Groovydoc API documentation for the main source code.
javadoc - Generates Javadoc API documentation for the main source code.
javadocAggregate - Generates aggregate javadoc for all the artifacts
previewPrintableDocs - Creates preview of a printable documentation to build/docs/printable_preview
previewSite - Creates preview of a site to build/docs/site
testFixturesJavadoc - Generates Javadoc API documentation for the 'testFixtures' feature.

Gradle Enterprise tasks
-----------------------
buildScanPublishPrevious - Publishes the data captured by the last build.
provisionGradleEnterpriseAccessKey - Provisions a new access key for this build environment.

Help tasks
----------
allDependencies - Shows dependencies of all projects
allDependencyInsight - Shows insights where the dependency is used. For instance: allDependencyInsight --configuration compile --dependency org.jsoup:jsoup
buildEnvironment - Displays all buildscript dependencies declared in root project 'jmeter'.
components - Displays the components produced by root project 'jmeter'. [incubating]
dependencies - Displays all dependencies declared in root project 'jmeter'.
dependencyInsight - Displays the insight into a specific dependency in root project 'jmeter'.
dependentComponents - Displays the dependent components of components in root project 'jmeter'. [incubating]
help - Displays a help message.
kotlinDslAccessorsReport - Prints the Kotlin code for accessing the currently available project extensions and conventions.
model - Displays the configuration model of root project 'jmeter'. [incubating]
outgoingVariants - Displays the outgoing variants of root project 'jmeter'.
projects - Displays the sub-projects of root project 'jmeter'.
properties - Displays the properties of root project 'jmeter'.
tasks - Displays the tasks runnable from root project 'jmeter' (some of the displayed tasks may belong to subprojects).

IDE tasks
---------
cleanEclipse - Cleans all Eclipse files.
cleanIdea - Cleans IDEA project files (IML, IPR)
eclipse - Generates all Eclipse files.
idea - Generates IDEA project files (IML, IPR, IWS)
openIdea - Opens the IDEA project

Publishing tasks
----------------
publish - Publishes all publications produced by this project.
publishAllPublicationsToNexusRepository - Publishes all Maven publications produced by this project to the nexus repository.
publishToMavenLocal - Publishes all Maven publications produced by this project to the local Maven cache.
publishToNexus - Publishes all Maven publications produced by this project to the 'nexus' Nexus repository.
pushPreviewSite - Builds and publishes site preview

Release tasks
-------------
prepareVote - Stage artifacts and prepare text for vote mail
publishDist - Publish release artifacts to SVN and Nexus
removeStaleArtifacts - Removes stale artifacts from dist.apache.org (dry run with -PasfDryRun)

Verification tasks
------------------
autostyleApply - Applies code formatting steps to sourcecode in-place.
autostyleCheck - Checks that sourcecode satisfies formatting steps.
batchTests - Executes all the batch tests (add -PallBatch to see individual batch tasks)
check - Runs all checks.
checkstyle - Executes Checkstyle verifications
jacocoTestCoverageVerification - Verifies code coverage metrics based on specified rules for the test task.
jacocoTestReport - Generates code coverage report for the test task.
rat - Runs Apache Rat audit tool
spotbugsMain
spotbugsTest
spotbugsTestFixtures
style - Formats code (license header, import order, whitespace at end of line, ...) and executes Checkstyle verifications
test - Runs the unit tests.
verifyReleaseDependencies - Verifies if binary release archive contains the expected set of external jars

Of course, the list is not that perfect yet, however, Gradle enables to fine-tune the list of tasks that are shown here, and there can be project-specific explanation.

For instance, launch GUI is important task for trying JMeter, so there’s an entry for that:

Development tasks
-----------------
runGui - Builds and starts JMeter GUI

It is way way more accessible than Maven’s command line.

One more case: PostgreSQL JDBC driver is very conservative (it is like “almost zero” dependencies), and is more-or-less trivial build sequence.
However, it is important that test results and output is easy to analyze.

Gradle enables customization for test output, so I created a small plugin that adds coloring and detects slow tests: https://github.com/vlsi/vlsi-release-plugins/blob/6af213a405048b7145ce4a4840ba260d878c4a1e/plugins/gradle-extensions-plugin/README.md

That enables to get meaningful information even from the very simple logs at Travis, GitHub Actions, etc.
Here’s a case from a recent PR failure: https://travis-ci.org/github/pgjdbc/pgjdbc/jobs/740769399#L6163

Even though I agree not every project needs detailed test output, however, testing is basically the only and the most often used thing in pgjdbc, so printing details helps a LOT.

I believe making similar customization with Maven is hard to impossible. How do you do that? Fork maven-surefire-plugin? Contribute a fix to maven-surefire-plugin? In theory, it is doable, however, in practice, it is not :frowning:

Just in case, there’s Gradle Build Scan feature that provides extra visibility on the build, logs, etc (e.g. https://scans.gradle.com/s/xy2bw3skgcxuo )

#8

Hi Curtis,

it’s me again…

a couple of things happened in the meanwhile, we finally merged the Gradle branch into master on Scenery, nothing exploded so far, Ulrik is still happy :upside_down_face:

faster build time, local and CI ones, tooling integration and plugin development is a breeze, while being concise, statically typed and full of DSLs is a nice bonus as well.

Now, to address some of the cons you mentioned

Some of the TLC would be much easier and quicker than you think, I assume.
But true, a complete switching of the whole community doesn’t make sense at the moment, however planning something might pay off in the long run.

Gradle can automatically generate a script from a pom, it has a richer way of defining dependencies and offers multiple ways to solve version conflicts (Maven has the nearest first strategy only).

Gradle extends much further over GAV coordinates, with variance (1 and 2), expressed via attributes saved in a metadata file alongside the pom.

Also, Gradle is truly boosted and driven by its huge community

continue to overcome link limit

#9

Anyway, coming back to the cost/benefit, I think that some of the cost you are taking in account right now is implicit in using maven and its constructs.
Looking at the imagej and scijava github organizations I have the feeling there is tons of duplications and dispersion (take this with a lot of salt though, you know I’m quite new to these worlds).
To say, imagej might become a multi module project, the project names pattern suggests me that first.
I gave a try and got it compile quickly, quite some tests pass already.

Moreover, when you talk about build times, I tend to imagine builds at higher level (with CI, code coverage, signing, publishing, etc etc) and conglomerates (multi-module projects indeed). In these scenarios the difference will be larger, both time-wise that use-wise (practicality). Besides, Gradle has very fined caching mechanisms, both at build time, but also at configuration phase, overall performances is a strong thematic.

Gradle 6.4 brought JPMS first class support, this could make a big difference in a big environment such as the sci community, delivering releases with only the modules that one really needs.

While in the meanwhile melting pot problems are boosting multiple solutions, Gradle 7 on 26 February will bring on the table built in support for catalogs, a centralized dependencies declaration. Also the community is gathering among multiple plugins (refreshVersion and gradle-versions-plugin)

Since dependency management is so complex and Gradle is struggling to catch up with all the issues and wishes of its users, and after I talked briefly with Mark last Friday, I’m convinced is not worth to implement (and maintain) your own system for handling dependencies and their updates (via the update sites).
Probably Imagej should be unified, and the sciJava core modules as well in its own. Then, each additional plugin, especially the external ones, should have unit tests and functional tests (for those which makes sense), where it’s tested at Fiji level distribution (or similar). But again, pinch of salt :grimacing:

This way library thematic would be much less critical and safer.

Last but not least, I think Gradle might be more attracting for developers to join in and bite the bullet.

#10

Hey Giuseppe @gBarbieri ,

I’m just curious, is there any reason why you think the ImageJ/Fiji/SciJava ecosystem should be gradled? I’m not sure if you’re aware: You are talking about updating hundreds of repositories and convincing a hundred developers.
I spent some time ago switching a couple of projects from gradle to maven (they were all named ClearSomething, ask Ulrik about them :wink: ), mostly for performance reasons, to standardize them and to prevent buggy maven-gradle-mixture projects. Obviously, maven and gradle both have advantages and disadvantages. After switching to B, you wish to go back to A and vice versa.

So I’m wondering why you think the switch is necessary… Is there anything you can’t do with maven where we might help you with?

Cheers,
Robert

3 Likes
#11

Hey Robert,

I still don’t have a strong preference, except that more recent JVM-based projects I run into are most often Gradle-based.

The main selling point that I’ve run into is based on coupling scenery and sciview versions. As you know, version properties from dependencies do not propagate and that is a frustration for projects that depend on sciview.

  • pom-scijava has versions specified for both sciview and scenery
  • in the scenery world we often have extended bursts of development where scenery and sciview are getting independently updated a few times per day, we always have to link sciview to a scenery version
  • these development bursts are motivated by downstream projects these days. when we go to update the downstream project we always have to set versions for both sciview and scenery instead of just being able to update the sciview version (if you don’t do that then you get an outdated scenery version from pom-scijava)
  • it is not realistic to keep pom-scijava updated during these bursts of development
  • since the maven approach requires this to be done by developers manually it is easy to make mistakes and lose time debugging due to a missing version update
  • it is important for us to be able to maintain this development approach because of our heavy reliance on jitpack. using this approach you don’t have to do risky things like making sure 4-5 people maintain identical snapshot couplings while switching between multiple branches and trying to work on both scenery and sciview at the same time, instead you just send people jitpack hashes and everything is reproducible during the dev process.
1 Like
#12

Not sure what you mean with that. Maybe this solves it: I’m very happy with having an own parent pom, where I do specify versions:

#13

Addendum:… such a thing is also possible in gradle: :slightly_smiling_face:

#14

My 2 cents, advocating for a conservative view.

I’m clearly not as advanced as all of you in these dependency management mechanism. So I may not see what we could lack in the future by sticking to maven.

But a switch to gradle looks a bit scary to me. I actually like the fact that maven is pretty stiff, not very modular. I understand pretty much everything which is in a pom file, whereas I find harder to understand what happens in a gradle configuration file. Actually it doesn’t look like a config file, but rather like a real programming language (and it is - it’s groovy).

I understand that some of you want to use gradle in very big projects like sciview and scenery. If you need to update many repositories quickly, I understand that it’s convenient, and also probably if you need to deal with native libraries ?

Maybe regarding your initial points ( don’t take this personally - I exaggerate on purpose from my experience):

  • flexibility : I clearly don’t see that as an advantage
  • performance : you mean build time ? with build times of a 1 minute max on a small laptop - no issues so far.
  • user experience : being not flexible seems to give a better user experience
  • dependency management : with versions declared in pom and scijava maven repo + travis, I don’t see how that could be better

Now if gradle can work in conjunction with maven, why do you want a switch for everything else ?

3 Likes
#15

Yeah, the other option for us was to make our own parent pom, which I was doing for a little while. However, I think there are some Kotlin related arguments in favor of Gradle as well, but @gBarbieri and @skalarproduktraum can say more about that.

Plus, if we’re hooked on IntelliJ, Kotlin, and Gradle, then maybe JetBrains will try to acquire us!

2 Likes
#16

Let us know if we can help you fixing those issues :slightly_smiling_face:

#17

Hey Robert,

other than what I already said, I can narrow it down to the fact that essentially Maven resolution strategy is broken by design, dependent to the declaration order (literally) of the multiple dependencies in the pom, forcing developers to weird workarounds with excludes in order to overcome its nearest resolution.

Its conflict resolution is incomplete and the exclude on a single transitive dependency is applied to the whole graph.

Moreover, the pom metadata design has flaws, pulling sometimes useless dependencies in the (compile) configuration that has nothing to do with and the alignment is also incomplete and the optional turns out to be just documentation.

It’s funny because I didn’t know all thiese things since a while ago, but after researching and investigating now I have much more confidence over this.

I also cant stand xml in the year 2021 :d, but I know this is a personal taste.

If you have ~ one hour spare, I strongly suggest you to look this video

Hundreds of repositories are a tough starting point, but they would also require state-of-the-art dependency alignment and conflict resolution.

All of this contributes to the melting pot you are having, slowing down the development cycle(s) and making you folks lagging behind, especially in terms of supporting the latest java jdk versions (in less than 10months we’ll be 3 LTS versions behind while commercial support by Oracle is already expired).

Please don’t get me wrong, what has been developed totally made sense and was advanced at the time when it was designed and I’m sure it served the users fantastically for many years, but nowadays is clearly showing its age and its limits.

My 2c from a sciJava profane : )

Hi Nico,

I’d suggest you as well the video I linked above

Gradle has its shenanigans and dynamically typed languages such as Groovy even more. However the advantages outweighs the versus and now it comes also Kotlin to the rescue : )

Moreover, having a real programming language to express build configuration means having type safety, less error prone, completion, highlighting, navigation, refractoring, documentation, etc etc…

Sure, dont worry, Nico, it makes absolutely sense. But let me put those points under another perspective.

Flexibility: I wrote a parser for the scijava pom xml with the jdk utils in some tens of lines. I’m also attempting to migrate the lwjgl ant compilation. Interacting directly with shell commands from the build script turned out to be relatively easy and straightforward.

Performances: you are used to 1m max. Now try to think if that would take max 10s. It’s another level, you don’t have to swap your brain to something else in the meanwhile, if you were doing it of course.

Reduced flexibility. Here we enters conventions and good practises over enforcements.

Dependency could be better, I’m pretty confident (last famous words :p)

Maven can indeed coexists along side with Gradle, you are totally right, so let me rephrase that: I’d like (you folks) to give Gradle a try

1 Like
#18

I don’t see how. Gradle will not solve this issue. We would still need the melting pot builds, even with Gradle instead of Maven. And the only way to speed up release cycles is to invest maintainer time. Gradle will not eliminate NoSuchMethodError or ClassNotFoundException—Java (and/or Kotlin) code must be updated. In short: we are lagging behind because I am lagging behind. Not because of the tools.

No, we are one LTS version behind: built on Java 8 rather than Java 11. Java 17 LTS is planned for release in September, at which point we will be two versions behind, assuming we haven’t completed the Java 11 migration yet. But which Java version we use is orthogonal to which build system we use.

I agree with @NicoKiaru about the build times. SciJava Common builds in 4.6s (without tests) on my machine with Maven. If it builds 10x faster (0.46s) with Gradle, it really doesn’t matter for me—even more so due to nigh-instantaneous incremental compilation from IDEs. With tests, SJC takes 8.7s. ImgLib2 builds in 5.3s sans tests, or 14.1s with tests. When unit test time dominates execution that much, the build system stops mattering too much. So I don’t think performance is a good argument for Gradle here.

Most emphatically: as a long-time software maintainer, it’s never easier and quicker than I think. My time estimates remain terrible, and I always under-anticipate the negative impact on the community. When switching to Gradle, you need to measure not just the time of the conversion (which is what you’re getting a very good sense of now, @gBarbieri, since you’ve been doing it), but also all the ripple effects afterward. That includes all the stuff that will break downstream, and all the community support that will need to happen over time. Trust me: this switch would/will be a disruptive change, and will absorb many hours of core maintainer time for years to come afterward.

Don’t get me wrong—I’m not trying to argue against Gradle in general. Just pushing back on the points which I feel are misleading. I’m still going to give Gradle a serious try as soon as I can. My time is very limited right now, but I love playing with tools. :smile_cat:

2 Likes
#19

We shall have an umbrella project, in gradle terms a software product, which is based on multiple software components, such as imagej, fiji, imglib2, sciJava, ome, scifio, etc etc.
Each of this component would be based, in turn, on its corresponding modules.

{ I took a look at imagej and I think the inner modules (imagej, -common, -launcher, -notebook, -ops, -updater, -plugins*, -mesh-io, -legacy, -deprecated, -scripting, -ui*) should be part of a so-called multi project, naming pattern and especially the dependency between them suggests so.
However since you already had this and you went down the modularization way, there is no problem keeping them separated as they are, if independent versioning is important for you folks }

With this umbrella project you may directly modify anywhere the code, from the latest plugin to the deepest core logic and see the effect immediately.
Any NoSuchMethodError or ClassNotFoundException would be ruled out on the spot.
Also, you would be able to run all the tests of each single component and module with a single command.

Mea culpa, I got mystified from the java 15 highlighting, you are right.
However testing the whole ecosystem with a new jdk would be just a matter of changing a single parameter in the build and then run compilation and tests.

At this point you should understand how much would performances and scalability matter.

Yeah, that’s may be true, however since Gradle extends and is fully compatible with Maven, you actually do not have to (force to) migrate anyone. Who wants to keep using Maven is absolutely fine to do so.
You should just implement Gradle along side.

You are going to have a lot of fun, dude :stuck_out_tongue: