Changelogs / release notes for ImageJ / Fiji / SciJava & friends

Hi all,

let’s discuss how to move towards changelogs / release notes for ImageJ / Fiji / SciJava repositories. It would be great to come up with a semi-automatic strategy that can be used by core developers and contributors.

In gitter we discussed this already to some extend and want to continue here.

Some of my thoughts:

Where do we need changelogs?

There could be any subset of these options (tell me if I missed one):

  • a changelog for each (core) repository, attached to releases or in a separate file
  • a changelog for each Fiji release, maybe living on the Wiki
  • a changelog entry for each upload to an update site, included in the db.xml file

What’s part of the changelogs?

  • user related changes
  • developer related changes (distinguishing between fixes / features / breaking changes)
  • responsible developers / contributors

How are they created?

  • manually (per release / upload)
  • semi-automatic: we retrieve descriptions of changes and give the option to edit them before writing them into a changelog
  • fully automatic: no edit step in between

How would a (semi)automatic changelog work?

Retrieve commit messages

  • pro: I got the feeling that for many ImageJ repos the commit messages are more informative than the PRs (if the PR even exist)
  • contra: they are not editable without rewriting the history. I think this is a mayor downside.
  • contra: there would be way more commits than PRs to review before releasing a changelog

Retrieve PR messages

  • pro: editable
  • pro: we could use a template
  • pro: with templates the PR readability would improve and it might be easier to review them
  • contra: not always a PR for all changes
  • contra: we need to change our habits / this is additional work for maintainers to enforce a template

Technically both is possible, I played a bit around with retrieving commit messages, here is my test.

The scikit-image people have a semi-automatic approach using PR messages. I think we can learn a lot from them. Have a look at their PRs, their script to generate release notes and the release procedure where the script is used.

5 Likes

One more note: I’m also writing down a plan for how we could get into a stable release cycle for Fiji - whatever we come up with here could be embedded there. I’ll post the google doc soon.

2 Likes

Thanks a lot for starting this discussion @frauzufall
Inspired by your test, I wrote a small script to auto-generate release notes for Paintera. This works for now but I will happily apply a community-agreed-upon standard once it is ready for use. I have a strong opinion that the notes should be generated from PR messages. It is my opinion that any project that does not yet add changes through pull-requests from meaningful feature branches should adopt that practice anyway. Anything else is just not good practice and it is going to be hard to retrieve meaningful and coherent breaking change/feature/patch reports from individual commits.

I was not aware of the scikit-image solution but I ended up with a similar scheme: https://github.com/saalfeldlab/paintera/pull/292

As far as I can tell, there are two minor differences:

  1. A merged PR can have multiple tags (e.g. BREAKING and PATCH) in my scheme
  2. scikit-image lists all commit messages

That being said, a summary of my opinion: Changelogs should be generated from pull requests and projects should be maintained in a compatible way.

3 Likes

Adding an example of how my auto-generated release notes look like (after a few manual edits): https://github.com/saalfeldlab/paintera/releases/tag/paintera-0.18.1

Coming up in a twitter conversation, springboot seem to provide such a framework already: https://github.com/spring-io/github-release-notes-generator
I haven’t looked into it yet but a quick glance at the README suggests that this is exactly what we are looking for.

Here are the reasons—in my opinion—why we do not currently have a changelog process in place for components of the Fiji software stack:

  1. There are hundreds of components in Fiji, ImageJ, ImgLib2, SCIFIO and SciJava. I sometimes cut dozens or hundreds of releases in a short period of time. If I had to write the changelogs manually, or even semi-manually, then new releases of these components would be even less frequent than they currently are. The scikit-image release process works because the number of components being released is small, with relatively infrequent releases. I need a changelog process that scales, or I won’t be able to use it.

  2. We do not always use a PR-based workflow when merging new features. Small features are sometimes done as single commits directly to master, described appropriately in the commit message. Large features are sometimes done as non-fast-forward topic branch merges without filing a PR. I appreciate the perspective that using PRs makes things tidier, but it also slows down development, especially if review is always required. The reality of our community is that we simply do not have enough reviewers to double-check all the work being done in a timely manner—even ImgLib2, which is one of our most carefully maintained projects (thanks, @tpietzsch!!), currently has 10 open PRs spanning back almost two years. IMO, it is not a black-and-white issue. An automatic changelog generator could look for merges of topic branches as well as single commits to the master branch, especially if the commit messages are sufficiently descriptive.

  3. All the automatically generated changelogs I have seen are clearly written by a machine, and do not read well—to the point of being IMO nearly useless. Take for instance the Apache Maven release notes (example). It is simply a release announcement, followed by links to all issues that were closed during the release. The way I’d like to achieve such a list is through more robust use of GitHub Issues and milestones, such that each issue when fixed gets assigned to a milestone whose name matches the first release including the commit that closed the issue. For example, suppose commit abcdef12 closes issue #35 in my foo/bar project. Then when I next cut a release of bar, say at version 1.2.3, a 1.2.3 milestone is created (if it doesn’t already exist) and all newly closed issues including #35 get moved to that milestone—which can be detected automatically because bar-1.2.3 is the first tag containing abcdef12 which closed #35. In this way, the milestones can now function as the changelog for each version. I briefly describe this at https://imagej.net/Issues in the Milestones section, but in practice no one is doing post-mortem issue classification manually that way. I figured it would be straightforward to write a script to do it retroactively for all issues across all SciJava projects, and voila: instant changelogs. But of course, this only handles things filed as issues, not direct commits adding some feature that was never documented or discussed in an issue tracker.

So I do think there are ways we could move forward with automation and tooling of changelogs. But I think it’s a tough thing to do well. The route I’d go is described above in (3), as opposed to scraping git histories and pull request descriptions, because it would be easy to code, (nearly) no effort on maintainers once it’s coded, and still be driven by the more human level of GitHub issues whose purpose is to track work on the project at a similar level. But whatever we do, it has to strike a very difficult balance between A) adding the minimal process overhead possible; while B) resulting in a changelog that offers some value beyond just browsing each project’s issue tracker and commit history.

An alternative that would be less effort would be forgo generation of a custom changelog file with every release, in favor of simply generating a basic release announcement that provides targeted issue tracker and git history links. Those curious to dig in can click those links. For major releases, we could also add a one-paragraph blurb to the release announcement summarizing what’s new.

1 Like

Hey all,

:heart:

Before we go into detail, my two cents: I’m afraid there is no purely technical solution to this. We need a social solution. We need to change our habits in order to make this happen. I see it mostly from a users/workflow developers perspective: Quite some serious issues could have been prevented or at least workload could have been minimized if there were release notes. I would love to see them coming to the Fijiverse!

IMHO, the person who cuts the release is not responsible for writing release notes. The developers should write a text line down if they introduce major changes, new features or break backwards compatibility. Something like

affineTransform() has been deprecated, 
use affineTransform2D() or affineTransform3D() instead.

is the bare minimum. Github releases and release drafts are a nice tool for that btw.

I think that’s fine as we concluded at some point to have just two releases per year:
https://imagej.net/2016-12-20_-Fiji%2B_KNIP_hackathon#Stable_releases_of_Fiji

This gives all developers (core, plugins and workflows) time to adapt on changing APIs without the fear of breaking workflows at apparently random release dates. Furthermore, also core developers can relax then because the momentum of the release cycle is slowed down.

Cool to see this discussion! :star_struck:

Cheers,
Robert

1 Like

I can see @ctrueden’s concerns but for me that means that I will probably just use/adapt my small scripts for my own projects or have a look at the spring-io github-release-notes-generator. Adding tags to pull request commit messages is very little overhead for maintainers and contributors for my use cases and could also be extended to general commits and it offers more flexibility because it allows to specify what is a patch, new feature, or breaking change. At least for repositories where changes are introduced via pull requests only, this seems like a good solution to me.

1 Like

I substantially agree, although a technical solution can facilitate better social behavior here. For example, the ITK project uses commit message prefixes when committing (e.g. [ENH], [BUG]), so that release notes can be more effectively and automatically synthesized from the commit history. This is in line with what @hanslovsky suggests:

We need to be precise in our discussion of what is being released here. Two stable releases of Fiji as a whole, to users via the update site, yes. That is a very different thing, though, than releases of components to a Maven repository. The entire development workflow of SciJava components is built around the pillar of release early, release often. We have release-ready master branches, so we can release as swiftly and painlessly as possible via the release-version.sh script. We do this to avoid snapshots whenever possible because reproducibility is a primary requirement for us, and also because we believe in “speed of iteration beats quality of iteration.” Two releases per component per year is not in line with this philosophy.

So: are we talking about release notes for the toplevel ImageJ and Fiji projects, for end users, distributed via the update sites? Or are we discussing release notes per component per release to a Maven repository? I thought we meant the latter.

1 Like

Yes, I strongly agree. It makes things a lot easier for me to figure out, what exactly has happened since the last release. Based on that I can either just leave the auto-generated release notes as they are (which is already useful information in itself) or add some more informative text without having to go through the commit history.

That is a very interesting question, indeed. I was thinking the latter, too.

Indeed it would be nice if, for a new release of e.g. Fiji, the changelog somehow included or at least summarized things from its dependencies. Otherwise, the sc.fiji:fiji autogenerated changelogs are going to be quite useless: just a laundry list of bumped component version updates, without explanation of what is fixed. I suppose we could, in the commit messages of fiji.git, write out the major takeaways when component versions are updated, but I think this would become very tedious quite quickly. Another layer of tooling is probably needed in some capacity for “higher level” release notes of components that are aggregations of lower-level components.

Another option could be the Release Drafter GitHub app. It collects merged pull request titles in a draft release, so the release notes are kept ready to publish any time.

3 Likes

@imagejan Nice!

@turekg Do you have the bandwidth to explore ways we could start using Release Drafter or github-release-notes-generator across the Fiji component stack?