[FOLIO-2885] SPIKE: consider improvements to the release process Created: 19/Nov/20  Updated: 16/Mar/21  Resolved: 16/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: Jakub Skoczen Assignee: John Malconian
Resolution: Done Votes: 0
Labels: devops-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-3023 better CI/GitHub integration Open
relates to FOLIO-3024 better Jira/GitHub integration Draft
relates to DEBT-8 Scalable Continous Deployment infrast... In Review
Sprint: DevOps Sprint 109, DevOps Sprint 107, DevOps Sprint 108, DevOps Sprint 106
Development Team: FOLIO DevOps

 Description   

Consider what can be done to improve the process of making individual module releases.

This is especially problematic for bugfix releases where POs would like see individual bugfixes released and deployed (e.g to the bugfest env) as soon as they are ready, while development teams (and devops) prefer that multiple bugfixes are bundled into a single release to minimize the preparation (and deployment) overhead.

During the DevOps Team standup we have discussed the following improvement ideas:

  • simplify the release process (tagging, building, publishing artefacts) to the point where it becomes as simple as running a single job in Jenkins, with all tasks automated. Note: we discussed that certain tasks like preparing release notes cannot/should not be automated so the overhead cannot be removed completely
  • create a new branch of platform-complete that uses artefacts automatically built from release branches (branches created when the FOLIO release process starts) whenever there is a commit to the branch. Switch the bugfest to automatically deploy new modules when there is a new "release branch" artefact created. This would allow individual teams avoid making any bugfix releases until the bugfix process is considered complete while at the same time allow POs to test each individual fix on bugfest as as soon as it completed and merged to the release branch.


 Comments   
Comment by Zak Burke [ 09/Dec/20 ]

Notes:

  • At minimum, publishing a release from a developer's workstation requires three sets of credentials (GitHub, Jira, Jenkins), if not more (Slack to post to #releases, GitHub API to create release notes).
  • Most developers do not have permission to create new versions in Jira. Hence, in order to correctly set the fix-version for a ticket, they must ask their tech lead to create the necessary version.
  • Tags take some time to propagate from GitHub to Jenkins, so while there is an HTTP API for triggering builds in Jenkins, it has to be engaged thoughtfully. A script that pushes a tag to GitHub and immediately calls the Jenkins API to build from that tag will inevitably fail.
  • The Jira API has no callbacks/webhooks registry, so the last four steps are impossible to automate, at least from a developer's workstation, without polling.
  • Finding the tickets that need their fix-version set can be tricky. If developers are good about starting commit messages like FOO-1234 frob the foo, this can make it possible to use git to find the tickets that need to be labeled. Note, however, that the release ticket is not associated with git at all and therefore cannot be found in this way, so it needs to be tracked and updated separately.
  • Outside of "push the tag to GitHub, build from the tag in Jenkins", there is a metric shittonne of Jira work that goes into a release and it would be oh so lovely to automate some of that away:
    • in Jira, create a patch version
    • in Jira, create a release ticket
    • in Jira, set the release-ticket's "Epic Link" field
    • in Jira, assign the version to relevant tickets, i.e. both the code tickets and the release ticket
    • in Jira, IFF the Jenkins build completes successfully, release the version
    • in Jira, IFF the Jenkins build completes successfully, close the release ticket (or set its status to "Awaiting deployment")

Questions:

  • Could some of this machinery be invoked by pushing to a different git remote or to a particular branch? At present, we maintain release branches such as b3.4 and b3.5, etc. What if in addition we maintained a releases branch so that any commit to this branch triggered a release process?
  • Could we make use of the built-in npm version command, which can handle package.json updates, git commits and tags? This might necessitate using true patch versions in CI, e.g. 3.4.0-alpha.1 in place of the current 3.4.0000529, which has a lot of ripple effects, but would probably be a Good Thing in the end.
Comment by Julian Ladisch [ 10/Dec/20 ]

I suggest to use semantic versioning instead of release branches like b5.6-honeysuckle and b5.6-iris.
If a module mod-foo doesn't have new features it uses the b5.6 branch for the version 5.6.7 that is used in honeysuckle and iris.
The two release branches b5.6-honeysuckle and b5.6-iris need to be manually maintained to always contain the same content. This is work and it is error-prone; this should be avoided.
Semantic versioning can be used instead:
If the bugfest environment runs mod-foo-5.6.7 or mod-foo-5.6.7-SNAPSHOT and a new mod-foo-5.6.7-SNAPSHOT or mod-foo-5.6.8-SNAPSHOT artifact is created then deploy it into the bugfest environment.
The rules:

  • major and minor version matches
  • bugfix version is greater than or equal to bugfix version on bugfest
  • it is a -SNAPSHOT version.

These artifacts are not automatically deployed to bugfest where the current version is mod-foo-5.6.7 or mod-foo-5.6.7-SNAPSHOT:

  • mod-foo-6.0.0-SNAPSHOT (major version mismatch)
  • mod-foo-5.7.0-SNAPSHOT (minor version mismatch)
  • mod-foo-5.6.6-SNAPSHOT (bugfix version is smaller, we don't want automatic downgrade)
  • mod-foo-5.6.7 (we don't want to downgrade from mod-foo-5.6.7-SNAPSHOT)
Comment by Julian Ladisch [ 10/Dec/20 ]

Tags take some time to propagate from GitHub to Jenkins

We should consider using GitHub Actions to build, test and publish artifacts. GitHub Actions are generally available since March 2020. GitHub Action Minutes are free for public repositories.

Merging a pull request or publishing a release can trigger an action:

https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/migrating-from-jenkins-to-github-actions
https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-nodejs
https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-java-with-maven
https://docs.github.com/en/free-pro-team@latest/actions/guides/publishing-nodejs-packages#publishing-packages-using-yarn
https://docs.github.com/en/free-pro-team@latest/actions/guides/publishing-java-packages-with-maven
https://docs.github.com/en/free-pro-team@latest/actions/guides/publishing-docker-images

Comment by John Malconian [ 10/Dec/20 ]

Lots of good ideas and suggestions here. I think there is potential for better Jira/CI/CD integration. I am also interested in taking a look at Github Actions to eventually replace or at least offload Jenkins.

Comment by Marc Johnson [ 11/Dec/20 ]

Julian Ladisch

I suggest to use semantic versioning instead of release branches like b5.6-honeysuckle and b5.6-iris.
If a module mod-foo doesn't have new features it uses the b5.6 branch for the version 5.6.7 that is used in honeysuckle and iris.

I did not know that teams did this, it doesn't seem to fit with the current release guidance.

That said, I wouldn't expect (with RAML Module Builder and stripes upgrades each release) that many modules would retain the same version across official distributions.

The two release branches b5.6-honeysuckle and b5.6-iris need to be manually maintained to always contain the same content. This is work and it is error-prone; this should be avoided.

This seems like a particularly awkward anti-pattern. There should not be more than one definition of a module version.

Comment by Jakub Skoczen [ 14/Jan/21 ]

Julian Ladisch Marc Johnson Using branches like b5.8-iris is NOT aligned with our release guidelines. Who is doing that?

Comment by Julian Ladisch [ 14/Jan/21 ]

This has been proposed in the first post, see above:

What if in addition we maintained a releases branch so that any commit to this branch triggered a release process?

Comment by Marc Johnson [ 14/Jan/21 ]

Julian Ladisch

This has been proposed in the first post, see above

Who / what is this comment a response to? Is it to Jakub Skoczen question about who is using a branch like b5.8-iris?

Generated at Thu Feb 08 23:23:58 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.