[FOLIO-2913] SPIKE: Evaluate Github Actions Created: 11/Dec/20  Updated: 11/Mar/21  Resolved: 11/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: John Malconian Assignee: Ankita Sen
Resolution: Done Votes: 0
Labels: devops-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-3067 Replace Jenkins UI and Stripes CI pip... Closed
Sprint: DevOps: Sprint 105, DevOps Sprint 109, DevOps Sprint 107, DevOps Sprint 108, DevOps: Sprint 104, DevOps Sprint 106
Development Team: FOLIO DevOps

 Description   

Evaluate feasibility of using GitHub Actions as a CI pipeline for individual front-end and backend module repos.

Deliverables

  • pipeline/template for Java/Maven based modules
  • pipeline/template for Node-based module
  • a PoC for using GH action templates instead of individual pipelines for each module


 Comments   
Comment by John Malconian [ 15/Dec/20 ]

There are primarily two Jenkins pipelines that are used for about 90% of all folio-org modules repos.

For Stripes and ui-* Nodejs-based modules: https://github.com/folio-org/jenkins-pipeline-libs/blob/master/vars/buildNPM.groovy
For Maven/Java-based backend modules: https://github.com/folio-org/jenkins-pipeline-libs/blob/master/vars/buildMvn.groovy

The idea for this spike is to develop a POC GitHub Actions workflow for each based on the corresponding Jenkins shared pipelines above.

The high-level workflow for each pipeline essentially consists of the following steps (non-release builds)

  • Check out source
  • Lint code (NPM-based modules only)
  • Lint RAML (Maven-based only)
  • Build module from source (either via 'yarn' or 'maven')
  • Run unit tests
  • Run Sonarqube scan and publish to Sonarcloud
  • Build Docker image (if applicable)
  • Test Docker image (optional)
  • Publish Docker image (master branch only)
  • Publish Maven jar files (master branch only)
  • Generate module descriptor (if applicable)
  • Publish module descriptor to folio-registry ( master branch only)
  • Publish API documentation (master branch only)

Docker images are published to Docker Hub - 'folioci' (snapshots) and 'folioorg' (releases)
NPM and Maven artifacts are published to a Nexus repository (https://repository.folio.org)
Module Descriptors are published to FOLIO module registry (http://folio-registry.aws.indexdata.com)

There are also test results and code coverage report artifacts that are published in each pipeline.

Additionally, we should demonstrate that these Github Actions workflows can be maintained centrally and be used by individual repos. https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/sharing-workflows-with-your-organization

The best way to test Github Actions workflows would be to run them in parallel with Jenkins CI pipelines for a particular Nodejs and Maven-based repo. Start by working on a non-master branch. Care must be taken to not publish duplicate artifacts when running on master branch.

Github Actions documentation: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions
Starter Actions workflows: https://github.com/actions/starter-workflows/
Nodejs-based workflows: https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-nodejs
Maven-based workflows: https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-java-with-maven

Comment by Ankita Sen [ 05/Jan/21 ]

Added a github actions node.js CI to repository Stripes, branch FOLIO-2913 Closed . The buildNPM.yml file contents are as follows:

name: Nodejs CI

on:
push:
branches: [ FOLIO-2913 Closed ]
pull_request:
branches: [ FOLIO-2913 Closed ]

jobs:
build:

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v1
- run: npm ci
- run: yarn lint
- run: yarn test

TODO[ 07.01.2021 ]:

  • File yet to be tested for errors, vary basic pipleine, needs further development.
  • Adding SonarQube and publishModDescriptor stages.
Comment by Ankita Sen [ 11/Jan/21 ]

The test CI job works and runs sucessfully. The yarn lint job however throws an error, which after research I couldn't find a viable solution to. The error reads as follows:

/bin/sh: 1: eslint: not found
error Command failed with exit code 127.

The current CI yaml config file contents are as follows:
name: buildNPM stripes CI

on:
push:
branches: FOLIO-2913 Closed

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: folio-org/checkout@v2
with:
ref: FOLIO-2913 Closed
- run: echo "Running CI..."
- run: mkdir -p ci
- run: echo "<html><body><pre>" ci/lint.html
- run: yarn lint 2>&1 >> ci/lint.html
- run: echo "</pre><body></html>" >> ci/lint.html
- run: rm -rf ci
test:
runs-on: ubuntu-latest
steps:
- uses: folio-org/checkout@v2
with:
ref: FOLIO-2913 Closed
- run: xvfb-run --server-args="-screen 0 1024x768x24" yarn test

Comment by Ankita Sen [ 12/Jan/21 ]

NPM pipeline update:

  • eslint error solved.
  • new error while setting folio registry : error Couldn't find any versions for "@folio/stripes-components" that matches "~9.0.0" (Need Help with this!!!)
  • sonarqube and publishmoddescriptor not implemented yet

Maven pipeline update:

  • added pipeline buildMaven.yml
  • currently contain lint job + deploy job
  • both are failing. Investigating about the errors.
Comment by John Malconian [ 12/Jan/21 ]

Hi Ankita Sen - Some feedback:

1) FOLIO_REGISTRY (folio-registry.aws.indexdata.com) hosts FOLIO module descriptors. It is not an NPM repository which is what you are really looking for in this step (you will need this, however, when you implement publishing the module descriptor step).

2) The FOLIO project uses Nexus for hosting FOLIO NPM packages and Maven artifacts. There are multiple repositories hosted on this Nexus server. You can view them at https://repository.folio.org. Non-master/main branch builds do not publish artifacts, but you do need to configure these repositories in the build pipeline in order to satisfy dependencies. For Nodejs and Maven-based FOLIO artifacts there are two types of repositories for each - snapshot artifacts and release artifacts. Snapshot artifacts are generated in CI on the master branch only. Release artifacts are generated when a module is officially tagged in Git and released. Branch builds require dependencies from the snapshot repositories.

The FOLIO snapshot repositories are at:
NPM: https://repository.folio.org/repository/npm-folioci/
Maven: https://repository.folio.org/repository/maven-snapshots/

The FOLIO release repositories are at:
NPM: https://repository.folio.org/repository/npm-folio/
Maven: https://repository.folio.org/repository/maven-releases/

For now, you'll want to set NPM_SNAPSHOT_REGISTRY=https://repository.folio.org/repository/npm-folioci/. Later, when we experiment with publishing artifacts to these repositories, we'll need credentials.

3) I'm not sure that the 'stripes' and 'raml-module-builder' repositories are representative of most other module repositories, and, therefore, may not be ideal candidates for this POC. The 'stripes' repository is a really a meta package and 'raml-module-builder' is a runtime library - not a module. There are no module descriptors to post for either git repository and the latter repo does not create Docker artifacts.

Comment by Ankita Sen [ 21/Jan/21 ]

NPM pipeline update:

  • Setup, lint, test, publishModDescriptor(only for master branch) and SonarQube checks done.
  • Pipeline runs without errors.
  • Issue: in the SonarQube action, env variables are not read so currently the projectName is hardcoded to the current repo(ui-orders). Workaround in progress

Maven pipeline update:

  • Setup JDK 11 done
  • RAML linter in progress
Comment by John Malconian [ 26/Jan/21 ]

Hi Ankita Sen - I think we can combine some of the 'jobs' so that we are not replicating steps like checkout and 'yarn install', for example. I propose renaming the 'setup' job to something like 'Build'. The 'Build' job can then be used to execute the build, linting, and unit tests. Sonarqube doesn't require a build, so that can be a standalone job. Publishing the module descriptor is a bit more complicated. The UI repos can generate a module descriptor without requiring 'yarn install', but the Maven-based projects require a build to generate the module-descriptor. Furthermore, even though a build is not necessary to generate a UI module descriptor, you would still need to set the snapshot version of the artifact. And to complicate things a little more, a module descriptor should never be published before a docker image or npm package.

Comment by John Malconian [ 26/Jan/21 ]

There must be some way (hopefully) of temporarily saving artifacts in between 'jobs' in Github Actions. So, for example, creating an artifact in one 'job' and then using that artifact in a subsequent 'job'. This might help in some cases. https://docs.github.com/en/actions/guides/storing-workflow-data-as-artifacts

Comment by John Malconian [ 09/Feb/21 ]

Next POC demo, Tuesday, March 2, 2021 - DevOps meeting.

Comment by David Crossley [ 03/Mar/21 ]

A useful tool for developing workflows is https://github.com/kislyuk/yq
Often i have been confused by simple YAML syntax errors.
Great when i remember to use it.

Comment by John Malconian [ 04/Mar/21 ]

I started going through the buildNPM Github Actions workflow one line at a time to verify that things are working as expected. I have a branch of ui-users where I'm testing and refining some parts of the workflow - FOLIO-2913-malc.

Some notes:

1. Updates to translations in UI repos do not require a full workflow check. I'm experimenting with implementing this feature by specifying something like:

on: 
  push:
    paths-ignore:
      - 'translations/**'
  pull_request:
     paths-ignore:
      - 'translations/**'

2. I'm defining some vars at job.<jobid>.env so they are easier to find.

3. I've added the setup-node@v2 action so that we can control the version(s) of nodejs that is used to build and test.

4. For "snapshot" builds, the 'folioci_npmver.sh' was not setting the correct version. A job number needs to be appended to the version. I've done that like this:

      - name: Set FOLIO NPM snapshot version
        run: |
          git clone https://github.com/folio-org/folio-tools.git
          npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
          rm -rf folio-tools
        env:
          JOB_ID: ${{ github.run_number }}

If possible I'd like to explore the possibility of just including that whole shell script in the workflow to save a 'git clone' step.

5. I don't think we need to publish a Lint output artifact. Devs can just look at the build log for Lint output.

      - name: Yarn lint
        run: yarn lint
        continue-on-error: true

6. I think we need to publish junit test results for the 'yarn test' step whether it passes or fails. There seems to be an action that can do this here: https://github.com/marketplace/actions/publish-unit-test-results

7. publish the yarn.lock only if build fails.

8. The Sonarqube job requires the code coverage report generated by 'yarn test'. Therefore, the sonarqube job 'needs' the build job and the coverage report generated from 'yarn test' - specifically the Jest coverage report. I guess we can do an artifact upload/download to accomplish this.

9. We should publish code coverage reports for both BigTest and Jest which are in html format. I'm not sure what's the best way to accomplish this. I'm not sure if there is an "html publisher github action" available similar to Jenkins HTML Publisher plugin.

Also some tasks I need to do so that we can test master branch builds and publishing artifacts:

  • Set up test Nexus snapshot repository
  • Set up test Okapi registry

I'd like to focus now on the workflows for UI/Stripes repositories this point. When we roll this out, we will role it out to these repos first. One thing we need to be careful of is snapshot versioning. The GA workflows will probably have lower versions that existing artifacts generated by Jenkins. Therefore, we will need to address this, maybe by padding the version genersated by foliociver.sh with some extra '0's.

Comment by John Malconian [ 04/Mar/21 ]

I have a working 'build' job workflow here with some of the junit and artifacts.

https://github.com/folio-org/ui-orders/blob/FOLIO-2913-malc/.github/workflows/buildNPM.yml

Comment by David Crossley [ 09/Mar/21 ]

Sometimes one workflow will require the successful outcome of another workflow.

For example, a back-end module will do the "api-lint" task on all branches. The "api-doc" task will only happen on merge to mainline, and on release. Also the "api-doc" task would not run if there are errors reported by "api-lint" (which would distort the generated API documentation).

There is a demonstration in "mod-notes" of workflow_run (and a supporting community answer):

https://github.com/folio-org/mod-notes/tree/folio-2913-depend-workflow/.github/workflows

Comment by John Malconian [ 09/Mar/21 ]

Oh that's cool, David Crossley. Can you think of a scenario or use case in our CI environment where it makes more sense to use workflow_run rather than just combining jobs into a single workflow?

Generated at Thu Feb 08 23:24:11 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.