[FOLIO-2913] SPIKE: Evaluate Github Actions Created: 11/Dec/20 Updated: 11/Mar/21 Resolved: 11/Mar/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P3 |
| Reporter: | John Malconian | Assignee: | Ankita Sen |
| Resolution: | Done | Votes: | 0 |
| Labels: | devops-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||
| Sprint: | DevOps: Sprint 105, DevOps Sprint 109, DevOps Sprint 107, DevOps Sprint 108, DevOps: Sprint 104, DevOps Sprint 106 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| Description |
|
Evaluate feasibility of using GitHub Actions as a CI pipeline for individual front-end and backend module repos. Deliverables
|
| Comments |
| Comment by John Malconian [ 15/Dec/20 ] |
|
There are primarily two Jenkins pipelines that are used for about 90% of all folio-org modules repos. For Stripes and ui-* Nodejs-based modules: https://github.com/folio-org/jenkins-pipeline-libs/blob/master/vars/buildNPM.groovy The idea for this spike is to develop a POC GitHub Actions workflow for each based on the corresponding Jenkins shared pipelines above. The high-level workflow for each pipeline essentially consists of the following steps (non-release builds)
Docker images are published to Docker Hub - 'folioci' (snapshots) and 'folioorg' (releases) There are also test results and code coverage report artifacts that are published in each pipeline. Additionally, we should demonstrate that these Github Actions workflows can be maintained centrally and be used by individual repos. https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/sharing-workflows-with-your-organization The best way to test Github Actions workflows would be to run them in parallel with Jenkins CI pipelines for a particular Nodejs and Maven-based repo. Start by working on a non-master branch. Care must be taken to not publish duplicate artifacts when running on master branch. Github Actions documentation: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions |
| Comment by Ankita Sen [ 05/Jan/21 ] |
|
Added a github actions node.js CI to repository Stripes, branch
name: Nodejs CI TODO[ 07.01.2021 ]:
|
| Comment by Ankita Sen [ 11/Jan/21 ] |
|
The test CI job works and runs sucessfully. The yarn lint job however throws an error, which after research I couldn't find a viable solution to. The error reads as follows: /bin/sh: 1: eslint: not found The current CI yaml config file contents are as follows: |
| Comment by Ankita Sen [ 12/Jan/21 ] |
|
NPM pipeline update:
Maven pipeline update:
|
| Comment by John Malconian [ 12/Jan/21 ] |
|
Hi Ankita Sen - Some feedback: 1) FOLIO_REGISTRY (folio-registry.aws.indexdata.com) hosts FOLIO module descriptors. It is not an NPM repository which is what you are really looking for in this step (you will need this, however, when you implement publishing the module descriptor step). 2) The FOLIO project uses Nexus for hosting FOLIO NPM packages and Maven artifacts. There are multiple repositories hosted on this Nexus server. You can view them at https://repository.folio.org. Non-master/main branch builds do not publish artifacts, but you do need to configure these repositories in the build pipeline in order to satisfy dependencies. For Nodejs and Maven-based FOLIO artifacts there are two types of repositories for each - snapshot artifacts and release artifacts. Snapshot artifacts are generated in CI on the master branch only. Release artifacts are generated when a module is officially tagged in Git and released. Branch builds require dependencies from the snapshot repositories. The FOLIO snapshot repositories are at: The FOLIO release repositories are at: For now, you'll want to set NPM_SNAPSHOT_REGISTRY=https://repository.folio.org/repository/npm-folioci/. Later, when we experiment with publishing artifacts to these repositories, we'll need credentials. 3) I'm not sure that the 'stripes' and 'raml-module-builder' repositories are representative of most other module repositories, and, therefore, may not be ideal candidates for this POC. The 'stripes' repository is a really a meta package and 'raml-module-builder' is a runtime library - not a module. There are no module descriptors to post for either git repository and the latter repo does not create Docker artifacts. |
| Comment by Ankita Sen [ 21/Jan/21 ] |
|
NPM pipeline update:
Maven pipeline update:
|
| Comment by John Malconian [ 26/Jan/21 ] |
|
Hi Ankita Sen - I think we can combine some of the 'jobs' so that we are not replicating steps like checkout and 'yarn install', for example. I propose renaming the 'setup' job to something like 'Build'. The 'Build' job can then be used to execute the build, linting, and unit tests. Sonarqube doesn't require a build, so that can be a standalone job. Publishing the module descriptor is a bit more complicated. The UI repos can generate a module descriptor without requiring 'yarn install', but the Maven-based projects require a build to generate the module-descriptor. Furthermore, even though a build is not necessary to generate a UI module descriptor, you would still need to set the snapshot version of the artifact. And to complicate things a little more, a module descriptor should never be published before a docker image or npm package. |
| Comment by John Malconian [ 26/Jan/21 ] |
|
There must be some way (hopefully) of temporarily saving artifacts in between 'jobs' in Github Actions. So, for example, creating an artifact in one 'job' and then using that artifact in a subsequent 'job'. This might help in some cases. https://docs.github.com/en/actions/guides/storing-workflow-data-as-artifacts |
| Comment by John Malconian [ 09/Feb/21 ] |
|
Next POC demo, Tuesday, March 2, 2021 - DevOps meeting. |
| Comment by David Crossley [ 03/Mar/21 ] |
|
A useful tool for developing workflows is https://github.com/kislyuk/yq |
| Comment by John Malconian [ 04/Mar/21 ] |
|
I started going through the buildNPM Github Actions workflow one line at a time to verify that things are working as expected. I have a branch of ui-users where I'm testing and refining some parts of the workflow - FOLIO-2913-malc. Some notes: 1. Updates to translations in UI repos do not require a full workflow check. I'm experimenting with implementing this feature by specifying something like: on:
push:
paths-ignore:
- 'translations/**'
pull_request:
paths-ignore:
- 'translations/**'
2. I'm defining some vars at job.<jobid>.env so they are easier to find. 3. I've added the setup-node@v2 action so that we can control the version(s) of nodejs that is used to build and test. 4. For "snapshot" builds, the 'folioci_npmver.sh' was not setting the correct version. A job number needs to be appended to the version. I've done that like this: - name: Set FOLIO NPM snapshot version
run: |
git clone https://github.com/folio-org/folio-tools.git
npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
rm -rf folio-tools
env:
JOB_ID: ${{ github.run_number }}
If possible I'd like to explore the possibility of just including that whole shell script in the workflow to save a 'git clone' step. 5. I don't think we need to publish a Lint output artifact. Devs can just look at the build log for Lint output. - name: Yarn lint
run: yarn lint
continue-on-error: true
6. I think we need to publish junit test results for the 'yarn test' step whether it passes or fails. There seems to be an action that can do this here: https://github.com/marketplace/actions/publish-unit-test-results 7. publish the yarn.lock only if build fails. 8. The Sonarqube job requires the code coverage report generated by 'yarn test'. Therefore, the sonarqube job 'needs' the build job and the coverage report generated from 'yarn test' - specifically the Jest coverage report. I guess we can do an artifact upload/download to accomplish this. 9. We should publish code coverage reports for both BigTest and Jest which are in html format. I'm not sure what's the best way to accomplish this. I'm not sure if there is an "html publisher github action" available similar to Jenkins HTML Publisher plugin. Also some tasks I need to do so that we can test master branch builds and publishing artifacts:
I'd like to focus now on the workflows for UI/Stripes repositories this point. When we roll this out, we will role it out to these repos first. One thing we need to be careful of is snapshot versioning. The GA workflows will probably have lower versions that existing artifacts generated by Jenkins. Therefore, we will need to address this, maybe by padding the version genersated by foliociver.sh with some extra '0's. |
| Comment by John Malconian [ 04/Mar/21 ] |
|
I have a working 'build' job workflow here with some of the junit and artifacts. https://github.com/folio-org/ui-orders/blob/FOLIO-2913-malc/.github/workflows/buildNPM.yml |
| Comment by David Crossley [ 09/Mar/21 ] |
|
Sometimes one workflow will require the successful outcome of another workflow. For example, a back-end module will do the "api-lint" task on all branches. The "api-doc" task will only happen on merge to mainline, and on release. Also the "api-doc" task would not run if there are errors reported by "api-lint" (which would distort the generated API documentation). There is a demonstration in "mod-notes" of workflow_run (and a supporting community answer): https://github.com/folio-org/mod-notes/tree/folio-2913-depend-workflow/.github/workflows |
| Comment by John Malconian [ 09/Mar/21 ] |
|
Oh that's cool, David Crossley. Can you think of a scenario or use case in our CI environment where it makes more sense to use workflow_run rather than just combining jobs into a single workflow? |