[FOLIO-3067] Replace Jenkins UI and Stripes CI pipelines with Github Action pipelines Created: 09/Mar/21 Updated: 13/Feb/23 Resolved: 13/Feb/23 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Umbrella | Priority: | TBD |
| Reporter: | John Malconian | Assignee: | Ankita Sen |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Replace existing Jenkins CI pipelines with comparable Github Actions workflows for all Stripes and UI module repositories in folio-org. This is an umbrella issue. Specific tasks should be linked to this issue. |
| Comments |
| Comment by John Malconian [ 09/Mar/21 ] |
|
Following the work completed by Ankita Sen in the spike
|
| Comment by Ankita Sen [ 25/May/21 ] |
|
The build Job fails when the FOLIO_NPM_REGISTRY env variable is set to
https://repository.folio.org/repository/npm-folioci-test/
The logs show the following error: npm ERR! code E401 npm ERR! Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager" npm ERR! A complete log of this run can be found in: npm ERR! /home/runner/.npm/_logs/2021-05-21T15_12_30_188Z-debug.log On doing further research I found the error related to setting the REALM configuration of the repository. I do not have rights to do that, so would need John Malconian's help. Last week the error was thought to be the outcome of the Nexus server being down and Ian Hardy had created a ticket for that, but even after the issue was resolved the same error was faced. |
| Comment by John Malconian [ 25/May/21 ] |
|
Ankita Sen Can you link to the latest buildNPM workflow you are working with (the one that fails in the comment above). |
| Comment by Ankita Sen [ 26/May/21 ] |
|
John Malconian it is updated in the .github workflow repository. Below is the link https://github.com/folio-org/.github/blob/master/workflow-templates/buildNPM.yml Also for the non-master steps, whenever the FOLIO_NPM_REGISTRY is set to the folioci-test registry, the yarn install step fails with the below error: error Couldn't find package "@***/eslint-config-stripes" on the "npm" registry. info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command. |
| Comment by John Malconian [ 26/May/21 ] |
|
Ok. I see now. So we need FOLIO_NPM_REGISTRY to be set to its original value (https://repository.folio.org/repository/npm-folioci), because that's where the module gets its dependencies from. However, we need to define a separate registry (for testing purposes only) when we publish the new NPM. Also, I see you have the publish step early in the workflow. - run: |
git clone https://github.com/folio-org/folio-tools.git
npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
rm -rf folio-tools
npm publish
env:
JOB_ID: ${{ github.run_number }}
NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}
Instead, 'npm publish' should be performed as a separate step right before publishing the module descriptor, because we do not want to publish a package that fails unit testing or any other quality gates. Something like this: 1. Remove this step. - name: Set FOLIO NPM snapshot version and publish to NPM snapshot repository
if: contains(github.REF,'master')
uses: actions/setup-node@v2
with:
node-version: ${{ env.NODEJS_VERSION }}
registry-url: https://repository.folio.org/repository/npm-folioci-test/
- run: |
git clone https://github.com/folio-org/folio-tools.git
npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
rm -rf folio-tools
npm publish
env:
JOB_ID: ${{ github.run_number }}
NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}
2. Add the following the step right before the publish module descriptor step. if: contains(github.REF,'master')
uses: actions/setup-node@v2
with:
node-version: ${{ env.NODEJS_VERSION }}
registry-url: https://repository.folio.org/repository/npm-folioci-test/
- run: npm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}
We may need to do some cleanup before the publish step, but let's worry about that later. Let me know if this works. If you still have an issue authenticating to https://repository.folio.org/repository/npm-folioci-test/ with FOLIO_NEXUS_NPM_TOKEN, let me know. It's quite possible I have the wrong token defined. |
| Comment by John Malconian [ 26/May/21 ] |
|
One other thing to note: We will be changing the default github repo branch from 'master' to 'main' in the near future, so conditionals for 'master' should also include 'main'. If there's something we can use that covers both cases that would be ideal - I thought I saw somewhere that you can use a GA built-in variable like $default_branch (or something like that) that just knows what the default branch is for a repo. |
| Comment by John Malconian [ 26/May/21 ] |
|
More feedback: This step should be run on any branch - not just the default or master branch. - name: Generate Module descriptor
if: contains(github.REF,'master')
run: stripes mod descriptor --full --strict | jq '.[]' > ${{ env.MODULE_DESCRIPTOR_DIR }}/${{ env.FOLIO_NAME }}.json
But more importantly, not all Node-based repos include a module descriptor. Therefore, we should add an environment variable called PUBLISH_MODULE_DESCRIPTOR that is either set to (true or false). And the modify 'Generate Module descriptor' step and 'Publish module descriptor' step to only run if PUBLISH_MODULE_DESCRIPTOR is set to 'true'. |
| Comment by Ankita Sen [ 01/Jun/21 ] |
|
John Malconian I have for the time being included both 'master' and 'main' branches in the conditional steps. I still however get the following error when trying to do an `npm publish` step:
Error: Error: Unable to determine the current version of @***/stripes-core on NPM.
npm view @***/stripes-core version exited with a status of 1.
npm ERR! code E401
npm ERR! Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager"
|
| Comment by John Malconian [ 04/Jun/21 ] |
|
Ankita Sen Currently doing testing and debugging on the 'malc-test' branch of stripes-core. I still haven't been able to resolve the 'npm publish' authentication issue. This is tricky. I believe that there is some additional complexity here due to, I think, the repository being a private Nexus repo. Will continue to investigate. |
| Comment by John Malconian [ 16/Jun/21 ] |
|
Adding comments discussed during last DevOps meeting: 1. Currently the builNPM workflow includes the following: on:
push:
paths-ignore:
- 'translations/**'
pull_request:
paths-ignore:
- 'translations/**'
which is meant to bypass triggering the workflow when only translations updates have been made. However, there is a new requirement described in
|
| Comment by John Malconian [ 16/Jun/21 ] |
|
2. Another issue we need to address is ensuring artifact snapshot versions are sufficiently higher than ones generated from Jenkins. Snapshot versions are based on build numbers. If we make no changes, GA workflows snapshot artifacts will be generated with lower build numbers than the artifacts generated from Jenkins. This is a problem. A potential easy solution is to pad the snapshot artifact versions with two extra zeros. This change can be made in this script: https://github.com/folio-org/folio-tools/github-actions-scripts/folioci_npmver.sh |
| Comment by Zak Burke [ 08/Jul/21 ] |
|
Nevermind; I see Ankita Sen has a follow-up PR that runs cleanly. Sorry for the noise! |