[FOLIO-3067] Replace Jenkins UI and Stripes CI pipelines with Github Action pipelines Created: 09/Mar/21  Updated: 13/Feb/23  Resolved: 13/Feb/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: TBD
Reporter: John Malconian Assignee: Ankita Sen
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by FOLIO-3070 Github workflow for Stripes and UI re... Closed
is blocked by FOLIO-3069 Provide environments to test GitHub a... Closed
is blocked by FOLIO-3071 Finalize GA branch build workflow for... Closed
Relates
relates to FOLIO-2913 SPIKE: Evaluate Github Actions Closed
relates to FOLIO-3192 allow GitHub actions to upload artifa... Closed
relates to FOLIO-3205 rollout GitHub workflow to the stripe... Closed
relates to FOLIO-3272 rollout GitHub workflow to the ui- UI... Closed
relates to FOLIO-3184 execute formatjs-compile in CI for tr... Closed
relates to FOLIO-3452 Roll out Github Actions to remaining ... Closed
relates to FOLIO-3504 GA pipeline failing in some repos but... Closed
Sprint:
Development Team: FOLIO DevOps

 Description   

Replace existing Jenkins CI pipelines with comparable Github Actions workflows for all Stripes and UI module repositories in folio-org. This is an umbrella issue. Specific tasks should be linked to this issue.



 Comments   
Comment by John Malconian [ 09/Mar/21 ]

Following the work completed by Ankita Sen in the spike FOLIO-2913 Closed , we can now proceed toward a goal of replacing existing folio-org UI and Stripes modules Jenkins CI pipelines with Github Actions workflows. The following prerequisites need to be met before rollout can commence.

  • Github Actions workflows for branch builds, PRs, master branch builds, and tagged release builds need to be developed, tested, and reviewed. Workflow parameters that can be customized on a per repository basis should be adequately documented.
  • Testing should be performed on selected Stripes and UI repositories. I propose testing and initial implementation and refinement on the following folio-org repositories - stripes-core, ui-orders, ui-users, and ui-plugin-find-organization followed by incremental rollouts to remaining Stripes and UI repositories.
  • Workflows should be deployed as workflow templates that are maintained for the folio-org Github organization.
Comment by Ankita Sen [ 25/May/21 ]

The build Job fails when the FOLIO_NPM_REGISTRY env variable is set to 

https://repository.folio.org/repository/npm-folioci-test/

The logs show the following error:

npm ERR! code E401
npm ERR! Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager"

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2021-05-21T15_12_30_188Z-debug.log

On doing further research I found the error related to setting the REALM configuration of the repository. I do not have rights to do that, so would need John Malconian's help. Last week the error was thought to be the outcome of the Nexus server being down and Ian Hardy had created a ticket for that, but even after the issue was resolved the same error was faced.

Comment by John Malconian [ 25/May/21 ]

Ankita Sen Can you link to the latest buildNPM workflow you are working with (the one that fails in the comment above).

Comment by Ankita Sen [ 26/May/21 ]

John Malconian it is updated in the .github workflow repository. Below is the link

https://github.com/folio-org/.github/blob/master/workflow-templates/buildNPM.yml

Also for the non-master steps, whenever the FOLIO_NPM_REGISTRY is set to the folioci-test registry, the yarn install step fails with the below error:

 
 

error Couldn't find package "@***/eslint-config-stripes" on the "npm" registry.    info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command. 

 
 

Comment by John Malconian [ 26/May/21 ]

Ok. I see now.

So we need FOLIO_NPM_REGISTRY to be set to its original value (https://repository.folio.org/repository/npm-folioci), because that's where the module gets its dependencies from. However, we need to define a separate registry (for testing purposes only) when we publish the new NPM.

Also, I see you have the publish step early in the workflow.

- run: |
          git clone https://github.com/folio-org/folio-tools.git
          npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
          rm -rf folio-tools
          npm publish
        env:
          JOB_ID: ${{ github.run_number }}
          NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}

Instead, 'npm publish' should be performed as a separate step right before publishing the module descriptor, because we do not want to publish a package that fails unit testing or any other quality gates. Something like this:

1. Remove this step.

    - name: Set FOLIO NPM snapshot version and publish to NPM snapshot repository
        if: contains(github.REF,'master')
        uses: actions/setup-node@v2
        with:
          node-version: ${{ env.NODEJS_VERSION }}
          registry-url: https://repository.folio.org/repository/npm-folioci-test/
      - run: |
          git clone https://github.com/folio-org/folio-tools.git
          npm --no-git-tag-version version `folio-tools/github-actions-scripts/folioci_npmver.sh`
          rm -rf folio-tools
          npm publish
        env:
          JOB_ID: ${{ github.run_number }}
          NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}

2. Add the following the step right before the publish module descriptor step.

        if: contains(github.REF,'master')
        uses: actions/setup-node@v2
        with:
          node-version: ${{ env.NODEJS_VERSION }}
          registry-url: https://repository.folio.org/repository/npm-folioci-test/
      - run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.FOLIO_NEXUS_NPM_TOKEN }}

We may need to do some cleanup before the publish step, but let's worry about that later. Let me know if this works. If you still have an issue authenticating to https://repository.folio.org/repository/npm-folioci-test/ with FOLIO_NEXUS_NPM_TOKEN, let me know. It's quite possible I have the wrong token defined.

Comment by John Malconian [ 26/May/21 ]

One other thing to note: We will be changing the default github repo branch from 'master' to 'main' in the near future, so conditionals for 'master' should also include 'main'. If there's something we can use that covers both cases that would be ideal - I thought I saw somewhere that you can use a GA built-in variable like $default_branch (or something like that) that just knows what the default branch is for a repo.

Comment by John Malconian [ 26/May/21 ]

More feedback:

This step should be run on any branch - not just the default or master branch.

       - name: Generate Module descriptor
        if: contains(github.REF,'master') 
        run: stripes mod descriptor --full --strict | jq '.[]' > ${{ env.MODULE_DESCRIPTOR_DIR }}/${{ env.FOLIO_NAME }}.json

But more importantly, not all Node-based repos include a module descriptor. Therefore, we should add an environment variable called PUBLISH_MODULE_DESCRIPTOR that is either set to (true or false). And the modify 'Generate Module descriptor' step and 'Publish module descriptor' step to only run if PUBLISH_MODULE_DESCRIPTOR is set to 'true'.

Comment by Ankita Sen [ 01/Jun/21 ]

John Malconian I have for the time being included both 'master' and 'main' branches in the conditional steps. I still however get the following error when trying to do an `npm publish` step:

Error: Error: Unable to determine the current version of @***/stripes-core on NPM. 
npm view @***/stripes-core version exited with a status of 1.
npm ERR! code E401 
npm ERR! Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager"

 

Comment by John Malconian [ 04/Jun/21 ]

Ankita Sen Currently doing testing and debugging on the 'malc-test' branch of stripes-core. I still haven't been able to resolve the 'npm publish' authentication issue. This is tricky. I believe that there is some additional complexity here due to, I think, the repository being a private Nexus repo. Will continue to investigate.

Comment by John Malconian [ 16/Jun/21 ]

Adding comments discussed during last DevOps meeting:

1. Currently the builNPM workflow includes the following:

on: 
  push:
    paths-ignore:
      - 'translations/**'
  pull_request:
     paths-ignore:
      - 'translations/**'

which is meant to bypass triggering the workflow when only translations updates have been made. However, there is a new requirement described in FOLIO-3184 Closed that includes adding a CI step to compile translation files. Perhaps the easiest solution to address this requirement is to add a configurable environment variable that enables or bypasses this step ('yarn formatjs-compile') during every workflow run since some Node-based projects don't include translations files. Also, 'paths-ignore: - 'translations/**'' would need to be removed from the workflow.

Comment by John Malconian [ 16/Jun/21 ]

2. Another issue we need to address is ensuring artifact snapshot versions are sufficiently higher than ones generated from Jenkins. Snapshot versions are based on build numbers. If we make no changes, GA workflows snapshot artifacts will be generated with lower build numbers than the artifacts generated from Jenkins. This is a problem. A potential easy solution is to pad the snapshot artifact versions with two extra zeros. This change can be made in this script: https://github.com/folio-org/folio-tools/github-actions-scripts/folioci_npmver.sh

Comment by Zak Burke [ 08/Jul/21 ]

John Malconian, Ankita Sen: I was doing some work in stripes-core, the repo where the pilot work for this ticket is happening, and wanted to clean up the PR comments provided by the CI workflow since all PRs currently show failed checks even when they are safe to merge.

I have a PR up that cleans up a few things WRT how stripes is invoked to generate the MDs but I’m hesitant to merge additional changes to it because I’m driving waaaaay outside my lane here. I have a few questions/concerns:

  1. Do we in fact want to publish MDs when building PRs, i.e. before they have merged? It feels like that should be part of Automation/build-platform-complete-snapshot instead.
  2. The POST destination those MDs is a hard-coded AWS URL. That feels like a maintenance headache. Is there some other way we can pass that value into this script?

As noted on my comment in the PR, I think there's one final change to buildnpm.yml that'll allow this workflow to succeed. Should I go ahead and commit that change?

Nevermind; I see Ankita Sen has a follow-up PR that runs cleanly. Sorry for the noise!

Generated at Thu Feb 08 23:25:21 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.