[FOLIO-3070] Github workflow for Stripes and UI releases Created: 09/Mar/21  Updated: 26/Aug/21  Resolved: 10/Aug/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: John Malconian Assignee: Ankita Sen
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-3067 Replace Jenkins UI and Stripes CI pip... Closed
is blocked by FOLIO-3192 allow GitHub actions to upload artifa... Closed
Relates
relates to FOLIO-3263 update release documentation on dev.f... Closed
relates to FOLIO-3270 Handle default branch name in NPM Wor... Closed
Sprint: DevOps Sprint 120, DevOps Sprint 116, DevOps Sprint 114, DevOps Sprint 115, DevOps Sprint 110, DevOps Sprint 117, DevOps Sprint 111, DevOps Sprint 118, DevOps Sprint 112, DevOps Sprint 119, DevOps Sprint 113
Development Team: FOLIO DevOps

 Description   

Develop and test Github workflow for tagged releases of Stripes and UI modules. This workflow is slightly different from other branch workflows in that NPM artifacts are deployed to distinct release repositories. The conditions on which release workflows are triggered are different as well. Release workflows should be automatically triggered when a specifically formatted Github tag is created. e.g vX.X.X.



 Comments   
Comment by John Malconian [ 07/Jun/21 ]

Ankita Sen Issue with Nexus NPM repo has been resolved. See FOLIO-3192 Closed for more details. That's the good news. The bad news is the Publish module descriptor step is broken. It returns an error similar to:

Error: {"code":400,"message":"Failed to decode:Unexpected character ('-' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: (String)\"----------------------------806167094178927838742068\r\nContent-Disposition: form-data; name=\"file\"; filename=\"module-descriptor.json\"\r\nContent-Type: application/json\r\n\r\n{\n  \"id\": \"***_stripes-core-7.2.100093\",\n  \"name\": \"The starting point for Stripes applications\",\n  \"permissionSets\": [\n    {\n      \"permissionName\": \"settings.enabled\",\n      \"displayName\": \"UI: settings area is enabled\"\n    }\n  ],\n  \"requires\": [\n    {\n      \"id\": \"users-bl\",\n      \"version\": \"5.0 6.0\"\n    },\n    {\n      \"id\":\"[truncated 258 chars]; line: 1, column: 3]"}

I've verified that there is nothing wrong with the module descriptor itself. I am able to publish to the same folio registry manually. I think that there may be something up with the choice of GA module we are using to perform this step. Perhaps encoding related. Maybe we need to use a different Action. If all else fails we might be able to just use 'curl'. In fact that may be a good place to start. I'll leave this for you to take a crack at. You can see my latest progress on the 'malc-test' branch of stripes-core.

Let's use this issue for tracking progress on this rather than the umbrella issue.

Comment by John Malconian [ 08/Jun/21 ]

I think I've fixed the publish module descriptor step. Instead of using the 'files' parameter of fjogeleit/http-request we now use 'data' with a value that is set from a new read-file step:

      - name: Read module descriptor
        if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
        id: moduleDescriptor
        uses: juliangruber/read-file-action@v1
        with:
          path: ./module-descriptor.json

      - name: Publish module descriptor
        if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
        id: modDescriptorPost
        uses: fjogeleit/http-request-action@master
        with:
          url: 'http://ec2-3-88-110-24.compute-1.amazonaws.com/_/proxy/modules'
          method: 'POST'
          contentType: 'application/json; charset=utf-8'
          customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
          data: ${{ steps.moduleDescriptor.outputs.content }}
          username: ${{ secrets.FOLIO_REGISTRY_USERNAME }}
          password: ${{ secrets.FOLIO_REGISTRY_PASSWORD }}

      - name: Show FOLIO registry response
        if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
        run: echo "${{ steps.modDescriptorPost.outputs.response }}"
Comment by John Malconian [ 08/Jun/21 ]

I have one quibble about the conditionals checking for branch names:

if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master')...

If I'm not mistaken, 'contains' will match a substring. If I'm correct about that, branches with names like 'my-branch-master' will meet this condition and that would be problematic. I think we need to tighten that up. Also, does it make sense to use '$default-branch' instead of specifying 'master' or 'main'?

Comment by Ankita Sen [ 10/Jun/21 ]

John Malconian I fixed a few bugs related to the conditional running of steps. The problem with using $default-branch is that it just has the branch name whereas github.ref produced and string 'refs/heads/<branch_name>'. I have another implementation in mind which I will be testing shortly. As of now the Workflow runs without any failures.

Comment by Jakub Skoczen [ 15/Jun/21 ]

This has been enabled for testing in stripes-core, ui-users, ui-orders, ui-find-plugin-organizations. There's no rollout yet, see FOLIO-3205 Closed

Comment by John Malconian [ 16/Jun/21 ]

This shouldn't be closed. It pertains to the workflow needed to generated release artifacts based on release tags. This particular workflow is similar to the branch/master workflow with the following exceptions:

  • It should be automatically triggered when a git tag with the following regular expression format is pushed to the repo - '^v\d+\.\d+\.\d+$'. e.g. v1.2.3
Comment by John Malconian [ 16/Jun/21 ]

I've re-opened this issue.

Comment by John Malconian [ 15/Jul/21 ]

Ankita Sen I took a look at the release workflow at https://github.com/folio-org/ui-users/blob/FOLIO-3067-Ankita/.github/workflows/buildnpmRelease.yml and I think the issue that easily stands out is that the env.OKAPI_PULL registry url is incorrect.

It's currently set to:
'

{ "urls" : [ "https://repository.folio.org/repository/npm-folio-test/" ] }

' -
which is an NPM registry. You want Okapi to actually pull from a module descriptor registry. The correct URL is: 'https://folio-registry.folio-dev.folio.org'.

Let's make that change and see where we end up.

Comment by Jakub Skoczen [ 15/Jul/21 ]

https://folio-registry.dev.folio.org/

Comment by John Malconian [ 23/Jul/21 ]

Ankita Sen - I was able to fix the issue related to pulling module descriptors by adding a 'sleep 10' to the step that starts the okapi container (since we want to make sure that Okapi is fully initialized) AND by setting a higher timeout on the step that POSTs the request to pull module descriptors from the FOLIO module descriptor registry to the local Okapi container. The default timeout for a response is 5 seconds. I set it to 60 seconds. It normally doesn't take 60 seconds to pull all the module descriptors from the module descriptor registry but it takes more than 5 seconds which is why we were getting that 'no data received error'.

Also, I changed the order of the steps. The Okapi dependency check should take place after the module descriptor has been generated but BEFORE we publish artifacts to the NPM and FOLIO MD registry.

One other thing I'll leave to you: For the release workflow, the 'Set FOLIO NPM snapshot version' step should be omitted entirely since the release version is just whatever is specified in package.json. We should, however, add a step that compares the tag with the version specified in package.json to check if they match. You'd be surprised how often a mismatch occurs. The git tag should equal v${package.json version) and the build should fail if it doesn't match.

Comment by Ankita Sen [ 03/Aug/21 ]

John Malconian The release workflow is now ready. The version matching step I did a little differently. I wrote a shell script which reads and extracts the version from the package.json file and then git tags with the extracted version. Below is the script

PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[\",]//g' | tr -d '[[:space:]]') && git tag v$PACKAGE_VERSION

This step fails if there is any type of conflicts such as duplicate tag(it fails now because the tag mentioned in the package.json already exists in the repository of ui-users).

I have also pushed this workflow to the other three test repositories and the github workflow repository.

Comment by David Crossley [ 04/Aug/21 ]

I did a whitespace cleanup pull/1 on the "workflow-templates" area.

That will help to keep future modifications and copies to be free from clutter.

However, now wherever this template was copied to, will be out of synchronisation.

Comment by John Malconian [ 04/Aug/21 ]

Ankita Sen I've reviewed both the buildNPM and buildnpmRelease workflows on master in the .github repo.

1. Both workflows need some cleanup. e.g remove test branch from conditionals and substitute the NPM and module descriptor test registries with the real values (using vars).

2. In the release workflow, I don't understand this test:

PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[\",]//g' | tr -d '[[:space:]]') && git tag v$PACKAGE_VERSION

The tag should already exist and is what triggers the release workflow. All we need to do is make sure the version in the package.json matches the git tag we are building in the workflow. May be I'm not understanding something here. Also, you may want to consider getting 'version' from package.json using a JSON parser like: https://github.com/marketplace/actions/get-json-property

3. We really should try to use the OKAPI_IP var in the following steps instead of hard-coding 172.17.0.2. I don't think we can rely on this always being the same.

      - name: Pull all Module descriptors to local Okapi instance
        if: ${{ env.PUBLISH_MOD_DESCRIPTOR }}
        uses: fjogeleit/http-request-action@master
        with:
          url: http://172.17.0.2:9130/_/proxy/pull/modules
          method: 'POST'
          contentType: 'application/json; charset=utf-8'
          customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
          data: ${{ env.OKAPI_PULL }}
          timeout: 60000

      - name: Perform local Okapi dependency check
        if: ${{ env.PUBLISH_MOD_DESCRIPTOR }}
        uses: fjogeleit/http-request-action@master
        with:
          # use in production
          #url: 'http://172.17.0.2:9130/_/proxy/modules?preRelease=false&npmSnapshot=false'
          # line below is for testing only --malc
          url: 'http://172.17.0.2:9130/_/proxy/modules'
          method: 'POST'
          contentType: 'application/json; charset=utf-8'
          customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
          data: ${{ steps.moduleDescriptor.outputs.content }}

Comment by John Malconian [ 04/Aug/21 ]

Also, and most importantly, I think, the release workflow should only be triggered when a tag has been created. All the conditionals checking for 'master' or 'main' in the buildNPM workflow shouldn't be applied to the release workflow. See this example,

https://github.com/actions/runner/issues/1007#issuecomment-808904408

Comment by Jakub Skoczen [ 05/Aug/21 ]

Once this is complete we need to sync up with Zak Burke regarding rolling this out in individual repos (the goal is to use the new workflow for Kiwi releases).

Comment by John Malconian [ 05/Aug/21 ]

Ankita Sen - I've spent a good chunk of time testing the following condition today and am not able to get it to ever return true.

if: ${{ env.TAG != steps.version.outputs.prop }} 

I've tried several variations with no success including

if: ${{  env.TAG }}  != ${{ steps.version.outputs.prop }}

You can see an example here: https://github.com/folio-org/ga-test/runs/3255673370?check_suite_focus=true

Feel free to mess around with this in the ga-test repo. My work is on the 'v1.2' branch.

Comment by David Crossley [ 06/Aug/21 ]

Hooray. I too spent some workflow stuggle time. Mine finally yielded greenness.

Use the "id" of the relevant step, and use the "if: ${{ ..."

if: ${{ env.TAG_VERSION != steps.package_version.outputs.prop }}
Comment by John Malconian [ 06/Aug/21 ]

Gah!! Another stupid mistake on my part getting the id wrong. Thanks David Crossley

Comment by John Malconian [ 10/Aug/21 ]

I'm closing this issue since we are done with the development of the release workflow. Roll-out of the new workflows can be tracked in FOLIO-3067 Closed and FOLIO-3205 Closed

Generated at Thu Feb 08 23:25:23 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.