[FOLIO-3070] Github workflow for Stripes and UI releases Created: 09/Mar/21 Updated: 26/Aug/21 Resolved: 10/Aug/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | John Malconian | Assignee: | Ankita Sen |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||
| Sprint: | DevOps Sprint 120, DevOps Sprint 116, DevOps Sprint 114, DevOps Sprint 115, DevOps Sprint 110, DevOps Sprint 117, DevOps Sprint 111, DevOps Sprint 118, DevOps Sprint 112, DevOps Sprint 119, DevOps Sprint 113 | ||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||
| Description |
|
Develop and test Github workflow for tagged releases of Stripes and UI modules. This workflow is slightly different from other branch workflows in that NPM artifacts are deployed to distinct release repositories. The conditions on which release workflows are triggered are different as well. Release workflows should be automatically triggered when a specifically formatted Github tag is created. e.g vX.X.X. |
| Comments |
| Comment by John Malconian [ 07/Jun/21 ] |
|
Ankita Sen Issue with Nexus NPM repo has been resolved. See
Error: {"code":400,"message":"Failed to decode:Unexpected character ('-' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: (String)\"----------------------------806167094178927838742068\r\nContent-Disposition: form-data; name=\"file\"; filename=\"module-descriptor.json\"\r\nContent-Type: application/json\r\n\r\n{\n \"id\": \"***_stripes-core-7.2.100093\",\n \"name\": \"The starting point for Stripes applications\",\n \"permissionSets\": [\n {\n \"permissionName\": \"settings.enabled\",\n \"displayName\": \"UI: settings area is enabled\"\n }\n ],\n \"requires\": [\n {\n \"id\": \"users-bl\",\n \"version\": \"5.0 6.0\"\n },\n {\n \"id\":\"[truncated 258 chars]; line: 1, column: 3]"}
I've verified that there is nothing wrong with the module descriptor itself. I am able to publish to the same folio registry manually. I think that there may be something up with the choice of GA module we are using to perform this step. Perhaps encoding related. Maybe we need to use a different Action. If all else fails we might be able to just use 'curl'. In fact that may be a good place to start. I'll leave this for you to take a crack at. You can see my latest progress on the 'malc-test' branch of stripes-core. Let's use this issue for tracking progress on this rather than the umbrella issue. |
| Comment by John Malconian [ 08/Jun/21 ] |
|
I think I've fixed the publish module descriptor step. Instead of using the 'files' parameter of fjogeleit/http-request we now use 'data' with a value that is set from a new read-file step: - name: Read module descriptor
if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
id: moduleDescriptor
uses: juliangruber/read-file-action@v1
with:
path: ./module-descriptor.json
- name: Publish module descriptor
if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
id: modDescriptorPost
uses: fjogeleit/http-request-action@master
with:
url: 'http://ec2-3-88-110-24.compute-1.amazonaws.com/_/proxy/modules'
method: 'POST'
contentType: 'application/json; charset=utf-8'
customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
data: ${{ steps.moduleDescriptor.outputs.content }}
username: ${{ secrets.FOLIO_REGISTRY_USERNAME }}
password: ${{ secrets.FOLIO_REGISTRY_PASSWORD }}
- name: Show FOLIO registry response
if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master') || contains(github.REF,'main') || contains(github.REF,'malc-test')
run: echo "${{ steps.modDescriptorPost.outputs.response }}"
|
| Comment by John Malconian [ 08/Jun/21 ] |
|
I have one quibble about the conditionals checking for branch names: if: ${{ env.PUBLISH_MOD_DESCRIPTOR }} && contains(github.REF, 'master')... If I'm not mistaken, 'contains' will match a substring. If I'm correct about that, branches with names like 'my-branch-master' will meet this condition and that would be problematic. I think we need to tighten that up. Also, does it make sense to use '$default-branch' instead of specifying 'master' or 'main'? |
| Comment by Ankita Sen [ 10/Jun/21 ] |
|
John Malconian I fixed a few bugs related to the conditional running of steps. The problem with using $default-branch is that it just has the branch name whereas github.ref produced and string 'refs/heads/<branch_name>'. I have another implementation in mind which I will be testing shortly. As of now the Workflow runs without any failures. |
| Comment by Jakub Skoczen [ 15/Jun/21 ] |
|
This has been enabled for testing in stripes-core, ui-users, ui-orders, ui-find-plugin-organizations. There's no rollout yet, see
|
| Comment by John Malconian [ 16/Jun/21 ] |
|
This shouldn't be closed. It pertains to the workflow needed to generated release artifacts based on release tags. This particular workflow is similar to the branch/master workflow with the following exceptions:
|
| Comment by John Malconian [ 16/Jun/21 ] |
|
I've re-opened this issue. |
| Comment by John Malconian [ 15/Jul/21 ] |
|
Ankita Sen I took a look at the release workflow at https://github.com/folio-org/ui-users/blob/FOLIO-3067-Ankita/.github/workflows/buildnpmRelease.yml and I think the issue that easily stands out is that the env.OKAPI_PULL registry url is incorrect. It's currently set to: ' - Let's make that change and see where we end up. |
| Comment by Jakub Skoczen [ 15/Jul/21 ] |
| Comment by John Malconian [ 23/Jul/21 ] |
|
Ankita Sen - I was able to fix the issue related to pulling module descriptors by adding a 'sleep 10' to the step that starts the okapi container (since we want to make sure that Okapi is fully initialized) AND by setting a higher timeout on the step that POSTs the request to pull module descriptors from the FOLIO module descriptor registry to the local Okapi container. The default timeout for a response is 5 seconds. I set it to 60 seconds. It normally doesn't take 60 seconds to pull all the module descriptors from the module descriptor registry but it takes more than 5 seconds which is why we were getting that 'no data received error'. Also, I changed the order of the steps. The Okapi dependency check should take place after the module descriptor has been generated but BEFORE we publish artifacts to the NPM and FOLIO MD registry. One other thing I'll leave to you: For the release workflow, the 'Set FOLIO NPM snapshot version' step should be omitted entirely since the release version is just whatever is specified in package.json. We should, however, add a step that compares the tag with the version specified in package.json to check if they match. You'd be surprised how often a mismatch occurs. The git tag should equal v${package.json version) and the build should fail if it doesn't match. |
| Comment by Ankita Sen [ 03/Aug/21 ] |
|
John Malconian The release workflow is now ready. The version matching step I did a little differently. I wrote a shell script which reads and extracts the version from the package.json file and then git tags with the extracted version. Below is the script PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[\",]//g' | tr -d '[[:space:]]') && git tag v$PACKAGE_VERSION This step fails if there is any type of conflicts such as duplicate tag(it fails now because the tag mentioned in the package.json already exists in the repository of ui-users). I have also pushed this workflow to the other three test repositories and the github workflow repository. |
| Comment by David Crossley [ 04/Aug/21 ] |
|
I did a whitespace cleanup pull/1 on the "workflow-templates" area. That will help to keep future modifications and copies to be free from clutter. However, now wherever this template was copied to, will be out of synchronisation. |
| Comment by John Malconian [ 04/Aug/21 ] |
|
Ankita Sen I've reviewed both the buildNPM and buildnpmRelease workflows on master in the .github repo. 1. Both workflows need some cleanup. e.g remove test branch from conditionals and substitute the NPM and module descriptor test registries with the real values (using vars). 2. In the release workflow, I don't understand this test: PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[\",]//g' | tr -d '[[:space:]]') && git tag v$PACKAGE_VERSION
The tag should already exist and is what triggers the release workflow. All we need to do is make sure the version in the package.json matches the git tag we are building in the workflow. May be I'm not understanding something here. Also, you may want to consider getting 'version' from package.json using a JSON parser like: https://github.com/marketplace/actions/get-json-property 3. We really should try to use the OKAPI_IP var in the following steps instead of hard-coding 172.17.0.2. I don't think we can rely on this always being the same. - name: Pull all Module descriptors to local Okapi instance
if: ${{ env.PUBLISH_MOD_DESCRIPTOR }}
uses: fjogeleit/http-request-action@master
with:
url: http://172.17.0.2:9130/_/proxy/pull/modules
method: 'POST'
contentType: 'application/json; charset=utf-8'
customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
data: ${{ env.OKAPI_PULL }}
timeout: 60000
- name: Perform local Okapi dependency check
if: ${{ env.PUBLISH_MOD_DESCRIPTOR }}
uses: fjogeleit/http-request-action@master
with:
# use in production
#url: 'http://172.17.0.2:9130/_/proxy/modules?preRelease=false&npmSnapshot=false'
# line below is for testing only --malc
url: 'http://172.17.0.2:9130/_/proxy/modules'
method: 'POST'
contentType: 'application/json; charset=utf-8'
customHeaders: '{ "Accept": "application/json; charset=utf-8" }'
data: ${{ steps.moduleDescriptor.outputs.content }}
|
| Comment by John Malconian [ 04/Aug/21 ] |
|
Also, and most importantly, I think, the release workflow should only be triggered when a tag has been created. All the conditionals checking for 'master' or 'main' in the buildNPM workflow shouldn't be applied to the release workflow. See this example, https://github.com/actions/runner/issues/1007#issuecomment-808904408 |
| Comment by Jakub Skoczen [ 05/Aug/21 ] |
|
Once this is complete we need to sync up with Zak Burke regarding rolling this out in individual repos (the goal is to use the new workflow for Kiwi releases). |
| Comment by John Malconian [ 05/Aug/21 ] |
|
Ankita Sen - I've spent a good chunk of time testing the following condition today and am not able to get it to ever return true. if: ${{ env.TAG != steps.version.outputs.prop }}
I've tried several variations with no success including if: ${{ env.TAG }} != ${{ steps.version.outputs.prop }}
You can see an example here: https://github.com/folio-org/ga-test/runs/3255673370?check_suite_focus=true Feel free to mess around with this in the ga-test repo. My work is on the 'v1.2' branch. |
| Comment by David Crossley [ 06/Aug/21 ] |
|
Hooray. I too spent some workflow stuggle time. Mine finally yielded greenness. Use the "id" of the relevant step, and use the "if: ${{ ..." if: ${{ env.TAG_VERSION != steps.package_version.outputs.prop }}
|
| Comment by John Malconian [ 06/Aug/21 ] |
|
Gah!! Another stupid mistake on my part getting the id wrong. Thanks David Crossley |
| Comment by John Malconian [ 10/Aug/21 ] |
|
I'm closing this issue since we are done with the development of the release workflow. Roll-out of the new workflows can be tracked in
|