[FOLIO-3322] automating GitHub Action workflow roll out with possible tool Created: 26/Oct/21 Updated: 13/Jun/22 Resolved: 13/Jun/22 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | TBD |
| Reporter: | Ankita Sen | Assignee: | Ankita Sen |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue links: |
|
||||||||
| Sprint: | DevOps Sprint 130, DevOps Sprint 131, DevOps sprint 132, DevOps Sprint 133, DevOps Sprint 134, DevOps Sprint 135, DevOps Sprint 136, DevOps Sprint 137, DevOps Sprint 138, DevOps Sprint 139, DevOps Sprint 140, DevOps Sprint 141, DevOps Sprint 126, DevOps Sprint 127, DevOps Sprint 128, DevOps Sprint 129 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| Description |
|
Currently, rolling out the GA workflows is done in a relatively manual approach wherein for deprecating the Jenkinsfile and adding the workflows require at-least 2-3 PRs or cloning the repositories and doing them. A tool needs to be implemented which makes rolling out the workflows a little more automated to avoid human errors which have been faced while rolling out to the ui-* and stripes-* based repositories. |
| Comments |
| Comment by Jakub Skoczen [ 01/Nov/21 ] |
|
Ankita SenDavid Crossley John Malconian to meet and discuss the approach for building this tool. |
| Comment by David Crossley [ 02/Nov/21 ] |
|
One possible tool to assist, is this Python library: https://github.com/sigmavirus24/github3.py I use it with these facilities:
And Peter uses a different Python library for lokalise-push |
| Comment by Ankita Sen [ 22/Nov/21 ] |
|
David Crossley John Malconian Jakub Skoczen, I have created a new directory in folio-infrastructure[https://github.com/folio-org-priv/folio-infrastructure/tree/FOLIO-3322/github-actions-workflow-setup] and began writing the script. The script will contain the following steps:
Please let me know if we need to implement something else or something differently. Also, I would like to give a short demo of the script on 02.12.2021 (Thursday), after our standup is done |
| Comment by John Malconian [ 22/Nov/21 ] |
|
Curious to know how you intend to manage environment variables specific to the module/repo, but I guess I can wait until the demo to find out |
| Comment by David Crossley [ 23/Nov/21 ] |
|
The tool would also need to manage other Workflows (e.g. various upcoming backend ones). Perhaps read the list of workflows that are defined in the workflow-templates repository and compare with deployed ones that are relevant for the type of destination project repository. I presume that the tool will also need to update workflows in each repo to keep them synchronised with changes to the workflow-templates. |
| Comment by David Crossley [ 23/Nov/21 ] |
|
Regarding the environment variables, perhaps it could start with placeholder values, and thereafter let them be managed by the developers at each repository. When our tool does an update, then it could parse that section and retain their definitions. |
| Comment by David Crossley [ 23/Nov/21 ] |
|
Ankita Sen On Thursdays John, Jakub, and i have another meeting directly after the FOLIO DevOps one. Perhaps Mondays would be better. |
| Comment by Ankita Sen [ 01/Dec/21 ] |
|
David Crossley, John Malconian and Jakub Skoczen as suggested by David, I am moving the demo to Monday 06.12.2021 after our stand-up. |
| Comment by Jakub Skoczen [ 06/Dec/21 ] |
|
Complications with the github library. Demo moved to 13.12.2021 |
| Comment by Ankita Sen [ 13/Dec/21 ] |
|
Jakub Skoczen, David Crossley and John Malconian, here is a detailed update on the readiness of the script. Update 13.12.2021
Steps remaining to implement:
|
| Comment by Jakub Skoczen [ 13/Dec/21 ] |
|
Demo postponed to thursday 16th Dec. |
| Comment by John Malconian [ 13/Dec/21 ] |
|
Ankita Sen - How would the scenario where we need to update an existing workflow in a repo but preserve existing env vars be handled? |
| Comment by David Crossley [ 14/Dec/21 ] |
|
Yes, i too am concerned about readily updating existing workflows, and keeping them synchronised in all enabled repositories. See various previous comments in this ticket. |
| Comment by David Crossley [ 14/Dec/21 ] |
|
Also, it needs to also be able to maintain workflows beyond just these two "npm" ones, especially back-end repositories. See previous comments in this ticket. It could utilise this data file repos.json for details about each repository. The "repoType" is important, as that is otherwise difficult to determine. Also "defaultBranch". See some other workflows at dev.f.o which utilise that data. |
| Comment by David Crossley [ 14/Dec/21 ] |
|
Of course it is fantastic that this version is utilising the library to do the commits and PR. Nice. Good progress. And doing it for a group of repositories is great. So we can feed it with a list of repos. |
| Comment by David Crossley [ 05/Jan/22 ] |
|
Need a script args option for whether to disable Jenkinsfile. This will facilitate the rollout of various independent workflows for back-end repositories, prior to dealing with their primary backend workflows. For example the
|
| Comment by Jakub Skoczen [ 21/Feb/22 ] |
|
We are pushing the demo back to 28th because of President's Day |
| Comment by David Crossley [ 27/Feb/22 ] |
|
For frontend repositories, when the main workflows are added, perhaps the Jenkinsfile could be completely deleted rather than renamed. Before the PR is merged, the history of the Jenkinsfile can be consulted to ensure that any environment variables are proper. For backend repositories, this aspect will need more planning and co-ordination because some jobs will be separate workflows, and gradually implemented. The Jenkinsfile would need to be edited until all jobs are transitioned. |
| Comment by David Crossley [ 01/Mar/22 ] |
|
Ankita Sen Yesterday we discussed retrieving the main workflows from the "workflow-templates" repository rather storing copies. Here is one example of retrieving and parsing a YAML file via raw githubusercontent (using the Python "requests" and "yaml" modules.) |
| Comment by David Crossley [ 01/Mar/22 ] |
|
Ankita Sen I did investigate the github3.py module. Yes it does have a Contents.delete method (e.g. for deleting the old Jenkinsfile file on the new branch). |
| Comment by Jakub Skoczen [ 14/Mar/22 ] |
|
Ankita Sen we decided to proceed with testing the script by rolling out the pipeline across remaining UI repos. Ankita will create a ticket for this and link it here. |
| Comment by David Crossley [ 15/Mar/22 ] |
|
I added some contextual review comments to the PR pull/337 |
| Comment by Ankita Sen [ 21/Mar/22 ] |
|
I read the comments and did a bit more clean-up of the script. The create_branch_ref fails to create the ref and gives a 404 not found when I try to update the branch and create a pull request, I have tried it multiple times, I am not sure what I am missing, but it doesn't create the branch. That's why I took the supplying with SHA values along with the repository names approach. I have also created the issue FOLIO-3452 and will start rolling out the GA workflows to remaining ui-* and stripes-* repos using this tool and keep track in the issue. |
| Comment by Jakub Skoczen [ 21/Mar/22 ] |
|
John Malconian noted that we should handle local customization in the repo, Ankita Sen will look into that. |
| Comment by David Crossley [ 22/Mar/22 ] |
|
Ankita Sen Today i tried with create_branch_ref() and yes i too found that it does require the current SHA of default branch (despite the docs saying not needed). Rather than supplying the SHA as a script argument, my test script obtained it just prior to creating the new branch. Snippet of test code (omitting verification that branch does not already exist): default_branch_sha = repo.commit(repo.default_branch) branch_ref = repo.create_branch_ref("g3-branch-2", default_branch_sha) print("branch_ref={}".format(branch_ref.ref)) |
| Comment by David Crossley [ 28/Mar/22 ] |
|
Pylint is very useful when developing Python scripts. I have learned heaps through its recommendations. Do not try to follow all of its suggestions. But following some does help to create code which is more maintainable. One extremely important thing with Python code is indentation. Pylint is currently reporting some troubles around your line 115. |
| Comment by Jakub Skoczen [ 09/May/22 ] |
|
Ankita Sen what is the status update for this item? David had some comments above, will you be working on addressing them? |
| Comment by Ankita Sen [ 13/Jun/22 ] |
|
Merged and closed the PR with proper version of the developed script. |