[FOLIO-3657] Create additional disk space on Jenkins master and agent nodes Created: 05/Dec/22 Updated: 09/Dec/22 Resolved: 05/Dec/22 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | TBD |
| Reporter: | John Malconian | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Sprint: | DevOps Sprint 154 |
| Development Team: | FOLIO DevOps |
| RCA Group: | TBD |
| Description |
|
Storage capacity is quickly filling up on Jenkins master at - 95% capacity. Jenkins agent build nodes also have storage capacity issues - often filling up with Docker images used by various module unit tests and not cleaned up post-build for whatever reason. |
| Comments |
| Comment by John Malconian [ 05/Dec/22 ] |
|
On Jenkins master, there are a few different issues that need to be managed that result in storage usage. 1. Jenkins is supposed to prune build data from deleted branches and closed PRs after one day, but that is not happening - possibly due to Github cache settings that stop Jenkins from excessively using the Github API to sync repositories between Jenkins and Github. It's a bit of tightrope managing the cache vs syncing with Github due to the sheer number of repos and branches Jenkins is managing and the API quotas Github imposes. Therefore, periodic, manual repository scans must be initiated in Jenkins for each repository to prune old branches and PRs from Jenkins. 2. Certain jobs run on the Jenkins master (jobs related to the Rancher and performance framework environments are consuming more space and often require manual intervention to clean up. Jenkins attempts to do this but is lacking the proper permissions in some cases. Jenkins reports these cases in the console. Manual clean up by an admin is periodically required. On the Jenkins agent nodes we see regular diminished disk capacity due to docker images left behind from build jobs. There are typically used by backend modules for unit testing and are often not cleaned up post-test. Implement daily clean up scripts that will delete these Docker images and volumes. |
| Comment by John Malconian [ 05/Dec/22 ] |
|
For now, I've added an additional 500GB of capacity to the Jenkins master node and have implemented a script on each agent nodes that runs daily to clean up Docker images and volumes. |
| Comment by John Malconian [ 09/Dec/22 ] |
|
Added an additional 50GB to each Jenkins build node (2-5). |