[FOLIO-2460] test and move container cleanup job from K8s cron to jenkins Created: 10/Feb/20 Updated: 03/Jun/20 Resolved: 27/Mar/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | New Feature | Priority: | P3 |
| Reporter: | Jakub Skoczen | Assignee: | Ian Hardy |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||
| Sprint: | DevOps: sprint 84, DevOps: sprint 85, DevOps: sprint 83 | ||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||
| Description |
|
Module cleanup job for CI kubernetes cluster should be executed from Jenkins. Running as a kubernetes cron job has been problematic because it's not as easy to see the results of older cleanup jobs. Consider re-writing this job as the crob job has been problematic. Requirements:
|
| Comments |
| Comment by Ian Hardy [ 05/Mar/20 ] |
|
There's a lot of nuance to try to catch in this cleanup job, and I think a script is still the way to do it. To run such a script from jenkins we either need to add a bunch of new deps to the jenkins-slave-all image, or use another image as the build agent. I know John has thought about breaking up that build agent in the past. John Malconian any thoughts on what the best practice would be? I'm not sure whats all involved with adding a new agent and if doing that would be preferable to expanding the existing jenkins-slave-all image. If we did add a new image, I'd propose it would be a ci-scripts image with python3, plus whatever other scripting environments we may need (perl, etc). |
| Comment by Ian Hardy [ 05/Mar/20 ] |
|
I've been doing some testing with the --dry-run flag enabled on the existing cleanup script and it doesn't appear to be mis-deleting any modules. The script is the same as the one in the docker image with the "latest" tag in dockerhub, but I see the sha256 hash of the container differs. the kubernetes job must not be pulling the latest image. I think its worth re-using the script since it appears to work fine, but we do need to move this job into jenkins for better auditing. |
| Comment by Ian Hardy [ 23/Mar/20 ] |
|
John Malconian fyi re-enabled module cleanup on the default namespace. Example output: https://jenkins-aws.indexdata.com/job/Automation/job/kubernetes-module-cleanup/31/console |