[FOLIO-2460] test and move container cleanup job from K8s cron to jenkins Created: 10/Feb/20  Updated: 03/Jun/20  Resolved: 27/Mar/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P3
Reporter: Jakub Skoczen Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-2498 Migrate PR builds for platform-core t... Closed
relates to FOLIO-2521 Migrate PR builds for platform-comple... Closed
relates to FOLIO-2534 Move tenant cleanup job to jenkins Closed
Sprint: DevOps: sprint 84, DevOps: sprint 85, DevOps: sprint 83
Development Team: FOLIO DevOps

 Description   

Module cleanup job for CI kubernetes cluster should be executed from Jenkins. Running as a kubernetes cron job has been problematic because it's not as easy to see the results of older cleanup jobs. Consider re-writing this job as the crob job has been problematic.

Requirements:

  • Job is executed from jenkins, easy to see which modules were deleted and why
  • Job is able to correctly identify oldest module by semvar.


 Comments   
Comment by Ian Hardy [ 05/Mar/20 ]

There's a lot of nuance to try to catch in this cleanup job, and I think a script is still the way to do it. To run such a script from jenkins we either need to add a bunch of new deps to the jenkins-slave-all image, or use another image as the build agent. I know John has thought about breaking up that build agent in the past. John Malconian any thoughts on what the best practice would be? I'm not sure whats all involved with adding a new agent and if doing that would be preferable to expanding the existing jenkins-slave-all image.

If we did add a new image, I'd propose it would be a ci-scripts image with python3, plus whatever other scripting environments we may need (perl, etc).

Comment by Ian Hardy [ 05/Mar/20 ]

I've been doing some testing with the --dry-run flag enabled on the existing cleanup script and it doesn't appear to be mis-deleting any modules. The script is the same as the one in the docker image with the "latest" tag in dockerhub, but I see the sha256 hash of the container differs. the kubernetes job must not be pulling the latest image.

I think its worth re-using the script since it appears to work fine, but we do need to move this job into jenkins for better auditing.

Comment by Ian Hardy [ 23/Mar/20 ]

John Malconian fyi re-enabled module cleanup on the default namespace.

Example output: https://jenkins-aws.indexdata.com/job/Automation/job/kubernetes-module-cleanup/31/console

Generated at Thu Feb 08 23:20:48 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.