[FOLIO-3344] Clean up Jenkins workspaces after scratch environment build jobs Created: 15/Nov/21  Updated: 18/Nov/21  Resolved: 18/Nov/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: John Malconian Assignee: Hleb Surnovich
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File image (1).png     PNG File image.png    
Sprint: DevOps Sprint 127
Development Team: FOLIO DevOps

 Description   

Several jobs in the Jenkins "Scratch (Rancher) environment" folder are not cleaning up their workspaces after the job is run which results in disk filling up on the Jenkins server. These jobs include the manage-scratch-environment job and it's associated variations (1.1, 1.2, Firebird manage scratch env, etc). A clean up of the workspace should be performed whether the job was successful, failed, or terminated. A step similar to the following should be added to these Jenkins pipeline jobs:

 finally {
   stage('Clean') {
      cleanWs disableDeferredWipeout: true, deleteDirs: true, cleanWhenAborted: true, cleanWhenFailure: true, cleanWhenSuccess: true 


 Comments   
Comment by Hleb Surnovich [ 17/Nov/21 ]

Hello, John Malconian ! I took this ticket and tried to apply cleaning with CleanWS, as you mentioned above.
But I've encountered a problem: some of terraform temp files are created with root privileges, hence root is the owner – and jenkins can not delete them:
 


 
For now I can see the following solution: 

  • give Jenkins user sudo privileges
  • add sh' ' command wiith chown/chgrp
  • delete those files with ws clean plugin the way it was
  • But I'm not sure, if it's allowed to give jenkins sudo rights

cc: Aliaksei Luhavy, Aleh Litasau

Comment by John Malconian [ 17/Nov/21 ]

Thanks for taking a look at this Hleb Surnovich. I ended up just adding a cron job on the server that will clean up these files. It run's once per day. I think that should be ok for now. Feel free to close this issue. In general, I think it would preferable to run these jobs inside a Docker container like we do for most other jobs. That way, when the job is complete, the Jenkins workspace environment is automatically deleted. On the other hand, if you are using Terraform in these jobs I'm guessing that you have Terraform state files that need to persist and that may prohibit the use of a container. If so, persisting them to an S3 bucket might be an option instead of on the local filesystem. Just something to think about.

Comment by Dilshod_Khusanov [ 18/Nov/21 ]

Closed as solution is provided by John Malconian

Generated at Thu Feb 08 23:27:24 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.