[FOLIO-3713] Jenkins failure due to insufficient disk capacity Created: 17/Feb/23  Updated: 17/Feb/23  Resolved: 17/Feb/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps Sprint 159
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   

Julian reported a failed Jenkins job due to insufficient disk capacity on Slack.



 Comments   
Comment by John Malconian [ 17/Feb/23 ]

Looks like the insufficient disk capacity problems is on the main Jenkins server. /home is 100% full. This is where the Jenkins home/data directory is located. Increased disk capacity from 1.5TB to 2.0TB so that folks can continue to work.

From what I can tell so far there is a lot of disk capacity used on cached Github info about old branches and PRs for folio-org repos. I've initiated a scan (via script) of all repositories. The scan will refresh all Github caches and remove obsolete data. Jenkins is supposed to do this automatically, but I've found I've had to do it manually periodically. When this process is finished, approximately 250GB of disk capacity will be freed.

I've also reported to Oleksandr on Slack that Testing/* Jenkins jobs account from almost 200GB of disk usage. I'm hoping something can be done there to either reduce the artifacts that are kept from these jobs and/or adjust the rotation the jobs so that only data from the last X number of jobs is retained.

Comment by John Malconian [ 17/Feb/23 ]

~500GB of disk space has been reclaimed. That's pretty decent. Closing issue.

Generated at Thu Feb 08 23:30:09 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.