[FOLIO-3216] perf-testing job not cleaning up Jenkins workspace Created: 23/Jun/21  Updated: 02/Jul/21  Resolved: 02/Jul/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: John Malconian Assignee: Drif Abdenour
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps Sprint 117
Development Team: FOLIO DevOps

 Description   

Jenkins jobs running in https://jenkins-aws.indexdata.com/job/performance_framework/ run on the Jenkins master server. The Jenkins workspace for these jobs has accumulated ~38gb of data. It's possible the workspace is not getting cleaned up properly in all cases - perhaps when jobs are aborted or fail.



 Comments   
Comment by John Malconian [ 23/Jun/21 ]

The following is a list of directories under the performance_framework workspace and size of each directory.

4.1M	carrier-io-ptf
du: cannot access ‘carrier-io’: No such file or directory
287M	carrier-io-ptf@script
du: cannot access ‘carrier-io’: No such file or directory
4.1k	carrier-io-ptf@script@tmp
du: cannot access ‘carrier-io’: No such file or directory
4.1k	carrier-io-ptf@tmp
4.1k	carrier-io-shutdown-load-generator
4.1k	carrier-io-shutdown-load-generator@tmp
291M	carrier-io-single-test-1
291M	carrier-io-single-test-1@script
4.1k	carrier-io-single-test-1@script@tmp
4.1k	carrier-io-single-test-1@tmp
517M	perf-env
510M	perf-env-2
519M	perf-env-2@2
4.1k	perf-env-2@2@tmp
473M	perf-env-2@2_ws-cleanup_1624028394544
473M	perf-env-2@2_ws-cleanup_1624189733237
473M	perf-env-2@2_ws-cleanup_1624193938930
473M	perf-env-2@2_ws-cleanup_1624194262919
473M	perf-env-2@2_ws-cleanup_1624198481354
476M	perf-env-2@2_ws-cleanup_1624202375762
46M	perf-env-2@script
4.1k	perf-env-2@script@tmp
4.1k	perf-env-2@tmp
476M	perf-env-2_ws-cleanup_1624020884010
473M	perf-env-2_ws-cleanup_1624020942983
473M	perf-env-2_ws-cleanup_1624022226752
473M	perf-env-2_ws-cleanup_1624023514796
473M	perf-env-2_ws-cleanup_1624031346272
22M	perf-env-2_ws-cleanup_1624183658562
476M	perf-env-2_ws-cleanup_1624187627794
476M	perf-env-2_ws-cleanup_1624193926750
476M	perf-env-2_ws-cleanup_1624194254534
473M	perf-env-2_ws-cleanup_1624198473400
476M	perf-env-2_ws-cleanup_1624202363390
476M	perf-env-2_ws-cleanup_1624208570742
473M	perf-env-2_ws-cleanup_1624208937482
473M	perf-env-2_ws-cleanup_1624221957000
473M	perf-env-2_ws-cleanup_1624346329667
476M	perf-env-2_ws-cleanup_1624353670063
216M	perf-env-2_ws-cleanup_1624354105981
466M	perf-env-2_ws-cleanup_1624357032613
466M	perf-env-2_ws-cleanup_1624359205959
466M	perf-env-2_ws-cleanup_1624368990125
469M	perf-env-2_ws-cleanup_1624369206203
466M	perf-env-2_ws-cleanup_1624369524730
537M	perf-env-3
510M	perf-env-3@2
4.1k	perf-env-3@2@tmp
52M	perf-env-3@script
4.1k	perf-env-3@script@tmp
4.1k	perf-env-3@tmp
452M	perf-env-3_ws-cleanup_1624381789375
466M	perf-env-3_ws-cleanup_1624381913130
466M	perf-env-3_ws-cleanup_1624381991372
466M	perf-env-3_ws-cleanup_1624382113494
466M	perf-env-3_ws-cleanup_1624382389250
200M	perf-env-3_ws-cleanup_1624382413439
469M	perf-env-3_ws-cleanup_1624382658300
469M	perf-env-3_ws-cleanup_1624383312921
466M	perf-env-3_ws-cleanup_1624383583500
466M	perf-env-3_ws-cleanup_1624383806638
466M	perf-env-3_ws-cleanup_1624387706870
466M	perf-env-3_ws-cleanup_1624388224334
466M	perf-env-3_ws-cleanup_1624394417676
466M	perf-env-3_ws-cleanup_1624394575317
469M	perf-env-3_ws-cleanup_1624430378282
466M	perf-env-3_ws-cleanup_1624430535939
469M	perf-env-3_ws-cleanup_1624446437372
469M	perf-env-3_ws-cleanup_1624446674930
469M	perf-env-3_ws-cleanup_1624449075872
469M	perf-env-3_ws-cleanup_1624451652837
466M	perf-env-3_ws-cleanup_1624454490878
469M	perf-env-3_ws-cleanup_1624456824180
466M	perf-env-3_ws-cleanup_1624458509374
56M	perf-env@script
4.1k	perf-env@script@tmp
4.1k	perf-env@tmp
479M	perf-env_ws-cleanup_1620380592368
471M	perf-env_ws-cleanup_1621498095773
471M	perf-env_ws-cleanup_1621500611515
471M	perf-env_ws-cleanup_1621790172355
474M	perf-env_ws-cleanup_1621790235388
471M	perf-env_ws-cleanup_1621792109851
471M	perf-env_ws-cleanup_1621860906860
471M	perf-env_ws-cleanup_1621865690721
468M	perf-env_ws-cleanup_1623925355083
473M	perf-env_ws-cleanup_1623926638940
475M	perf-env_ws-cleanup_1623926847637
473M	perf-env_ws-cleanup_1623926916905
475M	perf-env_ws-cleanup_1623929527298
473M	perf-env_ws-cleanup_1624205846320
473M	perf-env_ws-cleanup_1624254240582
473M	perf-env_ws-cleanup_1624361663545
12M	perf-env_ws-cleanup_1624361767131
12M	perf-env_ws-cleanup_1624362037974
12M	perf-env_ws-cleanup_1624362088808
12M	perf-env_ws-cleanup_1624367429609
469M	perf-env_ws-cleanup_1624373872873
19M	perf-env_ws-cleanup_1624376090645
466M	perf-env_ws-cleanup_1624379487122
466M	perf-env_ws-cleanup_1624437076214
469M	perf-env_ws-cleanup_1624441969653
351M	ptf-carrier-io
291M	ptf-carrier-io@script
4.1k	ptf-carrier-io@script@tmp
8.2k	ptf-carrier-io@tmp
458M	ptf-env
25k	ptf-register-okapi-1
1.2M	ptf-register-okapi@libs
43M	ptf-register-okapi@script
4.1k	ptf-register-okapi@script@tmp
43M	ptf-regiter-okapi
1.2M	ptf-regiter-okapi@libs
45M	ptf-regiter-okapi@script
4.1k	ptf-regiter-okapi@script@tmp
4.1k	ptf-regiter-okapi@tmp
4.1k	test
4.1k	test@tmp
Comment by John Malconian [ 23/Jun/21 ]

Last modification times of each directory.

drwxr-xr-x 19 jenkins jenkins 4096 Jun 23 14:59 perf-env-3
drwxr-xr-x  2 jenkins jenkins 4096 Jun 23 14:59 perf-env-3@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:59 perf-env-3_ws-cleanup_1624460351822
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:28 perf-env-3_ws-cleanup_1624458509374
drwxr-xr-x  2 jenkins jenkins 4096 Jun 23 14:02 perf-env-3@script@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:00 perf-env-3_ws-cleanup_1624456824180
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 13:21 perf-env-3_ws-cleanup_1624454490878
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 12:34 perf-env-3_ws-cleanup_1624451652837
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:51 perf-env-3_ws-cleanup_1624449075872
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:11 perf-env-3_ws-cleanup_1624446674930
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:07 perf-env-3_ws-cleanup_1624446437372
drwxr-xr-x  2 jenkins jenkins 4096 Jun 23 10:05 perf-env@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 09:52 perf-env_ws-cleanup_1624441969653
drwxr-xr-x 19 jenkins jenkins 4096 Jun 23 09:52 perf-env
drwxr-xr-x  2 jenkins jenkins 4096 Jun 23 09:52 perf-env@script@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 08:31 perf-env_ws-cleanup_1624437076214
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 06:42 perf-env-3_ws-cleanup_1624430535939
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 06:39 perf-env-3_ws-cleanup_1624430378282
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 20:42 perf-env-3_ws-cleanup_1624394575317
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 20:40 perf-env-3_ws-cleanup_1624394417676
drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 19:39 perf-env-3@2
drwxr-xr-x  2 jenkins jenkins 4096 Jun 22 19:39 perf-env-3@2@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 18:57 perf-env-3_ws-cleanup_1624388224334
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 18:48 perf-env-3_ws-cleanup_1624387706870
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:43 perf-env-3_ws-cleanup_1624383806638
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:39 perf-env-3_ws-cleanup_1624383583500
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:35 perf-env-3_ws-cleanup_1624383312921
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:24 perf-env-3_ws-cleanup_1624382658300
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:20 perf-env-3_ws-cleanup_1624382413439
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:19 perf-env-3_ws-cleanup_1624382389250
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:15 perf-env-3_ws-cleanup_1624382113494
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:13 perf-env-3_ws-cleanup_1624381991372
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:11 perf-env-3_ws-cleanup_1624381913130
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 17:09 perf-env-3_ws-cleanup_1624381789375
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 16:31 perf-env_ws-cleanup_1624379487122
drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 16:26 perf-env-3@script
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 15:34 perf-env_ws-cleanup_1624376090645
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 14:57 perf-env_ws-cleanup_1624373872873
drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 13:45 perf-env-2
drwxr-xr-x  2 jenkins jenkins 4096 Jun 22 13:45 perf-env-2@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 13:45 perf-env-2_ws-cleanup_1624369524730
drwxr-xr-x  2 jenkins jenkins 4096 Jun 22 13:45 perf-env-2@script@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 13:40 perf-env-2_ws-cleanup_1624369206203
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 13:36 perf-env-2_ws-cleanup_1624368990125
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 13:10 perf-env_ws-cleanup_1624367429609
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 11:41 perf-env_ws-cleanup_1624362088808
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 11:40 perf-env_ws-cleanup_1624362037974
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 11:36 perf-env_ws-cleanup_1624361767131
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 11:34 perf-env_ws-cleanup_1624361663545
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 10:53 perf-env-2_ws-cleanup_1624359205959
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 10:17 perf-env-2_ws-cleanup_1624357032613
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 09:28 perf-env-2_ws-cleanup_1624354105981
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 09:21 perf-env-2_ws-cleanup_1624353670063
drwxr-xr-x  3 jenkins jenkins 4096 Jun 22 07:18 perf-env-2_ws-cleanup_1624346329667
drwxr-xr-x  3 jenkins jenkins 4096 Jun 21 05:44 perf-env_ws-cleanup_1624254240582
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 20:45 perf-env-2_ws-cleanup_1624221957000
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 17:08 perf-env-2_ws-cleanup_1624208937482
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 17:02 perf-env-2_ws-cleanup_1624208570742
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 16:17 perf-env_ws-cleanup_1624205846320
drwxr-xr-x 19 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2
drwxr-xr-x  2 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2_ws-cleanup_1624202375762
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 15:19 perf-env-2_ws-cleanup_1624202363390
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 14:14 perf-env-2@2_ws-cleanup_1624198481354
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 14:14 perf-env-2_ws-cleanup_1624198473400
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 13:04 perf-env-2@2_ws-cleanup_1624194262919
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 13:04 perf-env-2_ws-cleanup_1624194254534
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 12:58 perf-env-2@2_ws-cleanup_1624193938930
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 12:58 perf-env-2_ws-cleanup_1624193926750
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 11:48 perf-env-2@2_ws-cleanup_1624189733237
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 11:13 perf-env-2_ws-cleanup_1624187627794
drwxr-xr-x  3 jenkins jenkins 4096 Jun 20 10:07 perf-env-2_ws-cleanup_1624183658562
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 15:49 perf-env-2_ws-cleanup_1624031346272
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 14:59 perf-env-2@2_ws-cleanup_1624028394544
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 13:38 perf-env-2_ws-cleanup_1624023514796
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 13:17 perf-env-2_ws-cleanup_1624022226752
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 12:55 perf-env-2_ws-cleanup_1624020942983
drwxr-xr-x  3 jenkins jenkins 4096 Jun 18 12:54 perf-env-2_ws-cleanup_1624020884010
drwxr-xr-x 19 jenkins jenkins 4096 Jun 18 12:53 perf-env-2@script
drwxr-xr-x  3 jenkins jenkins 4096 Jun 17 11:32 perf-env_ws-cleanup_1623929527298
drwxr-xr-x  3 jenkins jenkins 4096 Jun 17 10:48 perf-env_ws-cleanup_1623926916905
drwxr-xr-x  3 jenkins jenkins 4096 Jun 17 10:47 perf-env_ws-cleanup_1623926847637
drwxr-xr-x  3 jenkins jenkins 4096 Jun 17 10:43 perf-env_ws-cleanup_1623926638940
drwxr-xr-x  3 jenkins jenkins 4096 Jun 17 10:22 perf-env_ws-cleanup_1623925355083
drwxr-xr-x  3 jenkins jenkins 4096 May 24 14:14 perf-env_ws-cleanup_1621865690721
drwxr-xr-x  3 jenkins jenkins 4096 May 24 12:55 perf-env_ws-cleanup_1621860906860
drwxr-xr-x  3 jenkins jenkins 4096 May 23 17:48 perf-env_ws-cleanup_1621792109851
drwxr-xr-x  3 jenkins jenkins 4096 May 23 17:17 perf-env_ws-cleanup_1621790235388
drwxr-xr-x  3 jenkins jenkins 4096 May 23 17:16 perf-env_ws-cleanup_1621790172355
drwxr-xr-x  3 jenkins jenkins 4096 May 20 08:50 perf-env_ws-cleanup_1621500611515
drwxr-xr-x  3 jenkins jenkins 4096 May 20 08:08 perf-env_ws-cleanup_1621498095773
drwxr-xr-x  3 jenkins jenkins 4096 May  7 09:43 perf-env_ws-cleanup_1620380592368
drwxr-xr-x 19 jenkins jenkins 4096 Feb 16 13:52 perf-env@script
drwxr-xr-x  3 jenkins jenkins 4096 Feb  3 22:26 ptf-env
drwxr-xr-x  2 jenkins jenkins 4096 Dec  7  2020 ptf-register-okapi@script@tmp
drwxr-xr-x  4 jenkins jenkins 4096 Nov 23  2020 ptf-register-okapi@libs
drwxr-xr-x 19 jenkins jenkins 4096 Nov 23  2020 ptf-register-okapi@script
drwxr-xr-x  2 jenkins jenkins 4096 Nov 17  2020 ptf-regiter-okapi@script@tmp
drwxr-xr-x  4 jenkins jenkins 4096 Nov 17  2020 ptf-regiter-okapi@libs
drwxr-xr-x 19 jenkins jenkins 4096 Nov 17  2020 ptf-regiter-okapi
drwxr-xr-x  2 jenkins jenkins 4096 Nov 17  2020 ptf-regiter-okapi@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Nov 17  2020 ptf-register-okapi-1
drwxr-xr-x 19 jenkins jenkins 4096 Nov 12  2020 ptf-regiter-okapi@script
drwxr-xr-x  2 jenkins jenkins 4096 Oct 20  2020 carrier-io-single-test-1@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Oct 20  2020 carrier-io-single-test-1
drwxr-xr-x  2 jenkins jenkins 4096 Oct 20  2020 carrier-io-single-test-1@script@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Oct 20  2020 carrier-io-single-test-1@script
drwxr-xr-x  3 jenkins jenkins 4096 Sep 18  2020 ptf-carrier-io@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Sep 18  2020 ptf-carrier-io
drwxr-xr-x  2 jenkins jenkins 4096 Sep 18  2020 ptf-carrier-io@script@tmp
drwxr-xr-x  2 jenkins jenkins 4096 Sep 18  2020 test@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Sep 11  2020 ptf-carrier-io@script
drwxr-xr-x  2 jenkins jenkins 4096 Sep 10  2020 test
drwxr-xr-x  2 jenkins jenkins 4096 Sep  3  2020 carrier-io-shutdown-load-generator@tmp
drwxr-xr-x  2 jenkins jenkins 4096 Sep  3  2020 carrier-io-ptf@tmp
drwxr-xr-x  2 jenkins jenkins 4096 Sep  3  2020 carrier-io-ptf@script@tmp
drwxr-xr-x  3 jenkins jenkins 4096 Sep  2  2020 carrier-io-ptf
drwxr-xr-x  7 jenkins jenkins 4096 Sep  2  2020 carrier-io-ptf@script
drwxr-xr-x  2 jenkins jenkins 4096 Sep  2  2020 carrier-io ptf@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Sep  2  2020 carrier-io ptf
drwxr-xr-x  2 jenkins jenkins 4096 Sep  2  2020 carrier-io ptf@script@tmp
drwxr-xr-x  7 jenkins jenkins 4096 Sep  2  2020 carrier-io ptf@script
drwxr-xr-x  2 jenkins jenkins 4096 Sep  1  2020 carrier-io-shutdown-load-generator
Comment by John Malconian [ 23/Jun/21 ]

I deleted any '_ws_cleanup' directories older than 1 day. That left the following directories intact. I think some may be associated with currently running jobs so I didn't remove them. Disk usage now down from 38g to 12gb.

drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 06:39 perf-env-3_ws-cleanup_1624430378282
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 06:42 perf-env-3_ws-cleanup_1624430535939
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:07 perf-env-3_ws-cleanup_1624446437372
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:11 perf-env-3_ws-cleanup_1624446674930
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 11:51 perf-env-3_ws-cleanup_1624449075872
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 12:34 perf-env-3_ws-cleanup_1624451652837
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 13:21 perf-env-3_ws-cleanup_1624454490878
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:00 perf-env-3_ws-cleanup_1624456824180
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:28 perf-env-3_ws-cleanup_1624458509374
drwxr-xr-x  3 jenkins jenkins 4096 Jun 23 14:59 perf-env-3_ws-cleanup_1624460351822
Generated at Thu Feb 08 23:26:28 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.