[FOLIO-3216] perf-testing job not cleaning up Jenkins workspace Created: 23/Jun/21 Updated: 02/Jul/21 Resolved: 02/Jul/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | John Malconian | Assignee: | Drif Abdenour |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Sprint: | DevOps Sprint 117 |
| Development Team: | FOLIO DevOps |
| Description |
|
Jenkins jobs running in https://jenkins-aws.indexdata.com/job/performance_framework/ run on the Jenkins master server. The Jenkins workspace for these jobs has accumulated ~38gb of data. It's possible the workspace is not getting cleaned up properly in all cases - perhaps when jobs are aborted or fail. |
| Comments |
| Comment by John Malconian [ 23/Jun/21 ] |
|
The following is a list of directories under the performance_framework workspace and size of each directory. 4.1M carrier-io-ptf du: cannot access ‘carrier-io’: No such file or directory 287M carrier-io-ptf@script du: cannot access ‘carrier-io’: No such file or directory 4.1k carrier-io-ptf@script@tmp du: cannot access ‘carrier-io’: No such file or directory 4.1k carrier-io-ptf@tmp 4.1k carrier-io-shutdown-load-generator 4.1k carrier-io-shutdown-load-generator@tmp 291M carrier-io-single-test-1 291M carrier-io-single-test-1@script 4.1k carrier-io-single-test-1@script@tmp 4.1k carrier-io-single-test-1@tmp 517M perf-env 510M perf-env-2 519M perf-env-2@2 4.1k perf-env-2@2@tmp 473M perf-env-2@2_ws-cleanup_1624028394544 473M perf-env-2@2_ws-cleanup_1624189733237 473M perf-env-2@2_ws-cleanup_1624193938930 473M perf-env-2@2_ws-cleanup_1624194262919 473M perf-env-2@2_ws-cleanup_1624198481354 476M perf-env-2@2_ws-cleanup_1624202375762 46M perf-env-2@script 4.1k perf-env-2@script@tmp 4.1k perf-env-2@tmp 476M perf-env-2_ws-cleanup_1624020884010 473M perf-env-2_ws-cleanup_1624020942983 473M perf-env-2_ws-cleanup_1624022226752 473M perf-env-2_ws-cleanup_1624023514796 473M perf-env-2_ws-cleanup_1624031346272 22M perf-env-2_ws-cleanup_1624183658562 476M perf-env-2_ws-cleanup_1624187627794 476M perf-env-2_ws-cleanup_1624193926750 476M perf-env-2_ws-cleanup_1624194254534 473M perf-env-2_ws-cleanup_1624198473400 476M perf-env-2_ws-cleanup_1624202363390 476M perf-env-2_ws-cleanup_1624208570742 473M perf-env-2_ws-cleanup_1624208937482 473M perf-env-2_ws-cleanup_1624221957000 473M perf-env-2_ws-cleanup_1624346329667 476M perf-env-2_ws-cleanup_1624353670063 216M perf-env-2_ws-cleanup_1624354105981 466M perf-env-2_ws-cleanup_1624357032613 466M perf-env-2_ws-cleanup_1624359205959 466M perf-env-2_ws-cleanup_1624368990125 469M perf-env-2_ws-cleanup_1624369206203 466M perf-env-2_ws-cleanup_1624369524730 537M perf-env-3 510M perf-env-3@2 4.1k perf-env-3@2@tmp 52M perf-env-3@script 4.1k perf-env-3@script@tmp 4.1k perf-env-3@tmp 452M perf-env-3_ws-cleanup_1624381789375 466M perf-env-3_ws-cleanup_1624381913130 466M perf-env-3_ws-cleanup_1624381991372 466M perf-env-3_ws-cleanup_1624382113494 466M perf-env-3_ws-cleanup_1624382389250 200M perf-env-3_ws-cleanup_1624382413439 469M perf-env-3_ws-cleanup_1624382658300 469M perf-env-3_ws-cleanup_1624383312921 466M perf-env-3_ws-cleanup_1624383583500 466M perf-env-3_ws-cleanup_1624383806638 466M perf-env-3_ws-cleanup_1624387706870 466M perf-env-3_ws-cleanup_1624388224334 466M perf-env-3_ws-cleanup_1624394417676 466M perf-env-3_ws-cleanup_1624394575317 469M perf-env-3_ws-cleanup_1624430378282 466M perf-env-3_ws-cleanup_1624430535939 469M perf-env-3_ws-cleanup_1624446437372 469M perf-env-3_ws-cleanup_1624446674930 469M perf-env-3_ws-cleanup_1624449075872 469M perf-env-3_ws-cleanup_1624451652837 466M perf-env-3_ws-cleanup_1624454490878 469M perf-env-3_ws-cleanup_1624456824180 466M perf-env-3_ws-cleanup_1624458509374 56M perf-env@script 4.1k perf-env@script@tmp 4.1k perf-env@tmp 479M perf-env_ws-cleanup_1620380592368 471M perf-env_ws-cleanup_1621498095773 471M perf-env_ws-cleanup_1621500611515 471M perf-env_ws-cleanup_1621790172355 474M perf-env_ws-cleanup_1621790235388 471M perf-env_ws-cleanup_1621792109851 471M perf-env_ws-cleanup_1621860906860 471M perf-env_ws-cleanup_1621865690721 468M perf-env_ws-cleanup_1623925355083 473M perf-env_ws-cleanup_1623926638940 475M perf-env_ws-cleanup_1623926847637 473M perf-env_ws-cleanup_1623926916905 475M perf-env_ws-cleanup_1623929527298 473M perf-env_ws-cleanup_1624205846320 473M perf-env_ws-cleanup_1624254240582 473M perf-env_ws-cleanup_1624361663545 12M perf-env_ws-cleanup_1624361767131 12M perf-env_ws-cleanup_1624362037974 12M perf-env_ws-cleanup_1624362088808 12M perf-env_ws-cleanup_1624367429609 469M perf-env_ws-cleanup_1624373872873 19M perf-env_ws-cleanup_1624376090645 466M perf-env_ws-cleanup_1624379487122 466M perf-env_ws-cleanup_1624437076214 469M perf-env_ws-cleanup_1624441969653 351M ptf-carrier-io 291M ptf-carrier-io@script 4.1k ptf-carrier-io@script@tmp 8.2k ptf-carrier-io@tmp 458M ptf-env 25k ptf-register-okapi-1 1.2M ptf-register-okapi@libs 43M ptf-register-okapi@script 4.1k ptf-register-okapi@script@tmp 43M ptf-regiter-okapi 1.2M ptf-regiter-okapi@libs 45M ptf-regiter-okapi@script 4.1k ptf-regiter-okapi@script@tmp 4.1k ptf-regiter-okapi@tmp 4.1k test 4.1k test@tmp |
| Comment by John Malconian [ 23/Jun/21 ] |
|
Last modification times of each directory. drwxr-xr-x 19 jenkins jenkins 4096 Jun 23 14:59 perf-env-3 drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 14:59 perf-env-3@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:59 perf-env-3_ws-cleanup_1624460351822 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:28 perf-env-3_ws-cleanup_1624458509374 drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 14:02 perf-env-3@script@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:00 perf-env-3_ws-cleanup_1624456824180 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 13:21 perf-env-3_ws-cleanup_1624454490878 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 12:34 perf-env-3_ws-cleanup_1624451652837 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:51 perf-env-3_ws-cleanup_1624449075872 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:11 perf-env-3_ws-cleanup_1624446674930 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:07 perf-env-3_ws-cleanup_1624446437372 drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 10:05 perf-env@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 09:52 perf-env_ws-cleanup_1624441969653 drwxr-xr-x 19 jenkins jenkins 4096 Jun 23 09:52 perf-env drwxr-xr-x 2 jenkins jenkins 4096 Jun 23 09:52 perf-env@script@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 08:31 perf-env_ws-cleanup_1624437076214 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 06:42 perf-env-3_ws-cleanup_1624430535939 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 06:39 perf-env-3_ws-cleanup_1624430378282 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 20:42 perf-env-3_ws-cleanup_1624394575317 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 20:40 perf-env-3_ws-cleanup_1624394417676 drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 19:39 perf-env-3@2 drwxr-xr-x 2 jenkins jenkins 4096 Jun 22 19:39 perf-env-3@2@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 18:57 perf-env-3_ws-cleanup_1624388224334 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 18:48 perf-env-3_ws-cleanup_1624387706870 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:43 perf-env-3_ws-cleanup_1624383806638 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:39 perf-env-3_ws-cleanup_1624383583500 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:35 perf-env-3_ws-cleanup_1624383312921 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:24 perf-env-3_ws-cleanup_1624382658300 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:20 perf-env-3_ws-cleanup_1624382413439 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:19 perf-env-3_ws-cleanup_1624382389250 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:15 perf-env-3_ws-cleanup_1624382113494 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:13 perf-env-3_ws-cleanup_1624381991372 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:11 perf-env-3_ws-cleanup_1624381913130 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 17:09 perf-env-3_ws-cleanup_1624381789375 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 16:31 perf-env_ws-cleanup_1624379487122 drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 16:26 perf-env-3@script drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 15:34 perf-env_ws-cleanup_1624376090645 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 14:57 perf-env_ws-cleanup_1624373872873 drwxr-xr-x 19 jenkins jenkins 4096 Jun 22 13:45 perf-env-2 drwxr-xr-x 2 jenkins jenkins 4096 Jun 22 13:45 perf-env-2@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 13:45 perf-env-2_ws-cleanup_1624369524730 drwxr-xr-x 2 jenkins jenkins 4096 Jun 22 13:45 perf-env-2@script@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 13:40 perf-env-2_ws-cleanup_1624369206203 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 13:36 perf-env-2_ws-cleanup_1624368990125 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 13:10 perf-env_ws-cleanup_1624367429609 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 11:41 perf-env_ws-cleanup_1624362088808 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 11:40 perf-env_ws-cleanup_1624362037974 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 11:36 perf-env_ws-cleanup_1624361767131 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 11:34 perf-env_ws-cleanup_1624361663545 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 10:53 perf-env-2_ws-cleanup_1624359205959 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 10:17 perf-env-2_ws-cleanup_1624357032613 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 09:28 perf-env-2_ws-cleanup_1624354105981 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 09:21 perf-env-2_ws-cleanup_1624353670063 drwxr-xr-x 3 jenkins jenkins 4096 Jun 22 07:18 perf-env-2_ws-cleanup_1624346329667 drwxr-xr-x 3 jenkins jenkins 4096 Jun 21 05:44 perf-env_ws-cleanup_1624254240582 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 20:45 perf-env-2_ws-cleanup_1624221957000 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 17:08 perf-env-2_ws-cleanup_1624208937482 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 17:02 perf-env-2_ws-cleanup_1624208570742 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 16:17 perf-env_ws-cleanup_1624205846320 drwxr-xr-x 19 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2 drwxr-xr-x 2 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2@tmp drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 15:19 perf-env-2@2_ws-cleanup_1624202375762 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 15:19 perf-env-2_ws-cleanup_1624202363390 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 14:14 perf-env-2@2_ws-cleanup_1624198481354 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 14:14 perf-env-2_ws-cleanup_1624198473400 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 13:04 perf-env-2@2_ws-cleanup_1624194262919 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 13:04 perf-env-2_ws-cleanup_1624194254534 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 12:58 perf-env-2@2_ws-cleanup_1624193938930 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 12:58 perf-env-2_ws-cleanup_1624193926750 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 11:48 perf-env-2@2_ws-cleanup_1624189733237 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 11:13 perf-env-2_ws-cleanup_1624187627794 drwxr-xr-x 3 jenkins jenkins 4096 Jun 20 10:07 perf-env-2_ws-cleanup_1624183658562 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 15:49 perf-env-2_ws-cleanup_1624031346272 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 14:59 perf-env-2@2_ws-cleanup_1624028394544 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 13:38 perf-env-2_ws-cleanup_1624023514796 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 13:17 perf-env-2_ws-cleanup_1624022226752 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 12:55 perf-env-2_ws-cleanup_1624020942983 drwxr-xr-x 3 jenkins jenkins 4096 Jun 18 12:54 perf-env-2_ws-cleanup_1624020884010 drwxr-xr-x 19 jenkins jenkins 4096 Jun 18 12:53 perf-env-2@script drwxr-xr-x 3 jenkins jenkins 4096 Jun 17 11:32 perf-env_ws-cleanup_1623929527298 drwxr-xr-x 3 jenkins jenkins 4096 Jun 17 10:48 perf-env_ws-cleanup_1623926916905 drwxr-xr-x 3 jenkins jenkins 4096 Jun 17 10:47 perf-env_ws-cleanup_1623926847637 drwxr-xr-x 3 jenkins jenkins 4096 Jun 17 10:43 perf-env_ws-cleanup_1623926638940 drwxr-xr-x 3 jenkins jenkins 4096 Jun 17 10:22 perf-env_ws-cleanup_1623925355083 drwxr-xr-x 3 jenkins jenkins 4096 May 24 14:14 perf-env_ws-cleanup_1621865690721 drwxr-xr-x 3 jenkins jenkins 4096 May 24 12:55 perf-env_ws-cleanup_1621860906860 drwxr-xr-x 3 jenkins jenkins 4096 May 23 17:48 perf-env_ws-cleanup_1621792109851 drwxr-xr-x 3 jenkins jenkins 4096 May 23 17:17 perf-env_ws-cleanup_1621790235388 drwxr-xr-x 3 jenkins jenkins 4096 May 23 17:16 perf-env_ws-cleanup_1621790172355 drwxr-xr-x 3 jenkins jenkins 4096 May 20 08:50 perf-env_ws-cleanup_1621500611515 drwxr-xr-x 3 jenkins jenkins 4096 May 20 08:08 perf-env_ws-cleanup_1621498095773 drwxr-xr-x 3 jenkins jenkins 4096 May 7 09:43 perf-env_ws-cleanup_1620380592368 drwxr-xr-x 19 jenkins jenkins 4096 Feb 16 13:52 perf-env@script drwxr-xr-x 3 jenkins jenkins 4096 Feb 3 22:26 ptf-env drwxr-xr-x 2 jenkins jenkins 4096 Dec 7 2020 ptf-register-okapi@script@tmp drwxr-xr-x 4 jenkins jenkins 4096 Nov 23 2020 ptf-register-okapi@libs drwxr-xr-x 19 jenkins jenkins 4096 Nov 23 2020 ptf-register-okapi@script drwxr-xr-x 2 jenkins jenkins 4096 Nov 17 2020 ptf-regiter-okapi@script@tmp drwxr-xr-x 4 jenkins jenkins 4096 Nov 17 2020 ptf-regiter-okapi@libs drwxr-xr-x 19 jenkins jenkins 4096 Nov 17 2020 ptf-regiter-okapi drwxr-xr-x 2 jenkins jenkins 4096 Nov 17 2020 ptf-regiter-okapi@tmp drwxr-xr-x 3 jenkins jenkins 4096 Nov 17 2020 ptf-register-okapi-1 drwxr-xr-x 19 jenkins jenkins 4096 Nov 12 2020 ptf-regiter-okapi@script drwxr-xr-x 2 jenkins jenkins 4096 Oct 20 2020 carrier-io-single-test-1@tmp drwxr-xr-x 7 jenkins jenkins 4096 Oct 20 2020 carrier-io-single-test-1 drwxr-xr-x 2 jenkins jenkins 4096 Oct 20 2020 carrier-io-single-test-1@script@tmp drwxr-xr-x 7 jenkins jenkins 4096 Oct 20 2020 carrier-io-single-test-1@script drwxr-xr-x 3 jenkins jenkins 4096 Sep 18 2020 ptf-carrier-io@tmp drwxr-xr-x 7 jenkins jenkins 4096 Sep 18 2020 ptf-carrier-io drwxr-xr-x 2 jenkins jenkins 4096 Sep 18 2020 ptf-carrier-io@script@tmp drwxr-xr-x 2 jenkins jenkins 4096 Sep 18 2020 test@tmp drwxr-xr-x 7 jenkins jenkins 4096 Sep 11 2020 ptf-carrier-io@script drwxr-xr-x 2 jenkins jenkins 4096 Sep 10 2020 test drwxr-xr-x 2 jenkins jenkins 4096 Sep 3 2020 carrier-io-shutdown-load-generator@tmp drwxr-xr-x 2 jenkins jenkins 4096 Sep 3 2020 carrier-io-ptf@tmp drwxr-xr-x 2 jenkins jenkins 4096 Sep 3 2020 carrier-io-ptf@script@tmp drwxr-xr-x 3 jenkins jenkins 4096 Sep 2 2020 carrier-io-ptf drwxr-xr-x 7 jenkins jenkins 4096 Sep 2 2020 carrier-io-ptf@script drwxr-xr-x 2 jenkins jenkins 4096 Sep 2 2020 carrier-io ptf@tmp drwxr-xr-x 7 jenkins jenkins 4096 Sep 2 2020 carrier-io ptf drwxr-xr-x 2 jenkins jenkins 4096 Sep 2 2020 carrier-io ptf@script@tmp drwxr-xr-x 7 jenkins jenkins 4096 Sep 2 2020 carrier-io ptf@script drwxr-xr-x 2 jenkins jenkins 4096 Sep 1 2020 carrier-io-shutdown-load-generator |
| Comment by John Malconian [ 23/Jun/21 ] |
|
I deleted any '_ws_cleanup' directories older than 1 day. That left the following directories intact. I think some may be associated with currently running jobs so I didn't remove them. Disk usage now down from 38g to 12gb. drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 06:39 perf-env-3_ws-cleanup_1624430378282 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 06:42 perf-env-3_ws-cleanup_1624430535939 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:07 perf-env-3_ws-cleanup_1624446437372 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:11 perf-env-3_ws-cleanup_1624446674930 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 11:51 perf-env-3_ws-cleanup_1624449075872 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 12:34 perf-env-3_ws-cleanup_1624451652837 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 13:21 perf-env-3_ws-cleanup_1624454490878 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:00 perf-env-3_ws-cleanup_1624456824180 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:28 perf-env-3_ws-cleanup_1624458509374 drwxr-xr-x 3 jenkins jenkins 4096 Jun 23 14:59 perf-env-3_ws-cleanup_1624460351822 |