...
This document contains the results of testing Data Export tests on 61 tenants (excluding the central tenant) on MCPT environment:
DE_instances T1(500-1K-2k). (MARC BIB) on Mobius environment with Data Export tests for 500, 1k, 2k. with Default instances export profile.
DE_instances T2(Max). (MARC BIB) on Mobius environment with Data Export tests for a maximal number of instances on each member tenant(The exact number of instances is in the section “Errors / Additional information”). with Default instances export profile.
DE_holdings T3(500-1K-2k).Holdings on Mobius environment with Data Export tests for 500, 1k, 2k. with Default holdings export profile.
DE_holdings T4(Max). Holdings on Mobius environment with Data Export tests for a maximal number of instances on each member tenant(The exact number of instances is in the section “Errors / Additional information”) with Default holdings export profile.
DE_authority T5(500-1K-2k). Authority on Mobius environment with Data Export tests for 500, 1k, 2k. with Default authority export profile.
Ticket:
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Recommendations & Jiras
Add more CPU for
Summary
The total duration:
...
Data export | Instances | Holdings | Authority | ||||||
500, sec | 1000 , sec | 2000 , sec | 500 , sec | 1000 , sec | 2000, sec | 500 , sec | 1000 , sec | 2000, sec | |
Average duration | 83 | 426 | 1184 | 10 | 87 | 405 | 5,9 | 6,2 | 10,6 |
Minimal duration | 12 | 32 | 1372127 | 5 | 16 | 32 | 5,8 | 5,7 | 1,15 |
Maximal duration | 131 | 571 | 1271372 | 29 | 127 | 504 | 6,6 | 11,7 | 15,8 |
75t percentile | 114 | 539 | 1350 | 12 | 113 | 490 | 5,8 | 6,1 | 14,4 |
90t percentile | 126 | 553 | 1362 | 19 | 119 | 500 | 5,9 | 6,5 | 15,5 |
...
DE_holdings T3(500-1K-2k) For 2000 records Holdings data export duration increased from 32 seconds for the first tenant to about 420 sec for the 22nd tenant, then the processing time was approximately from 420 to 470 sec for tenants 23 - 61.
...
...
DE_instances T2(Max) and DE_holdings T4(Max)
...
Both tests show a delay in the start time, but for DE_holdings T4(Max) maximal delay is 8 minutes for the files with the largest number of id (Tenant 42, export 640K records). The delay for the instances data-export for the same tenant is 9 hours and 50 minutes.
The duration of the longest data-export for holdings is 7 hours and 40 minutes, and 12 hours and 30 minutes for instances on 42nd tenants.
Recommendations & Jiras
Restart the tests DE_instances T2(Max) and DE_holdings T4(Max) with different POOL_SIZE, and BATCH_SIZE parameters to check Data-export duration improvement. [MDEXP-688] Make configurable params for Data-Export(POOL_SIZE, BATCH_SIZE) - FOLIO Jira (atlassian.net)
Restart Data-Import(Authority) test on mcpt to check how Kafka topics affect performance(found in the scope of data-preparation) [MODELINKS-213] Data-import process creating incorrect topics in Kafka - FOLIO Jira (atlassian.net)
...
Transaction name | DE 500, sec | DE 1K, sec | DE 2K, sec |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0001 | 12 | 32 | 127 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0002 | 16 | 51 | 207 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0003 | 23 | 73 | 330 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0004 | 22 | 100 | 494 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0005 | 27 | 122 | 606 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0006 | 29 | 131 | 704 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0007 | 28 | 154 | 782 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0008 | 34 | 170 | 875 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0009 | 39 | 191 | 936 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0010 | 38 | 216 | 965 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0011 | 40 | 228 | 1022 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0012 | 45 | 261 | 1054 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0013 | 45 | 277 | 1082 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0014 | 45 | 293 | 1108 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0015 | 52 | 308 | 1163 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0016 | 51 | 331 | 1161 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0017 | 56 | 372 | 1183 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0018 | 56 | 374 | 1209 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0019 | 63 | 399 | 1232 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0020 | 63 | 419 | 1254 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0021 | 68 | 432 | 1275 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0022 | 69 | 457 | 1271 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0023 | 72 | 479 | 1290 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0024 | 74 | 475 | 1306 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0025 | 74 | 489 | 1300 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0026 | 75 | 506 | 1319 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0027 | 85 | 503 | 1336 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0028 | 80 | 502 | 1331 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0029 | 91 | 535 | 1347 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0030 | 87 | 532 | 1344 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0031 | 87 | 530 | 1357 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0032 | 93 | 539 | 1349 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0033 | 97 | 535 | 1363 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0034 | 97 | 547 | 1357 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0035 | 97 | 560 | 1373 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0036 | 107 | 554 | 1349 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0037 | 102 | 557 | 1359 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0038 | 103 | 563 | 1370 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0039 | 109 | 571 | 1363 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0040 | 104 | 548 | 1364 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0041 | 109 | 552 | 1365 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0042 | 115 | 553 | 1362 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0043 | 119 | 497 | 1363 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0044 | 126 | 555 | 1363 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0045 | 121 | 550 | 1356 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0046 | 126 | 553 | 1352 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0047 | 120 | 544 | 1352 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0048 | 131 | 551 | 1347 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0049 | 126 | 546 | 1340 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0050 | 130 | 538 | 1336 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0051 | 127 | 533 | 1341 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0052 | 128 | 533 | 1337 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0053 | 127 | 529 | 1334 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0054 | 123 | 527 | 1320 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0055 | 122 | 516 | 1320 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0056 | 119 | 516 | 1316 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0057 | 114 | 508 | 1315 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0058 | 110 | 503 | 1306 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0059 | 108 | 501 | 1302 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0060 | 101 | 493 | 1298 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0061 | 98 | 487 | 1289 |
DE_Exporting MARC Bib records workflow Transaction cs00000001_0062 | 16 | 31 | 36 |
...
DE_instances T2(Max). The table below contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 1 set of data(all of the instances on each tenant).
...
Tenant | Number of exported instances | Duration | Shift of the start time | |
cs00000001_0001 | 65359 | 08:34:11.924 | 0:00:00 | |
cs00000001_0002 | 22370 | 02:01:17.113 | 00:00:04.576 | |
cs00000001_0003 | 136291 | 15:38:26.969 | 00:20:55.783 | |
cs00000001_0004 | 77785 | 10:42:11.942 | 00:02:52.106 | |
cs00000001_0005 | 110093 | 13:57:20.399 | 00:12:33.693 | |
cs00000001_0006 | 293580 | 17:24:21.58 | 02:58:03.642 | |
cs00000001_0007 | 279493 | 17:15:17.802 | 02:55:41.421 | |
cs00000001_0008 | 160024 | 16:21:18.866 | 00:56:26.929 | |
cs00000001_0009 | 110918 | 14:01:42.358 | 00:25:51.269 | |
cs00000001_0010 | 83254 | 11:36:56.235 | 00:14:09.228 | |
cs00000001_0011 | 94171 | 12:43:36.487 | 00:20:24.292 | |
cs00000001_0012 | 227489 | 16:52:02.623 | 02:24:44.754 | |
cs00000001_0013 | 19131 | 01:58:23.547 | 00:01:37.648 | |
cs00000001_0014 | 20746 | 02:16:27.599 | 00:01:52.916 | |
cs00000001_0015 | 26899 | 03:24:08.545 | 00:03:01.837 | |
cs00000001_0016 | 153333 | 15:46:35.492 | 01:22:21.469 | |
cs00000001_0017 | 16492 | 01:41:43.743 | 00:02:40.122 | |
cs00000001_0018 | 22263 | 02:43:54.315 | 00:04:28.863 | |
cs00000001_0019 | 5312 | 00:14:46.338 | 00:02:05.943 | |
cs00000001_0020 | 44259 | 06:30:26.536 | 00:12:04.781 | |
cs00000001_0021 | 83988 | 11:46:23.477 | 00:33:39.156 | |
cs00000001_0022 | 86900 | 12:03:06.865 | 00:36:22.655 | |
cs00000001_0023 | 91517 | 12:27:03.606 | 00:40:56.469 | |
cs00000001_0024 | 385123 | 15:35:51.105 | 05:43:10.805 | |
cs00000001_0025 | 84215 | 11:47:31.81 | 00:36:33.173 | |
cs00000001_0026 | 153912 | 15:32:54.171 | 01:42:36.675 | |
cs00000001_0027 | 48176 | 07:11:08.729 | 00:16:44.682 | |
cs00000001_0028 | 64556 | 09:30:37.567 | 00:25:12.347 | |
cs00000001_0029 | 5140 | 00:16:41.394 | 00:03:41.85 | |
cs00000001_0030 | 97589 | 12:52:29.07 | 00:50:52.289 | |
cs00000001_0031 | 71895 | 10:26:16.656 | 00:31:10.105 | |
cs00000001_0032 | 80074 | 11:22:00.789 | 00:37:31.626 | |
cs00000001_0033 | 45886 | 06:52:06.447 | 00:17:36.951 | |
cs00000001_0034 | 18240 | 02:14:18.252 | 00:07:47.49 | |
cs00000001_0035 | 135704 | 14:55:28.661 | 01:29:32.543 | |
cs00000001_0036 | 114170 | 13:56:14.427 | 01:08:43.67 | |
cs00000001_0037 | 24226 | 03:18:25.02 | 00:09:57.63 | |
cs00000001_0038 | 225563 | 16:13:32.965 | 03:06:16.602 | |
cs00000001_0039 | 128523 | 14:37:45.055 | 01:24:25.093 | |
cs00000001_0040 | 111992 | 13:47:45.898 | 01:08:36.465 | |
cs00000001_0041 | 61091 | 09:03:29.08 | 00:27:37.435 | |
cs00000001_0042 | 634675 | 12:24:24.995 | 09:48:38.838 | |
cs00000001_0043 | 2831 | 00:02:12.583 | 00:04:22.427 | |
cs00000001_0044 | 70871 | 10:19:26.438 | 00:35:14.19 | |
cs00000001_0045 | 78320 | 11:10:12.277 | 00:41:17.857 | |
cs00000001_0046 | 109677 | 13:38:14.993 | 01:08:58.069 | |
cs00000001_0047 | 48531 | 07:19:11.872 | 00:22:04.318 | |
cs00000001_0048 | 53346 | 08:01:00.977 | 00:24:42.919 | |
cs00000001_0049 | 6952 | 00:31:27.638 | 00:06:36.51 | |
cs00000001_0050 | 190736 | 15:54:19.153 | 02:34:40.768 | |
cs00000001_0051 | 351325 | 15:41:17.61 | 05:23:26.089 | |
cs00000001_0052 | 137940 | 14:54:13.405 | 01:39:56.924 | |
cs00000001_0053 | 61834 | 09:11:33.182 | 00:31:14.104 | |
cs00000001_0054 | 16297 | 02:00:46.997 | 00:09:40.899 | |
cs00000001_0055 | 176009 | 15:42:52.69 | 02:21:13.276 | |
cs00000001_0056 | 160976 | 15:28:32.362 | 02:06:05.583 | |
cs00000001_0057 | 22596 | 03:06:34.087 | 00:12:13.641 | |
cs00000001_0058 | 123819 | 14:19:45.503 | 01:27:29.477 | |
cs00000001_0059 | 37715 | 05:38:43.984 | 00:18:51.151 | |
cs00000001_0060 | 38395 | 05:45:32.811 | 00:19:17.503 | |
cs00000001_0061 | 87117 | 12:00:00.92 | 00:53:34.119 cs00000001_0062 | 1 |
DE_holdings T3(500-1K-2k). The table below contains durations for Data-export for jobs with “Default holdings export profile” on 61 tenants with 1 set of data(all of the instances on each tenant).
...
Tenant | Number of exported instances | Duration | Shift of the start time |
cs00000001_0001 | 66697 | 04:47:40.032 | 00:07:07.001 |
cs00000001_0002 | 22477 | 01:46:16.222 | 00:00:00.895 |
cs00000001_0003 | 136679 | 07:11:34.247 | 00:07:06.999 |
cs00000001_0004 | 78347 | 05:21:32.462 | 00:06:56.961 |
cs00000001_0005 | 110669 | 06:32:25.987 | 00:07:06.999 |
cs00000001_0006 | 354865 | 09:08:03.315 | 00:07:14.627 |
cs00000001_0007 | 281879 | 08:44:44.905 | 00:07:14.627 |
cs00000001_0008 | 183455 | 07:56:22.925 | 00:07:07.001 |
cs00000001_0009 | 113451 | 06:37:35.114 | 00:06:58.234 |
cs00000001_0010 | 85930 | 05:41:41.91 | 00:07:06.989 |
cs00000001_0011 | 96350 | 06:04:40.493 | 00:07:07.001 |
cs00000001_0012 | 234350 | 08:25:25.455 | 00:07:14.626 |
cs00000001_0013 | 20385 | 01:35:23.336 | 00:00:01.088 |
cs00000001_0014 | 20971 | 01:47:21.388 | 00:01:56.992 |
cs00000001_0015 | 32559 | 02:42:56.589 | 00:03:28.627 |
cs00000001_0016 | 153429 | 07:30:21.621 | 00:07:06.999 |
cs00000001_0017 | 17052 | 01:17:15.902 | 00:00:00.652 |
cs00000001_0018 | 22781 | 01:56:54.12 | 00:02:09.495 |
cs00000001_0019 | 5959 | 00:11:46.132 | 00:00:01.087 |
cs00000001_0020 | 45940 | 03:37:46.61 | 00:05:30.563 |
cs00000001_0021 | 90206 | 05:52:00.395 | 00:06:58.233 |
cs00000001_0022 | 87358 | 05:45:21.752 | 00:07:07.001 |
cs00000001_0023 | 92097 | 05:55:59.4 | 00:07:07.001 |
cs00000001_0024 | 393084 | 09:14:09.688 | 00:07:14.627 |
cs00000001_0025 | 85689 | 05:41:10.838 | 00:06:58.234 |
cs00000001_0026 | 155147 | 07:32:07.59 | 00:07:07.002 |
cs00000001_0027 | 49392 | 03:50:53.104 | 00:06:08.168 |
cs00000001_0028 | 66640 | 04:47:37.357 | 00:06:58.234 |
cs00000001_0029 | 7958 | 00:25:20.682 | 00:00:00.653 |
cs00000001_0030 | 102042 | 06:16:01.035 | 00:06:58.234 |
cs00000001_0031 | 76243 | 05:15:13.512 | 00:06:58.234 |
cs00000001_0032 | 86817 | 05:43:58.799 | 00:07:07.001 |
cs00000001_0033 | 47651 | 03:44:36.908 | 00:05:48.33 |
cs00000001_0034 | 19994 | 01:32:52.325 | 00:00:00.737 |
cs00000001_0035 | 136563 | 07:11:27.804 | 00:07:07.002 |
cs00000001_0036 | 115912 | 06:41:36.881 | 00:07:07.001 |
cs00000001_0037 | 25969 | 02:13:07.978 | 00:02:36.213 |
cs00000001_0038 | 244420 | 08:30:13.075 | 00:07:14.627 |
cs00000001_0039 | 130262 | 07:03:14.774 | 00:07:07.001 |
cs00000001_0040 | 116735 | 06:42:50.786 | 00:07:06.988 |
cs00000001_0041 | 61472 | 04:31:30.09 | 00:07:07.001 |
cs00000001_0042 | 650197 | 09:32:34.768 | 00:07:14.627 |
cs00000001_0043 | 830 | 00:00:50.612 | 00:00:00.814 |
cs00000001_0044 | 71477 | 05:01:47.812 | 00:06:58.234 |
cs00000001_0045 | 92823 | 05:57:44.596 | 00:06:54.272 |
cs00000001_0046 | 126251 | 06:57:56.793 | 00:06:58.233 |
cs00000001_0047 | 49518 | 03:51:26.841 | 00:06:08.004 |
cs00000001_0048 | 53596 | 04:05:29.871 | 00:07:06.989 |
cs00000001_0049 | 7458 | 00:26:40.751 | 00:00:47.596 |
cs00000001_0050 | 208836 | 08:11:54.75 | 0:07:07 |
cs00000001_0051 | 363363 | 09:10:02.246 | 0:07:07 |
cs00000001_0052 | 148889 | 07:25:35.755 | 00:07:07.001 |
cs00000001_0053 | 62321 | 04:34:11.182 | 0:07:07 |
cs00000001_0054 | 17155 | 01:26:12.309 | 00:01:39.154 |
cs00000001_0055 | 182363 | 07:55:36.678 | 0:07:07 |
cs00000001_0056 | 167863 | 07:43:34.721 | 00:07:14.627 |
cs00000001_0057 | 23743 | 02:01:53.176 | 00:02:20.403 |
cs00000001_0058 | 124395 | 06:55:00.179 | 00:07:06.988 |
cs00000001_0059 | 38685 | 03:08:57.995 | 00:04:22.604 |
cs00000001_0060 | 38775 | 03:09:17.833 | 00:04:27.063 |
cs00000001_0061 | 88457 | 05:47:58.582 | 00:07:07.002 cs00000001_0062 |
DE_authority T5(500-1K-2k). The table below contains durations for Data-export for jobs with “Default authority export profile” on 61 tenants with 1 set of data(all of the instances on each tenant).
Transaction name | DE 500, sec | DE 1K, sec | DE 2K, sec |
DE_Exporting AuthorityTransaction cs00000001_0001 | 6796 | 7119 | 11516 |
DE_Exporting AuthorityTransaction cs00000001_0002 | 5962 | 5779 | 16226 |
DE_Exporting AuthorityTransaction cs00000001_0003 | 5836 | 6311 | 21968 |
DE_Exporting AuthorityTransaction cs00000001_0004 | 5865 | 5750 | 26586 |
DE_Exporting AuthorityTransaction cs00000001_0005 | 5886 | 5745 | 28384 |
DE_Exporting AuthorityTransaction cs00000001_0006 | 5877 | 5753 | 28288 |
DE_Exporting AuthorityTransaction cs00000001_0007 | 5884 | 5737 | 33628 |
DE_Exporting AuthorityTransaction cs00000001_0008 | 5868 | 5730 | 39163 |
DE_Exporting AuthorityTransaction cs00000001_0009 | 5902 | 5751 | 44187 |
DE_Exporting AuthorityTransaction cs00000001_0010 | 5956 | 5869 | 44396 |
DE_Exporting AuthorityTransaction cs00000001_0011 | 5893 | 5787 | 49254 |
DE_Exporting AuthorityTransaction cs00000001_0012 | 5902 | 5858 | 55799 |
DE_Exporting AuthorityTransaction cs00000001_0013 | 5881 | 5756 | 54919 |
DE_Exporting AuthorityTransaction cs00000001_0014 | 5891 | 5739 | 61341 |
DE_Exporting AuthorityTransaction cs00000001_0015 | 5866 | 5755 | 66577 |
DE_Exporting AuthorityTransaction cs00000001_0016 | 5902 | 5732 | 66923 |
DE_Exporting AuthorityTransaction cs00000001_0017 | 5863 | 5928 | 70759 |
DE_Exporting AuthorityTransaction cs00000001_0018 | 5854 | 6484 | 77038 |
DE_Exporting AuthorityTransaction cs00000001_0019 | 5876 | 6102 | 83432 |
DE_Exporting AuthorityTransaction cs00000001_0020 | 5868 | 6468 | 82352 |
DE_Exporting AuthorityTransaction cs00000001_0021 | 5879 | 6918 | 90676 |
DE_Exporting AuthorityTransaction cs00000001_0022 | 5857 | 6046 | 94664 |
DE_Exporting AuthorityTransaction cs00000001_0023 | 5943 | 6604 | 85713 |
DE_Exporting AuthorityTransaction cs00000001_0024 | 5860 | 6171 | 105028 |
DE_Exporting AuthorityTransaction cs00000001_0025 | 5899 | 6299 | 105846 |
DE_Exporting AuthorityTransaction cs00000001_0026 | 5868 | 6131 | 115376 |
DE_Exporting AuthorityTransaction cs00000001_0027 | 5874 | 5944 | 111442 |
DE_Exporting AuthorityTransaction cs00000001_0028 | 5879 | 6011 | 117511 |
DE_Exporting AuthorityTransaction cs00000001_0029 | 5878 | 5792 | 127511 |
DE_Exporting AuthorityTransaction cs00000001_0030 | 5863 | 5840 | 128740 |
DE_Exporting AuthorityTransaction cs00000001_0031 | 5879 | 5820 | 128969 |
DE_Exporting AuthorityTransaction cs00000001_0032 | 5866 | 6501 | 132437 |
DE_Exporting AuthorityTransaction cs00000001_0033 | 5867 | 11505 | 138441 |
DE_Exporting AuthorityTransaction cs00000001_0034 | 5859 | 11731 | 144630 |
DE_Exporting AuthorityTransaction cs00000001_0035 | 5877 | 7080 | 144332 |
DE_Exporting AuthorityTransaction cs00000001_0036 | 5849 | 10923 | 145234 |
DE_Exporting AuthorityTransaction cs00000001_0037 | 5897 | 6041 | 151330 |
DE_Exporting AuthorityTransaction cs00000001_0038 | 5894 | 6145 | 150469 |
DE_Exporting AuthorityTransaction cs00000001_0039 | 5877 | 5739 | 155768 |
DE_Exporting AuthorityTransaction cs00000001_0040 | 5904 | 5744 | 156220 |
DE_Exporting AuthorityTransaction cs00000001_0041 | 5874 | 6202 | 156766 |
DE_Exporting AuthorityTransaction cs00000001_0042 | 5879 | 5816 | 157841 |
DE_Exporting AuthorityTransaction cs00000001_0043 | 6608 | 5880 | 158721 |
DE_Exporting AuthorityTransaction cs00000001_0044 | 5866 | 5771 | 156396 |
DE_Exporting AuthorityTransaction cs00000001_0045 | 5892 | 5854 | 157477 |
DE_Exporting AuthorityTransaction cs00000001_0046 | 5876 | 5738 | 152934 |
DE_Exporting AuthorityTransaction cs00000001_0047 | 5902 | 5750 | 153392 |
DE_Exporting AuthorityTransaction cs00000001_0048 | 5838 | 5745 | 149484 |
DE_Exporting AuthorityTransaction cs00000001_0049 | 5872 | 5860 | 149659 |
DE_Exporting AuthorityTransaction cs00000001_0050 | 5852 | 5795 | 149111 |
DE_Exporting AuthorityTransaction cs00000001_0051 | 5866 | 6178 | 143901 |
DE_Exporting AuthorityTransaction cs00000001_0052 | 5916 | 5882 | 144713 |
DE_Exporting AuthorityTransaction cs00000001_0053 | 5858 | 5747 | 139844 |
DE_Exporting AuthorityTransaction cs00000001_0054 | 5840 | 6032 | 135910 |
DE_Exporting AuthorityTransaction cs00000001_0055 | 5917 | 5903 | 136991 |
DE_Exporting AuthorityTransaction cs00000001_0056 | 5893 | 6455 | 132152 |
DE_Exporting AuthorityTransaction cs00000001_0057 | 5843 | 5769 | 126592 |
DE_Exporting AuthorityTransaction cs00000001_0058 | 5864 | 5814 | 125191 |
DE_Exporting AuthorityTransaction cs00000001_0059 | 5860 | 6123 | 120351 |
DE_Exporting AuthorityTransaction cs00000001_0060 | 5840 | 5844 | 118520 |
DE_Exporting AuthorityTransaction cs00000001_0061 | 5840 | 5890 | 114587 |
...
Service CPU Utilization
DE_instances T1(500-1K-2k). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 13%.
...
Service CPU usage during the test was less than 20%, and small spikes were on mod-data-export with CPU usage 35% at the beginning of the text.
...
DE_authority T5(500-1K-2k).
Service CPU usage during the test was stable and correlated with the number of exported records. Mod-consortia service increases CPU usage for a few seconds from 20 to 30%.
...
Memory Utilization
DE_instances T1(500-1K-2k). During testing, the memory usage does not increase
...
DE_holdings T4(Max). During testing, the memory usage(MU) does not increase. Only mod-inventory change MU from 20 to 27 and mod-data-export from 40 to 90%
...
DB CPU Utilization
DE_instances T1authority T5(500-1K-2k). Except During testing, the memory usage does not increase
...
DB CPU Utilization
DE_instances T1(500-1K-2k). Except for the database spikes that started at 18:15 and repeated every 30 minutes. CPU usage before tets was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;
...
DE_holdings T4(Max). CPU usage before tets was about 2.5%, during all Data Export holdings, CPU usage increased up to 5.3%;
...
DE_authority T5(500-1K-2k). During 3 tests CPU varied within 4-5%.
...
DB Connections
DE_instances T1(500-1K-2k). Maximal number of DB connections was 1190 for 500 DE; 1270 for 1000DE and 1340 for 2000 DE.
...
DE_holdings T4(Max). The number of DB connections at the beginning of the test was about 1500 and evenly decreased to 850 at the end of the test
...
DE_authority T5(500-1K-2k). During 3 tests NUmber of DB connections varied from 700 to 1100
...
DB Load
DE_instances T1(500-1K-2k).
...
DE_holdings T3(500-1K-2k).
...
DE_holdings T4(Max).
...
SQL queries
DE_instances T1authority T5(500-1K-2k).
...
SQL queries
DE_instances T2(T1(500-1K-2k).
...
DE_instances T2(Max).
...
DE_holdings T3(500-1K-2k).
...
DE_holdings T4(Max).
...
DE_authority T5(500-1K-2k).
...
Errors / Additional information
Number of instances and holdings on each of the 62 tenant
Expand | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Methodology/Approach
All tests were run using the JMeter script. Data has been pre-generated for all tenants
Infrastructure
PTF -environment mcpt
10 m6i11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances1 database instance, writer/readerNameMemory GIBvCPUsmax
Name Memory GIBv CPUsmax_connections
db.r6g.xlarge
32 GiB4 vCPUs27314xlarge 128 GiB; 16 vCPUMSK tenant
4 2 m5.2xlarge brokers in 2 1 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=truelog.retention.minutes=480default.replication.factor=3
Table contains modules, memory and CPU parametersThe table contains modules, memory, and CPU parameters
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
mcpt-pvt | ||||||||||
mod-consortia | 11 | 2 | 2048 | 1802 | 128 | 768 | 88 | 128 | FALSE | |
mod-organizations | 6 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE | |
mod-data-import | 7 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | FALSE | |
mod-search | 12 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | FALSE | |
mod-configuration | 6 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-permissions | 14 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | FALSE | |
mod-entities-links | 10 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | FALSE | |
mod-inventory-storage | 10 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE | |
mod-data-export-worker | 9 | 2 | 3072 | 2800 | 1024 | 2048 | 384 | 512 | FALSE | |
mod-source-record-storage | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-inventory | 10 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | FALSE | |
mod-di-converter-storage | 9 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-users | 11 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-source-record-manager | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-quick-marc | 10 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | FALSE | |
nginx-okapi | 2 | 2 | 1024 | 896 | 512 | 0 | 0 | 0 | FALSE | |
okapi-b | 1 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | FALSE | |
mod-data-export | 10 | 1 | 1024 | 896 | 1024 | 768 | 88 | 128 | FALSE |