Data Export Test Report (Mobius) part 2
Overview
This document contains the results of Data Export tests on 61 tenants (excluding the central tenant) on the MCPT environment, this report is the second part of the DE testing, the previous report is available Data Export Test Report (Mobius) | Errors / Additional information :
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Ticket:PERF-803: MOBIUS baseline tests: Data Export - part 2Closed
Summary
The tested Data-export jobs are quite light for the system and cause an increase in CPU on the services: mod-data-export, okapi and mod-inventory-storage up to 17%;
No memory leaks, memory consumption was stable during all of the tests;
Database CPU usage varied up to a maximum of 8%;
The number of failed DE jobs during concurrent testing was about 7% all with the same errors, but the duration of failed jobs was like the duration of completed, error messages are at the end of the report
The behavior of the export date was the same as in the first part of the report, with the latest processes starting with a delay. On the first tenants, where the file with instance ids was downloaded the data-export process is completed the fastest, but the data-export on the last tenants lasts the longest, as can be seen from the graphs below the tables
General metrics can be found in the tables below
Data-export duration statistics:
Data export | T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). | T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP). | T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP) | ||||||
500, sec | 1000 , sec | 2000 , sec | 500 , sec | 1000 , sec | 2000, sec | 500 , sec | 1000 , sec | 2000, sec | |
Average duration | 80 | 256 | 830 | 956 | 2110 | 4441 | 215 | 533 | 1281 |
Minimal duration | 21 | 19 | 74 | 384 | 982 | 1863 | 45 | 14 | 12 |
Maximal duration | 1028 | 360 | 987 | 1075 | 2257 | 4624 | 254 | 558 | 1374 |
75t percentile | 77 | 339 | 968 | 1029 | 2223 | 4603 | 244 | 657 | 1362 |
90t percentile | 108 | 356 | 982 | 1037 | 2241 | 4618 | 249 | 661 | 1368 |
CS42(MAX RECORDS) | 1 hrs 56 min | 1 hrs 49 min | 2hrs 18 min | 2hrs 10 min | 2hrs 30 min | 1 hrs 49 min | 32 min 7 sec | 44 min 3 sec | 55 min 31 sec |
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
| 500 records | 1000 records | 2000 records |
MARC | 10 seconds | 35 seconds | 1 minutes 45 seconds |
Holdings | 8,3 seconds | 26 seconds | 1 minutes 43 seconds |
Holdings MAX | 43 minutes 26 seconds | 41 minutes 16 seconds | 40 minutes 55 seconds |
authorityCS00 | 36 minutes 2 seconds | 35 minutes 37 seconds | 36 minutes 39 seconds |
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Tenant cs00000001_0003 | Starterted time | Finished time | Duration |
Test 3. Data-export . Max records. Default export profile | "2024-02-23T11:02:11.934+00:00" | "2024-02-23T11:29:32.025+00:00" | 00:27:20.091 |
Test 3. Data-export . 2000 records. Custom job profile | "2024-02-23T11:02:00.802+00:00" | "2024-02-23T11:03:30.687+00:00" | 00:01:29.885 |
Test 2. Data-export . Max records. Default export profile | "2024-02-23T10:26:59.739+00:00" | "2024-02-23T10:53:33.241+00:00" | 00:26:33.502 |
Test 2. Data-export . 1000 records. Custom job profile | "2024-02-23T10:26:43.425+00:00" | "2024-02-23T10:27:25.347+00:00" | 00:00:41.922 |
Test 1. Data-export . Max records. Default export profile | "2024-02-23T09:56:41.692+00:00" | "2024-02-23T10:23:11.426+00:00" | 00:26:29.734 |
Test 1. Data-export . 500 records. Custom job profile | "2024-02-23T09:56:20.276+00:00" | "2024-02-23T09:56:39.402+00:00" | 00:00:19.126 |
Recommendations & Jiras
Investigate Data-exports job that have “FAILED“ status and <Code>NoSuchKey</Code> <Message>The specified key does not exist.</Message>
Run data-import and data-export of authority records with default profiles to reproduce 5xx errors on okapi and mod-entity-link high memory utilization.
Test Results
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Default instances export profile”. Additional information about failed jobs is in the Additional Information/Error section
Tenant | 500 | 1000 | 2000 | |||||||||
Status | Started Date | Compl Date | Duration | Status | Started Date | Compl Date | Duration | Status | Started Date | Compl Date | Duration | |
cs00000001_0001 | COMPLETED | 08:41:18.306 | 08:41:42.679 | 00:00:24.373 | COMPLETED | 16:12:56.192 | 16:13:15.479 | 00:00:19.287 | COMPLETED | 20:43:32.753 | 20:44:47.318 | 00:01:14.565 |
cs00000001_0002 | COMPLETED | 08:41:25.000 | 08:41:47.885 | 00:00:22.885 | COMPLETED | 16:13:01.591 | 16:13:34.742 | 00:00:33.151 | COMPLETED | 20:43:37.932 | 20:45:58.223 | 00:02:20.291 |
cs00000001_0003 | COMPLETED | 08:41:31.646 | 08:41:53.671 | 00:00:22.025 | COMPLETED | 16:13:09.258 | 16:13:52.747 | 00:00:43.489 | COMPLETED | 20:43:46.351 | 20:47:26.980 | 00:03:40.629 |
cs00000001_0004 | COMPLETED | 08:41:36.937 | 08:41:58.192 | 00:00:21.255 | COMPLETED | 16:13:17.224 | 16:14:11.263 | 00:00:54.039 | COMPLETED | 20:43:56.513 | 20:49:11.379 | 00:05:14.866 |
cs00000001_0005 | COMPLETED | 08:41:42.698 | 08:42:03.951 | 00:00:21.253 | COMPLETED | 16:13:24.351 | 16:14:36.431 | 00:01:12.08 | COMPLETED | 20:44:03.713 | 20:50:33.894 | 00:06:30.181 |
cs00000001_0006 | COMPLETED | 08:41:47.885 | 08:42:08.914 | 00:00:21.029 | COMPLETED | 16:13:30.177 | 16:14:50.495 | 00:01:20.318 | COMPLETED | 20:44:12.660 | 20:51:55.779 | 00:07:43.119 |
cs00000001_0007 | FAIL | 08:41:53.167 | 08:42:15.317 | 00:00:22.15 | COMPLETED | 16:13:38.634 | 16:15:12.011 | 00:01:33.377 | COMPLETED | 20:44:21.387 | 20:53:27.125 | 00:09:05.738 |
cs00000001_0008 | COMPLETED | 08:41:59.692 | 08:42:25.276 | 00:00:25.584 | COMPLETED | 16:13:43.573 | 16:15:26.623 | 00:01:43.05 | COMPLETED | 20:44:29.600 | 20:54:27.147 | 00:09:57.547 |
cs00000001_0009 | COMPLETED | 08:42:04.965 | 08:42:33.644 | 00:00:28.679 | COMPLETED | 16:13:52.774 | 16:15:50.168 | 00:01:57.394 | COMPLETED | 20:44:37.979 | 20:55:25.725 | 00:10:47.746 |
cs00000001_0010 | COMPLETED | 08:42:10.940 | 08:42:42.604 | 00:00:31.664 | COMPLETED | 16:13:59.297 | 16:16:09.067 | 00:02:09.77 | COMPLETED | 20:44:46.845 | 20:56:21.134 | 00:11:34.289 |
cs00000001_0011 | FAIL | 08:42:17.625 | 08:42:50.114 | 00:00:32.489 | COMPLETED | 16:14:05.996 | 16:16:27.609 | 00:02:21.613 | COMPLETED | 20:44:54.729 | 20:57:13.442 | 00:12:18.713 |
cs00000001_0012 | COMPLETED | 08:42:25.296 | 08:43:00.791 | 00:00:35.495 | COMPLETED | 16:14:13.919 | 16:16:50.901 | 00:02:36.982 | COMPLETED | 20:45:00.316 | 20:57:40.427 | 00:12:40.111 |
cs00000001_0013 | COMPLETED | 08:42:30.529 | 08:43:09.030 | 00:00:38.501 | COMPLETED | 16:14:19.843 | 16:17:03.193 | 00:02:43.35 | COMPLETED | 20:45:09.757 | 20:58:33.223 | 00:13:23.466 |
cs00000001_0014 | COMPLETED | 08:42:36.508 | 08:43:19.801 | 00:00:43.293 | COMPLETED | 16:14:26.105 | 16:17:15.922 | 00:02:49.817 | COMPLETED | 20:45:16.057 | 20:58:58.640 | 00:13:42.583 |
cs00000001_0015 | FAIL | 08:42:43.408 | 08:43:29.083 | 00:00:45.675 | COMPLETED | 16:14:35.856 | 16:17:41.369 | 00:03:05.513 | COMPLETED | 20:45:23.453 | 20:59:23.966 | 00:14:00.513 |
cs00000001_0016 | COMPLETED | 08:42:50.139 | 08:43:35.961 | 00:00:45.822 | COMPLETED | 16:14:41.975 | 16:17:57.570 | 00:03:15.595 | COMPLETED | 20:45:34.973 | 20:59:48.715 | 00:14:13.742 |
cs00000001_0017 | COMPLETED | 08:42:56.821 | 08:43:44.103 | 00:00:47.282 | COMPLETED | 16:14:50.522 | 16:18:22.953 | 00:03:32.431 | COMPLETED | 20:45:42.889 | 21:00:13.600 | 00:14:30.711 |
cs00000001_0018 | COMPLETED | 08:43:01.562 | 08:43:55.090 | 00:00:53.528 | COMPLETED | 16:14:57.741 | 16:18:43.230 | 00:03:45.489 | COMPLETED | 20:45:50.660 | 21:00:37.526 | 00:14:46.866 |
cs00000001_0019 | FAIL | 08:43:09.901 | 08:44:00.858 | 00:00:50.957 | FAIL | 16:15:03.898 | 16:18:54.326 | 00:03:50.428 | COMPLETED | 20:45:58.742 | 21:01:00.588 | 00:15:01.846 |
cs00000001_0020 | COMPLETED | 08:43:15.211 | 08:44:12.492 | 00:00:57.281 | COMPLETED | 16:15:11.436 | 16:19:20.244 | 00:04:08.808 | COMPLETED | 20:46:07.696 | 21:01:24.518 | 00:15:16.822 |
cs00000001_0021 | COMPLETED | 08:43:24.138 | 08:44:18.231 | 00:00:54.093 | FAIL | 16:15:18.252 | 16:19:36.126 | 00:04:17.874 | COMPLETED | 20:46:17.326 | 21:01:47.076 | 00:15:29.75 |
cs00000001_0022 | COMPLETED | 08:43:29.110 | 08:44:24.802 | 00:00:55.692 | COMPLETED | 16:15:25.017 | 16:19:44.324 | 00:04:19.307 | COMPLETED | 20:46:26.716 | 21:02:09.121 | 00:15:42.405 |
cs00000001_0023 | COMPLETED | 08:43:34.515 | 08:44:37.741 | 00:01:03.226 | COMPLETED | 16:15:33.244 | 16:20:21.071 | 00:04:47.827 | COMPLETED | 20:46:37.567 | 21:02:30.237 | 00:15:52.67 |
cs00000001_0024 | FAIL | 08:43:44.103 | 08:44:44.705 | 00:01:00.602 | COMPLETED | 16:15:41.477 | 16:20:41.820 | 00:05:00.343 | COMPLETED | 20:46:41.029 | 21:02:51.477 | 00:16:10.448 |
cs00000001_0025 | COMPLETED | 08:43:49.254 | 08:44:52.111 | 00:01:02.857 | FAIL | 16:15:48.498 | 16:20:56.085 | 00:05:07.587 | COMPLETED | 20:46:48.851 | 21:02:52.107 | 00:16:03.256 |
cs00000001_0026 | FAIL | 08:43:55.090 | 08:44:58.450 | 00:01:03.36 | COMPLETED | 16:15:50.199 | 16:21:02.441 | 00:05:12.242 | COMPLETED | 20:47:00.344 | 21:03:11.997 | 00:16:11.653 |
cs00000001_0027 | COMPLETED | 08:44:00.862 | 08:45:05.258 | 00:01:04.396 | COMPLETED | 16:15:59.021 | 16:21:21.831 | 00:05:22.81 | COMPLETED | 20:47:13.135 | 21:03:30.979 | 00:16:17.844 |
cs00000001_0028 | COMPLETED | 08:44:06.633 | 08:45:18.533 | 00:01:11.9 | FAIL | 16:16:08.034 | 16:21:28.734 | 00:05:20.7 | COMPLETED | 20:47:13.641 | 21:03:31.564 | 00:16:17.923 |
cs00000001_0029 | COMPLETED | 08:44:13.255 | 08:45:25.471 | 00:01:12.216 | COMPLETED | 16:16:17.182 | 16:21:45.894 | 00:05:28.712 | COMPLETED | 20:47:26.502 | 21:03:48.240 | 00:16:21.738 |
cs00000001_0030 | COMPLETED | 08:44:19.680 | 08:45:32.942 | 00:01:13.262 | COMPLETED | 16:16:27.637 | 16:21:58.230 | 00:05:30.593 | COMPLETED | 20:47:39.164 | 21:04:04.837 | 00:16:25.673 |
cs00000001_0031 | COMPLETED | 08:44:26.226 | 08:45:40.800 | 00:01:14.574 | COMPLETED | 16:16:28.185 | 16:22:14.855 | 00:05:46.67 | COMPLETED | 20:47:40.264 | 21:04:05.925 | 00:16:25.661 |
cs00000001_0032 | COMPLETED | 08:44:31.947 | 08:45:47.403 | 00:01:15.456 | COMPLETED | 16:16:38.980 | 16:22:22.245 | 00:05:43.265 | COMPLETED | 20:47:54.143 | 21:04:21.329 | 00:16:27.186 |
cs00000001_0033 | COMPLETED | 08:44:37.766 | 08:45:54.482 | 00:01:16.716 | COMPLETED | 16:16:49.355 | 16:22:31.415 | 00:05:42.06 | COMPLETED | 20:47:54.165 | 21:04:21.329 | 00:16:27.164 |
cs00000001_0034 | FAIL | 08:44:44.734 | 08:46:02.199 | 00:01:17.465 | COMPLETED | 16:16:50.900 | 16:22:47.503 | 00:05:56.603 | COMPLETED | 20:48:11.145 | 21:04:37.074 | 00:16:25.929 |
cs00000001_0035 | COMPLETED | 08:44:51.413 | 08:46:10.557 | 00:01:19.144 | COMPLETED | 16:17:02.150 | 16:22:48.836 | 00:05:46.686 | COMPLETED | 20:48:10.524 | 21:04:35.934 | 00:16:25.41 |
cs00000001_0036 | FAIL | 08:44:58.453 | 08:46:18.757 | 00:01:20.304 | COMPLETED | 16:17:03.192 | 16:23:02.021 | 00:05:58.829 | COMPLETED | 20:48:27.791 | 21:04:49.677 | 00:16:21.886 |
cs00000001_0037 | COMPLETED | 08:45:04.009 | 08:46:18.767 | 00:01:14.758 | FAIL | 16:17:14.904 | 16:23:15.294 | 00:06:00.39 | COMPLETED | 20:48:28.325 | 21:04:50.943 | 00:16:22.618 |
cs00000001_0038 | COMPLETED | 08:45:10.759 | 08:46:26.979 | 00:01:16.22 | COMPLETED | 16:17:15.921 | 16:23:15.297 | 00:05:59.376 | COMPLETED | 20:48:47.245 | 21:05:02.441 | 00:16:15.196 |
cs00000001_0039 | COMPLETED | 08:45:17.825 | 08:46:34.947 | 00:01:17.122 | COMPLETED | 16:17:27.367 | 16:23:27.219 | 00:05:59.852 | COMPLETED | 20:48:47.272 | 21:05:03.655 | 00:16:16.383 |
cs00000001_0040 | COMPLETED | 08:45:24.020 | 08:46:43.387 | 00:01:19.367 | COMPLETED | 16:17:37.584 | 16:23:27.815 | 00:05:50.231 | COMPLETED | 20:49:07.141 | 21:05:16.105 | 00:16:08.964 |
cs00000001_0041 | COMPLETED | 08:45:25.495 | 08:46:51.713 | 00:01:26.218 | COMPLETED | 16:17:40.856 | 16:23:38.234 | 00:05:57.378 | COMPLETED | 20:49:07.932 | 21:05:16.106 | 00:16:08.174 |
cs00000001_0042 | COMPLETED | 08:47:41.141 | 10:44:05.834 | 01:56:24.693 | COMPLETED | 16:24:30.590 | 18:13:37.660 | 01:49:07.07 | COMPLETED | 21:06:03.875 | 00:24:43.478 | 02:18:39.603 |
cs00000001_0043 | COMPLETED | 08:45:39.452 | 08:46:59.947 | 00:01:20.495 | COMPLETED | 16:17:41.400 | 16:23:02.020 | 00:05:20.62 | COMPLETED | 20:48:29.519 | 20:56:20.036 | 00:07:50.517 |
cs00000001_0044 | COMPLETED | 08:45:46.640 | 08:47:06.799 | 00:01:20.159 | COMPLETED | 16:18:06.311 | 16:23:49.211 | 00:05:42.9 | COMPLETED | 20:49:28.535 | 21:05:26.314 | 00:15:57.779 |
cs00000001_0045 | FAIL | 08:45:54.498 | 08:47:12.755 | 00:01:18.257 | COMPLETED | 16:18:09.871 | 16:23:58.163 | 00:05:48.292 | COMPLETED | 20:49:48.172 | 21:05:34.512 | 00:15:46.34 |
cs00000001_0046 | COMPLETED | 08:46:02.202 | 08:47:18.442 | 00:01:16.24 | COMPLETED | 16:18:10.926 | 16:23:58.164 | 00:05:47.238 | COMPLETED | 20:49:48.196 | 21:05:36.270 | 00:15:48.074 |
cs00000001_0047 | COMPLETED | 08:46:02.225 | 08:47:18.442 | 00:01:16.217 | FAIL | 16:18:24.123 | 16:24:01.782 | 00:05:37.659 | FAIL | 20:50:09.653 | 21:05:38.160 | 00:15:28.507 |
cs00000001_0048 | COMPLETED | 08:46:10.574 | 08:47:22.681 | 00:01:12.107 | COMPLETED | 16:18:26.401 | 16:24:06.366 | 00:05:39.965 | COMPLETED | 20:50:09.701 | 21:05:43.870 | 00:15:34.169 |
cs00000001_0049 | COMPLETED | 08:46:18.761 | 08:47:29.539 | 00:01:10.778 | COMPLETED | 16:18:40.848 | 16:24:13.867 | 00:05:33.019 | COMPLETED | 20:50:09.679 | 21:05:42.628 | 00:15:32.949 |
cs00000001_0050 | COMPLETED | 08:46:26.232 | 08:47:32.182 | 00:01:05.95 | COMPLETED | 16:18:42.610 | 16:24:13.868 | 00:05:31.258 | FAIL | 20:50:33.358 | 21:05:49.392 | 00:15:16.034 |
cs00000001_0051 | COMPLETED | 08:46:26.260 | 08:47:34.374 | 00:01:08.114 | COMPLETED | 16:18:58.924 | 16:24:19.617 | 00:05:20.693 | COMPLETED | 20:50:33.400 | 21:05:50.748 | 00:15:17.348 |
cs00000001_0052 | COMPLETED | 08:46:34.943 | 08:47:34.375 | 00:00:59.432 | COMPLETED | 16:18:58.900 | 16:24:13.870 | 00:05:14.97 | COMPLETED | 20:50:33.380 | 21:05:50.748 | 00:15:17.368 |
cs00000001_0053 | COMPLETED | 08:46:43.383 | 08:47:36.503 | 00:00:53.12 | COMPLETED | 16:19:00.681 | 16:24:19.617 | 00:05:18.936 | COMPLETED | 20:50:58.398 | 21:05:56.115 | 00:14:57.717 |
cs00000001_0054 | COMPLETED | 08:46:51.739 | 08:47:37.351 | 00:00:45.612 | COMPLETED | 16:19:17.924 | 16:24:19.618 | 00:05:01.694 | COMPLETED | 20:50:58.420 | 21:05:56.115 | 00:14:57.695 |
cs00000001_0055 | COMPLETED | 08:46:02.225 | 08:47:18.442 | 00:02:11.391 | COMPLETED | 16:19:17.967 | 16:24:19.621 | 00:05:01.654 | COMPLETED | 20:50:57.826 | 21:05:51.800 | 00:14:53.974 |
cs00000001_0056 | COMPLETED | 08:46:10.574 | 08:47:22.681 | 00:01:57.137 | COMPLETED | 16:19:36.191 | 16:24:22.672 | 00:04:46.481 | COMPLETED | 20:51:25.458 | 21:05:58.811 | 00:14:33.353 |
cs00000001_0057 | COMPLETED | 08:46:18.761 | 08:47:29.539 | 00:01:57.114 | COMPLETED | 16:19:37.364 | 16:24:25.511 | 00:04:48.147 | COMPLETED | 20:51:24.836 | 21:05:56.115 | 00:14:31.279 |
cs00000001_0058 | COMPLETED | 08:46:26.232 | 08:47:32.182 | 00:01:59.658 | COMPLETED | 16:19:38.715 | 16:24:25.510 | 00:04:46.795 | COMPLETED | 20:51:25.482 | 21:05:59.833 | 00:14:34.351 |
cs00000001_0059 | COMPLETED | 08:46:26.260 | 08:47:34.374 | 00:01:59.169 | COMPLETED | 16:19:57.118 | 16:24:27.002 | 00:04:29.884 | COMPLETED | 20:51:26.048 | 21:05:59.833 | 00:14:33.785 |
cs00000001_0060 | COMPLETED | 08:46:34.943 | 08:47:34.375 | 00:01:48.399 | COMPLETED | 16:19:57.176 | 16:24:27.003 | 00:04:29.827 | COMPLETED | 20:51:54.664 | 21:05:59.837 | 00:14:05.173 |
cs00000001_0061 | COMPLETED | 08:46:43.383 | 08:47:36.503 | 00:01:44.223 | COMPLETED | 16:19:58.846 | 16:24:27.006 | 00:04:28.16 | COMPLETED | 20:51:54.644 | 21:05:59.833 | 00:14:05.189 |
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP). T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Custom job profile”
Tenant | 500 | 1000 | 2000 | ||||||
Started Date | Compl Date | Duration | Started Date | Compl Date | Duration | Started Date | Compl Date | Duration | |
cs00000001_0001 | 01:53:31.277 | 01:59:56.252 | 00:06:24.975 | 05:52:37.425 | 06:17:17.609 | 00:24:40.184 | 08:33:24.320 | 09:35:48.840 | 01:02:24.52 |
cs00000001_0002 | 01:53:33.676 | 02:03:51.730 | 00:10:18.054 | 05:52:39.672 | 06:20:28.788 | 00:27:49.116 | 08:33:30.420 | 09:40:39.008 | 01:07:08.588 |
cs00000001_0003 | 01:53:37.776 | 02:05:02.656 | 00:09:50.057 | 05:52:43.605 | 06:21:52.494 | 00:29:08.889 | 08:33:26.759 | 09:36:49.987 | 01:03:23.228 |
cs00000001_0004 | 01:53:37.776 | 02:04:40.883 | 00:11:03.107 | 05:52:43.606 | 06:21:31.649 | 00:28:48.043 | 08:33:39.397 | 09:43:03.062 | 01:09:23.665 |
cs00000001_0005 | 01:53:45.310 | 02:05:54.381 | 00:12:09.071 | 05:52:50.459 | 06:23:25.395 | 00:30:34.936 | 08:33:50.635 | 09:44:29.453 | 01:10:38.818 |
cs00000001_0006 | 01:53:47.612 | 02:07:23.082 | 00:13:35.47 | 05:52:50.482 | 06:23:27.473 | 00:30:36.991 | 08:33:42.743 | 09:43:27.986 | 01:09:45.243 |
cs00000001_0007 | 01:53:45.333 | 02:06:30.872 | 00:12:45.539 | 05:52:50.438 | 06:23:25.394 | 00:30:34.956 | 08:33:50.614 | 09:44:11.342 | 01:10:20.728 |
cs00000001_0008 | 01:53:47.611 | 02:07:08.968 | 00:13:21.357 | 05:52:50.505 | 06:23:27.473 | 00:30:36.968 | 08:33:39.420 | 09:43:19.105 | 01:09:39.685 |
cs00000001_0009 | 01:53:43.609 | 02:05:54.381 | 00:12:10.772 | 05:53:28.828 | 06:28:52.008 | 00:35:23.18 | 08:33:39.374 | 09:43:03.062 | 01:09:23.688 |
cs00000001_0010 | 01:54:16.112 | 02:10:28.535 | 00:16:09.214 | 05:53:30.440 | 06:29:11.485 | 00:35:41.045 | 08:33:50.656 | 09:44:29.444 | 01:10:38.788 |
cs00000001_0011 | 01:54:17.737 | 02:10:32.690 | 00:16:09.19 | 05:53:30.467 | 06:29:11.488 | 00:35:41.021 | 08:33:39.346 | 09:43:00.800 | 01:09:21.454 |
cs00000001_0012 | 05:53:37.572 | 06:29:33.067 | 00:15:19.326 | 05:53:37.572 | 06:29:33.067 | 00:16:22.458 | 08:33:42.790 | 09:43:44.708 | 01:10:01.918 |
cs00000001_0013 | 01:54:33.625 | 02:11:21.309 | 00:16:47.684 | 05:53:37.528 | 06:29:33.066 | 00:35:55.538 | 08:33:42.766 | 09:43:30.038 | 01:09:47.272 |
cs00000001_0014 | 01:54:33.648 | 02:11:27.235 | 00:16:35.552 | 01:54:33.648 | 06:29:33.067 | 00:35:55.495 | 08:35:18.364 | 09:50:42.159 | 01:15:23.795 |
cs00000001_0015 | 01:54:33.673 | 02:11:27.235 | 00:16:53.562 | 05:53:47.647 | 06:30:32.147 | 00:36:44.5 | 08:35:18.411 | 09:50:52.684 | 01:15:34.273 |
cs00000001_0016 | 01:54:33.604 | 02:11:07.953 | 00:16:34.349 | 05:53:47.672 | 06:30:32.147 | 00:36:44.475 | 08:35:18.387 | 09:50:42.159 | 01:15:23.772 |
cs00000001_0017 | 01:54:33.580 | 02:10:56.013 | 00:16:22.433 | 05:53:55.408 | 06:30:44.260 | 00:36:48.852 | 08:35:28.446 | 09:51:17.132 | 01:15:48.686 |
cs00000001_0018 | 01:54:37.112 | 02:12:07.180 | 00:16:33.535 | 05:53:55.297 | 06:30:44.262 | 00:36:48.965 | 08:35:40.571 | 09:51:29.248 | 01:15:48.677 |
cs00000001_0019 | 01:54:37.089 | 02:12:07.178 | 00:17:30.089 | 05:53:55.355 | 06:30:44.265 | 00:36:48.91 | 08:35:48.768 | 09:52:19.697 | 01:16:30.929 |
cs00000001_0020 | 01:55:03.483 | 02:12:13.422 | 00:17:09.939 | 05:54:05.743 | 06:30:48.303 | 00:36:42.56 | 08:35:57.104 | 09:52:19.697 | 01:16:22.593 |
cs00000001_0021 | 01:54:58.894 | 02:12:13.420 | 00:16:33.557 | 05:54:05.719 | 06:30:48.302 | 00:36:42.583 | 08:35:57.130 | 09:52:19.698 | 01:16:22.568 |
cs00000001_0022 | 01:55:03.484 | 02:12:13.423 | 00:17:09.939 | 05:54:05.962 | 06:31:01.285 | 00:36:55.323 | 08:36:08.686 | 09:52:51.817 | 01:16:43.131 |
cs00000001_0023 | 01:54:55.311 | 02:12:07.179 | 00:16:31.714 | 05:54:06.015 | 06:31:43.069 | 00:37:37.054 | 08:36:00.345 | 09:52:43.434 | 01:16:43.089 |
cs00000001_0024 | 01:54:55.403 | 02:12:11.220 | 00:17:10.792 | 05:54:05.887 | 06:30:52.466 | 00:36:46.579 | 08:36:03.614 | 09:52:43.435 | 01:16:39.821 |
cs00000001_0025 | 01:54:55.355 | 02:12:07.181 | 00:17:11.826 | 05:54:05.908 | 06:30:55.196 | 00:36:49.288 | 08:36:08.710 | 09:52:51.817 | 01:16:43.107 |
cs00000001_0026 | 01:54:58.840 | 02:12:11.223 | 00:17:12.383 | 05:54:05.796 | 06:30:48.303 | 00:36:42.507 | 08:36:03.589 | 09:52:43.434 | 01:16:39.845 |
cs00000001_0027 | 01:55:03.551 | 02:12:13.424 | 00:17:09.873 | 05:54:21.230 | 06:31:45.221 | 00:37:23.991 | 08:36:08.733 | 09:52:51.817 | 01:16:43.084 |
cs00000001_0028 | 01:55:03.485 | 02:12:13.423 | 00:17:09.938 | 05:54:21.261 | 06:31:53.248 | 00:37:31.987 | 08:36:14.068 | 09:53:11.187 | 01:16:57.119 |
cs00000001_0029 | 01:55:11.134 | 02:13:06.454 | 00:17:00.008 | 05:54:21.093 | 06:31:43.069 | 00:37:21.976 | 08:36:14.092 | 09:53:11.187 | 01:16:57.095 |
cs00000001_0030 | 01:55:11.132 | 02:13:06.454 | 00:17:55.322 | 05:54:21.143 | 06:31:43.070 | 00:37:21.927 | 08:36:21.700 | 09:53:21.679 | 01:16:59.979 |
cs00000001_0031 | 01:55:03.491 | 02:12:13.423 | 00:17:09.932 | 05:54:39.541 | 06:31:53.248 | 00:37:13.707 | 08:36:21.700 | 09:53:21.678 | 01:16:59.978 |
cs00000001_0032 | 01:55:03.492 | 02:12:13.423 | 00:17:01.96 | 05:54:41.152 | 06:31:53.248 | 00:37:12.096 | 08:36:21.700 | 09:53:19.725 | 01:16:58.025 |
cs00000001_0033 | 01:55:11.132 | 02:12:58.074 | 00:17:46.942 | 05:54:41.181 | 06:31:55.429 | 00:37:14.248 | 08:36:21.699 | 09:53:17.640 | 01:16:55.941 |
cs00000001_0034 | 01:55:11.132 | 02:12:46.054 | 00:17:34.922 | 05:54:41.238 | 06:31:55.429 | 00:37:14.191 | 08:36:21.700 | 09:53:19.725 | 01:16:58.025 |
cs00000001_0035 | 01:55:22.658 | 02:13:06.454 | 00:17:17.699 | 05:54:44.175 | 06:31:55.429 | 00:37:11.254 | 08:36:21.732 | 09:53:26.178 | 01:17:04.446 |
cs00000001_0036 | 01:56:00.037 | 02:13:08.248 | 00:17:25.928 | 05:53:47.672 | 06:30:32.147 | 00:36:44.475 | 08:36:21.721 | 09:53:21.679 | 01:16:59.958 |
cs00000001_0037 | 01:55:59.999 | 02:13:06.457 | 00:17:06.458 | 05:55:00.509 | 06:32:04.413 | 00:37:03.904 | 08:36:21.721 | 09:53:21.678 | 01:16:59.957 |
cs00000001_0038 | 01:56:10.856 | 02:13:33.865 | 00:17:23.009 | 05:55:00.456 | 06:32:04.413 | 00:37:03.957 | 08:36:21.700 | 09:53:21.678 | 01:16:59.978 |
cs00000001_0039 | 01:56:00.117 | 02:13:08.249 | 00:17:08.132 | 05:55:00.625 | 06:32:24.276 | 00:37:23.651 | 08:36:39.711 | 09:53:26.180 | 01:16:46.469 |
cs00000001_0040 | 01:56:00.095 | 02:13:08.247 | 00:16:41.703 | 05:56:23.855 | 06:32:51.378 | 00:36:27.523 | 08:36:39.666 | 09:53:26.179 | 01:16:46.513 |
cs00000001_0041 | 01:56:00.140 | 02:13:08.248 | 00:17:08.108 | 05:55:07.123 | 06:32:26.144 | 00:37:19.021 | 08:36:39.691 | 09:53:26.179 | 01:16:46.488 |
cs00000001_0042 | 01:53:47.606 | 05:54:19.529 | 02:10:05.948 | 05:52:52.772 | 08:23:32.266 | 02:30:39.494 | 09:53:47.797 | 11:43:11.528 | 01:49:23.731 |
cs00000001_0043 | 01:56:02.130 | 02:13:08.248 | 00:17:06.118 | 05:55:00.605 | 06:27:50.787 | 00:32:50.182 | 08:35:14.946 | 09:06:18.066 | 00:31:03.12 |
cs00000001_0044 | 01:56:10.775 | 02:13:08.249 | 00:16:33.158 | 05:54:21.230 | 06:31:45.221 | 00:37:23.991 | 08:36:52.312 | 09:53:26.180 | 01:16:33.868 |
cs00000001_0045 | 01:56:10.830 | 02:13:22.782 | 00:17:11.952 | 05:55:59.256 | 06:32:33.746 | 00:36:34.49 | 08:36:50.687 | 09:53:26.180 | 01:16:35.493 |
cs00000001_0046 | 01:56:10.681 | 02:13:08.248 | 00:16:41.273 | 05:55:58.523 | 06:32:33.745 | 00:36:35.222 | 08:36:52.358 | 09:53:35.261 | 01:16:42.903 |
cs00000001_0047 | 01:56:10.725 | 02:13:08.249 | 00:16:26.849 | 05:56:19.921 | 06:32:51.377 | 00:36:31.456 | 08:36:52.336 | 09:53:30.825 | 01:16:38.489 |
cs00000001_0048 | 01:56:41.745 | 02:13:33.868 | 00:16:52.123 | 05:56:19.654 | 06:32:37.655 | 00:36:18.001 | 08:37:14.473 | 09:53:35.261 | 01:16:20.788 |
cs00000001_0049 | 01:56:41.686 | 02:13:33.866 | 00:16:52.18 | 05:56:19.715 | 06:32:37.655 | 00:36:17.94 | 08:37:14.691 | 09:53:37.363 | 01:16:22.672 |
cs00000001_0050 | 01:57:23.999 | 02:13:33.874 | 00:16:09.875 | 05:56:19.814 | 06:32:37.667 | 00:36:17.853 | 08:37:14.643 | 09:53:37.374 | 01:16:22.731 |
cs00000001_0051 | 01:57:23.711 | 02:13:33.865 | 00:16:10.154 | 05:56:19.867 | 06:32:43.848 | 00:36:23.981 | 08:37:14.526 | 09:53:37.363 | 01:16:22.837 |
cs00000001_0052 | 01:57:23.767 | 02:13:33.866 | 00:16:10.099 | 05:56:19.921 | 06:32:51.377 | 00:36:31.456 | 08:37:24.892 | 09:53:43.623 | 01:16:18.731 |
cs00000001_0053 | 01:57:23.881 | 02:13:33.867 | 00:16:09.986 | 05:56:23.855 | 06:32:51.378 | 00:36:27.523 | 08:37:24.690 | 09:53:37.364 | 01:16:12.674 |
cs00000001_0054 | 01:57:24.053 | 02:13:33.867 | 00:16:09.814 | 05:56:23.805 | 06:32:51.377 | 00:36:27.572 | 08:37:37.098 | 09:53:43.622 | 01:16:06.524 |
cs00000001_0055 | 01:57:23.825 | 02:13:33.868 | 00:16:10.043 | 05:56:51.631 | 06:32:51.378 | 00:35:59.747 | 08:37:51.332 | 09:53:43.622 | 01:15:52.29 |
cs00000001_0056 | 01:57:34.036 | 02:13:33.866 | 00:15:59.83 | 05:57:42.946 | 06:32:51.378 | 00:35:08.432 | 08:37:51.432 | 09:53:43.622 | 01:15:52.19 |
cs00000001_0057 | 01:57:34.101 | 02:13:33.868 | 00:15:59.767 | 05:57:44.971 | 06:32:51.378 | 00:35:06.407 | 08:37:51.277 | 09:53:43.622 | 01:15:52.345 |
cs00000001_0058 | 01:57:36.251 | 02:13:33.868 | 00:15:57.617 | 05:57:44.919 | 06:32:51.378 | 00:35:06.459 | 08:37:53.578 | 09:53:43.622 | 01:15:50.044 |
cs00000001_0059 | 01:57:36.197 | 02:13:33.867 | 00:15:57.67 | 05:58:06.474 | 06:32:51.379 | 00:34:44.905 | 08:37:53.523 | 09:53:43.623 | 01:15:50.1 |
cs00000001_0060 | 01:57:42.456 | 02:13:34.509 | 00:15:52.053 | 05:58:06.388 | 06:32:51.378 | 00:34:44.99 | 08:38:22.838 | 09:53:43.623 | 01:15:20.785 |
cs00000001_0061 | 01:57:42.399 | 02:13:33.868 | 00:15:51.469 | 05:58:06.545 | 06:32:51.924 | 00:34:45.379 | 08:38:22.931 | 09:53:43.626 | 01:15:20.695 |
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
Tenant | Started Date | Compl Date | Duration | Started Date | Compl Date | Duration | Started Date | Compl Date | Duration |
cs00000001 | 14:30:48.756 | 15:02:56.495 | 00:32:07.739 | 15:25:21.738 | 16:09:25.170 | 00:44:03.432 | 17:29:00.215 | 18:24:32.154 | 00:55:31.939 |
cs00000001_0001 | 14:24:42.318 | 14:24:42.318 | 00:00:45.211 | 15:25:15.665 | 15:31:19.954 | 00:06:04.289 | 17:28:49.284 | 17:44:36.536 | 00:15:47.252 |
cs00000001_0002 | 14:24:43.082 | 14:25:31.223 | 00:00:48.141 | 15:25:17.867 | 15:32:31.980 | 00:07:14.113 | 17:28:50.497 | 17:45:49.205 | 00:16:58.708 |
cs00000001_0003 | 14:24:45.343 | 14:25:45.888 | 00:01:00.545 | 15:25:19.490 | 15:32:56.805 | 00:07:37.315 | 17:28:52.298 | 17:46:17.206 | 00:17:24.908 |
cs00000001_0004 | 14:24:47.523 | 14:26:21.848 | 00:01:34.325 | 15:25:19.468 | 15:32:53.154 | 00:07:33.686 | 17:28:55.078 | 17:47:24.722 | 00:18:29.644 |
cs00000001_0005 | 14:24:47.497 | 14:26:03.472 | 00:01:15.975 | 15:25:26.453 | 15:34:08.759 | 00:08:42.306 | 17:28:55.055 | 17:47:08.115 | 00:18:13.06 |
cs00000001_0006 | 14:24:51.872 | 14:26:58.001 | 00:02:06.129 | 15:25:34.475 | 15:35:08.741 | 00:09:34.266 | 17:29:10.529 | 17:49:43.272 | 00:20:32.743 |
cs00000001_0007 | 14:24:51.900 | 14:27:00.778 | 00:02:08.878 | 15:25:39.336 | 15:35:31.356 | 00:09:52.02 | 17:29:10.505 | 17:49:42.671 | 00:20:32.166 |
cs00000001_0008 | 14:24:55.497 | 14:27:40.752 | 00:02:45.255 | 15:25:39.358 | 15:35:36.215 | 00:09:56.857 | 17:29:15.627 | 17:50:15.416 | 00:20:59.789 |
cs00000001_0009 | 14:24:59.483 | 14:27:54.691 | 00:02:55.208 | 15:25:39.876 | 15:35:43.352 | 00:10:03.476 | 17:29:17.275 | 17:50:33.670 | 00:21:16.395 |
cs00000001_0010 | 14:24:59.508 | 14:28:06.621 | 00:03:07.113 | 15:25:41.078 | 15:35:43.957 | 00:10:02.879 | 17:29:22.272 | 17:50:51.494 | 00:21:29.222 |
cs00000001_0011 | 14:25:00.195 | 14:28:14.477 | 00:03:14.282 | 15:25:41.086 | 15:35:49.502 | 00:10:08.416 | 17:29:22.244 | 17:50:48.353 | 00:21:26.109 |
cs00000001_0012 | 14:25:04.852 | 14:28:20.675 | 00:03:15.823 | 15:25:47.008 | 15:36:04.014 | 00:10:17.006 | 17:29:28.675 | 17:51:20.287 | 00:21:51.612 |
cs00000001_0013 | 14:25:05.432 | 14:28:35.978 | 00:03:30.546 | 15:25:46.416 | 15:36:03.250 | 00:10:16.834 | 17:29:29.703 | 17:51:24.819 | 00:21:55.116 |
cs00000001_0014 | 14:25:07.135 | 14:28:41.482 | 00:03:34.347 | 15:25:50.212 | 15:36:16.623 | 00:10:26.411 | 17:29:29.732 | 17:51:24.821 | 00:21:55.089 |
cs00000001_0015 | 14:25:11.505 | 14:28:49.677 | 00:03:38.172 | 15:25:50.239 | 15:36:17.917 | 00:10:27.678 | 17:29:38.216 | 17:51:53.405 | 00:22:15.189 |
cs00000001_0016 | 14:25:11.481 | 14:28:47.521 | 00:03:36.04 | 15:25:55.589 | 15:36:33.118 | 00:10:37.529 | 17:29:37.763 | 17:51:49.102 | 00:22:11.339 |
cs00000001_0017 | 14:25:13.285 | 14:29:03.071 | 00:03:49.786 | 15:25:56.102 | 15:36:34.963 | 00:10:38.861 | 17:29:37.763 | 17:51:49.103 | 00:22:11.34 |
cs00000001_0018 | 14:25:18.535 | 14:29:07.800 | 00:03:49.265 | 15:25:58.614 | 15:36:44.878 | 00:10:46.264 | 17:29:47.800 | 17:52:17.763 | 00:22:29.963 |
cs00000001_0019 | 14:25:20.256 | 14:29:13.686 | 00:03:53.43 | 15:26:06.052 | 15:36:52.652 | 00:10:46.6 | 17:29:48.419 | 17:52:20.725 | 00:22:32.306 |
cs00000001_0020 | 14:25:20.281 | 14:29:21.852 | 00:04:01.571 | 15:26:06.028 | 15:36:51.427 | 00:10:45.399 | 17:29:48.396 | 17:52:17.762 | 00:22:29.366 |
cs00000001_0021 | 14:25:28.404 | 14:29:28.699 | 00:04:00.295 | 15:26:07.660 | 15:36:57.123 | 00:10:49.463 | 17:29:48.995 | 17:52:23.675 | 00:22:34.68 |
cs00000001_0022 | 14:25:30.184 | 14:29:30.962 | 00:04:00.778 | 15:26:09.197 | 15:37:08.209 | 00:10:59.012 | 17:29:49.018 | 17:52:23.676 | 00:22:34.658 |
cs00000001_0023 | 14:25:30.184 | 14:29:31.495 | 00:04:01.311 | 15:26:11.778 | 15:37:10.213 | 00:10:58.435 | 17:30:01.919 | 17:52:42.840 | 00:22:40.921 |
cs00000001_0024 | 14:25:30.184 | 14:29:30.435 | 00:04:00.251 | 15:26:19.761 | 15:37:13.777 | 00:10:54.016 | 17:30:01.941 | 17:52:43.450 | 00:22:41.509 |
cs00000001_0025 | 14:25:30.184 | 14:29:31.495 | 00:04:01.311 | 15:26:19.783 | 15:37:16.180 | 00:10:56.397 | 17:30:03.091 | 17:52:49.336 | 00:22:46.245 |
cs00000001_0026 | 14:25:31.763 | 14:29:44.320 | 00:04:12.557 | 15:26:20.346 | 15:37:16.180 | 00:10:55.834 | 17:30:03.112 | 17:52:50.019 | 00:22:46.907 |
cs00000001_0027 | 14:25:36.718 | 14:29:47.116 | 00:04:10.398 | 15:26:23.529 | 15:37:29.781 | 00:11:06.252 | 17:30:17.486 | 17:53:04.582 | 00:22:47.096 |
cs00000001_0028 | 14:25:43.614 | 14:29:47.648 | 00:04:04.034 | 15:26:25.997 | 15:37:31.677 | 00:11:05.68 | 17:30:17.536 | 17:53:04.583 | 00:22:47.047 |
cs00000001_0029 | 14:25:43.635 | 14:29:48.171 | 00:04:04.536 | 15:26:36.318 | 15:37:33.537 | 00:10:57.219 | 17:30:17.587 | 17:53:05.213 | 00:22:47.626 |
cs00000001_0030 | 14:25:43.589 | 14:29:47.648 | 00:04:04.059 | 15:26:37.039 | 15:37:38.474 | 00:11:01.435 | 17:30:19.321 | 17:53:11.505 | 00:22:52.184 |
cs00000001_0031 | 14:25:43.567 | 14:29:47.645 | 00:04:04.078 | 15:26:36.341 | 15:37:35.539 | 00:10:59.198 | 17:30:28.307 | 17:53:16.349 | 00:22:48.042 |
cs00000001_0032 | 14:25:44.235 | 14:29:49.921 | 00:04:05.686 | 15:26:36.992 | 15:37:36.079 | 00:10:59.087 | 17:30:33.304 | 17:53:16.349 | 00:22:43.045 |
cs00000001_0033 | 14:25:46.439 | 14:30:01.215 | 00:04:14.776 | 15:26:37.017 | 15:37:36.079 | 00:10:59.062 | 17:30:33.326 | 17:53:24.641 | 00:22:51.315 |
cs00000001_0034 | 14:25:53.953 | 14:30:02.375 | 00:04:08.422 | 15:26:40.769 | 15:37:48.169 | 00:11:07.4 | 17:30:33.350 | 17:53:25.842 | 00:22:52.492 |
cs00000001_0035 | 14:26:00.662 | 14:30:02.888 | 00:04:02.226 | 15:26:40.793 | 15:37:48.910 | 00:11:08.117 | 17:30:35.980 | 17:53:30.121 | 00:22:54.141 |
cs00000001_0036 | 14:26:00.600 | 14:30:02.888 | 00:04:02.288 | 15:26:48.214 | 15:37:48.911 | 00:11:00.697 | 17:30:36.034 | 17:53:30.122 | 00:22:54.088 |
cs00000001_0037 | 14:26:00.627 | 14:30:02.887 | 00:04:02.26 | 15:26:53.216 | 15:37:49.487 | 00:10:56.271 | 17:30:40.561 | 17:53:31.268 | 00:22:50.707 |
cs00000001_0038 | 14:26:00.707 | 14:30:04.701 | 00:04:03.994 | 15:26:56.783 | 15:37:50.130 | 00:10:53.347 | 17:30:50.085 | 17:53:32.414 | 00:22:42.329 |
cs00000001_0039 | 14:26:00.685 | 14:30:03.360 | 00:04:02.675 | 15:26:57.421 | 15:37:52.235 | 00:10:54.814 | 17:30:50.029 | 17:53:32.414 | 00:22:42.385 |
cs00000001_0040 | 14:26:02.068 | 14:30:10.702 | 00:04:08.634 | 15:26:57.458 | 15:37:59.809 | 00:11:02.351 | 17:30:50.106 | 17:53:33.585 | 00:22:43.479 |
cs00000001_0041 | 14:26:03.496 | 14:30:13.924 | 00:04:10.428 | 15:26:09.179 | 15:26:24.068 | 00:00:14.889 | 17:30:50.051 | 17:53:32.414 | 00:22:42.363 |
cs00000001_0042 | 14:25:44.235 | 14:29:49.921 | 00:04:05.686 | 15:26:46.587 | 15:36:44.876 | 00:09:58.289 | 17:29:39.707 | 17:29:51.801 | 00:00:12.094 |
cs00000001_0043 | 14:26:11.198 | 14:30:14.616 | 00:04:03.418 | 15:27:01.669 | 15:38:02.854 | 00:11:01.185 | 17:30:10.228 | 17:39:28.947 | 00:09:18.719 |
cs00000001_0044 | 14:26:15.107 | 14:30:14.619 | 00:03:59.512 | 15:27:02.233 | 15:38:02.854 | 00:11:00.621 | 17:31:04.416 | 17:53:43.867 | 00:22:39.451 |
cs00000001_0045 | 14:26:18.467 | 14:30:14.618 | 00:03:56.151 | 15:27:03.317 | 15:38:03.511 | 00:11:00.194 | 17:31:15.533 | 17:53:45.158 | 00:22:29.625 |
cs00000001_0046 | 14:26:20.713 | 14:30:14.619 | 00:03:53.906 | 15:27:13.457 | 15:38:03.511 | 00:10:50.054 | 17:31:15.507 | 17:53:45.157 | 00:22:29.65 |
cs00000001_0047 | 14:26:21.798 | 14:30:14.620 | 00:03:52.822 | 15:27:15.707 | 15:38:03.511 | 00:10:47.804 | 17:31:15.481 | 17:53:45.157 | 00:22:29.676 |
cs00000001_0048 | 14:26:21.241 | 14:30:14.620 | 00:03:53.379 | 15:27:18.004 | 15:38:03.512 | 00:10:45.508 | 17:31:15.559 | 17:53:49.697 | 00:22:34.138 |
cs00000001_0049 | 14:26:21.821 | 14:30:15.961 | 00:03:54.14 | 15:27:20.942 | 15:38:03.522 | 00:10:42.58 | 17:31:20.125 | 17:53:53.650 | 00:22:33.525 |
cs00000001_0050 | 14:26:21.846 | 14:30:22.124 | 00:04:00.278 | 14:26:23.532 | 14:30:22.113 | 00:03:58.581 | 17:31:20.147 | 17:53:54.706 | 00:22:34.559 |
cs00000001_0051 | 14:26:03.496 | 14:26:01.226 | 00:04:10.853 | 14:26:24.155 | 14:30:22.114 | 00:03:57.959 | 17:31:24.154 | 17:53:55.404 | 00:22:31.25 |
cs00000001_0052 | 14:25:44.209 | 14:30:14.616 | 00:04:09.499 | 14:26:28.901 | 14:30:22.114 | 00:03:53.213 | 17:31:24.730 | 17:53:55.405 | 00:22:30.675 |
cs00000001_0053 | 14:26:11.198 | 14:30:14.619 | 00:04:09.56 | 14:26:38.502 | 14:30:22.120 | 00:03:43.618 | 17:31:34.686 | 17:53:55.405 | 00:22:20.719 |
cs00000001_0054 | 14:26:15.107 | 14:30:14.618 | 00:04:08.079 | 14:26:37.825 | 14:30:22.114 | 00:03:44.289 | 17:31:45.822 | 17:53:55.405 | 00:22:09.583 |
cs00000001_0055 | 14:26:18.467 | 14:30:14.619 | 00:03:34.212 | 14:26:40.893 | 14:30:22.117 | 00:03:41.224 | 17:31:46.415 | 17:54:00.244 | 00:22:13.829 |
cs00000001_0056 | 14:26:20.713 | 14:30:14.620 | 00:03:57.999 | 14:26:40.849 | 14:30:22.117 | 00:03:41.268 | 17:31:45.788 | 17:53:55.405 | 00:22:09.617 |
cs00000001_0057 | 14:26:21.798 | 14:30:14.620 | 00:03:58.051 | 14:26:41.705 | 14:30:22.117 | 00:03:40.412 | 17:31:45.847 | 17:53:56.620 | 00:22:10.773 |
cs00000001_0058 | 14:26:21.241 | 14:30:15.961 | 00:03:33.063 | 14:26:44.945 | 14:30:22.117 | 00:03:37.172 | 17:31:45.868 | 17:53:56.619 | 00:22:10.751 |
cs00000001_0059 | 14:26:21.821 | 14:30:22.124 | 00:03:58.146 | 14:26:46.887 | 14:30:22.123 | 00:03:35.236 | 17:31:55.343 | 17:54:00.245 | 00:22:04.902 |
cs00000001_0060 | 14:26:21.846 | 14:30:10.702 | 00:03:57.914 | 14:26:44.988 | 14:30:22.121 | 00:03:37.133 | 17:31:50.896 | 17:54:00.245 | 00:22:09.349 |
cs00000001_0061 | 14:26:00.707 | 14:30:13.924 | 00:03:58.086 | 14:26:37.825 | 14:30:22.114 | 00:03:44.289 | 17:31:53.075 | 17:54:00.245 | 00:22:07.17 |
Resource Utilization
Service CPU Utilization
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Service CPU usage during the test was quite low, only the mod-data-export-b okapi-b mod-permissions-b services had a slight increase up to 18%. The test lasted after 23:15 bur there were some spikes from another test. but the CPU usage of selected services was the same.
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Memory Utilization
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
DB CPU Utilization
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). Except for the database spikes that every 30 minutes. CPU usage before tets was about 2.5%, during all Data Export of 500, 1000, and 2000 instances, CPU usage increased up to 4%;
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
DB Connections
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). The maximal number of DB connections was 1342 for 2000 DE.
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
DB Load
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
SQL queries
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Additional information
Errors
Data-exports jobs failed with the same errors.
Methodology/Approach
All tests were run using the JMeter script. Data has been pre-generated for all tenants
Infrastructure
PTF -environment mcpt
11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIBv CPUsmax_connections
db.r6g.4xlarge 128 GiB; 16 vCPU
MSK tenant
2 m5.2xlarge brokers in 1 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
The table contains modules, memory, and CPU parameters
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
mcpt-pvt | ||||||||||
mod-consortia | 11 | 2 | 2048 | 1802 | 128 | 768 | 88 | 128 | FALSE | |
mod-organizations | 6 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE | |
mod-data-import | 7 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | FALSE | |
mod-search | 12 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | FALSE | |
mod-configuration | 6 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-permissions | 14 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | FALSE | |
mod-entities-links | 10 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | FALSE | |
mod-inventory-storage | 10 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE | |
mod-data-export-worker | 9 | 2 | 3072 | 2800 | 1024 | 2048 | 384 | 512 | FALSE | |
mod-source-record-storage | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-inventory | 10 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | FALSE | |
mod-di-converter-storage | 9 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-users | 11 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-source-record-manager | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-quick-marc | 10 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | FALSE | |
nginx-okapi | 2 | 2 | 1024 | 896 | 512 | 0 | 0 | 0 | FALSE | |
okapi-b | 1 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | FALSE | |
mod-data-export | 10 | 1 | 1024 | 896 | 1024 | 768 | 88 | 128 | FALSE |