Data Export Test Report (Mobius) part 2
- Mykhailo Petryshyn
Overview
This document contains the results of Data Export tests on 61 tenants (excluding the central tenant) on the MCPT environment, this report is the second part of the DE testing, the previous report is available Data Export Test Report (Mobius) | Errors / Additional information :
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP).
T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP).
T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP)
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Ticket:https://folio-org.atlassian.net/browse/PERF-803
Summary
The tested Data-export jobs are quite light for the system and cause an increase in CPU on the services: mod-data-export, okapi and mod-inventory-storage up to 17%;
No memory leaks, memory consumption was stable during all of the tests;
Database CPU usage varied up to a maximum of 8%;
The number of failed DE jobs during concurrent testing was about 7% all with the same errors, but the duration of failed jobs was like the duration of completed, error messages are at the end of the report
The behavior of the export date was the same as in the first part of the report, with the latest processes starting with a delay. On the first tenants, where the file with instance ids was downloaded the data-export process is completed the fastest, but the data-export on the last tenants lasts the longest, as can be seen from the graphs below the tables
General metrics can be found in the tables below
Data-export duration statistics:
Data export | T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). | T2-DE_instances(500-1K-2k)(CJP)+DE_instancesCS42(MAX)(DEP). | T3-DE_instances(500-1K-2k)(DEP)+DE_authorityCS00(MAX)(DEP) | ||||||
500, sec | 1000 , sec | 2000 , sec | 500 , sec | 1000 , sec | 2000, sec | 500 , sec | 1000 , sec | 2000, sec | |
Average duration | 80 | 256 | 830 | 956 | 2110 | 4441 | 215 | 533 | 1281 |
Minimal duration | 21 | 19 | 74 | 384 | 982 | 1863 | 45 | 14 | 12 |
Maximal duration | 1028 | 360 | 987 | 1075 | 2257 | 4624 | 254 | 558 | 1374 |
75t percentile | 77 | 339 | 968 | 1029 | 2223 | 4603 | 244 | 657 | 1362 |
90t percentile | 108 | 356 | 982 | 1037 | 2241 | 4618 | 249 | 661 | 1368 |
CS42(MAX RECORDS) | 1 hrs 56 min | 1 hrs 49 min | 2hrs 18 min | 2hrs 10 min | 2hrs 30 min | 1 hrs 49 min | 32 min 7 sec | 44 min 3 sec | 55 min 31 sec |
T4- DE_instances(500-1K-2k)(DEP)+DE_authority(500-1K-2k)(DEP)+ )+holdings(MAX)(DEP)+DE_authorityCS00(MAX)(DEP)
| 500 records | 1000 records | 2000 records |
MARC | 10 seconds | 35 seconds | 1 minutes 45 seconds |
Holdings | 8,3 seconds | 26 seconds | 1 minutes 43 seconds |
Holdings MAX | 43 minutes 26 seconds | 41 minutes 16 seconds | 40 minutes 55 seconds |
authorityCS00 | 36 minutes 2 seconds | 35 minutes 37 seconds | 36 minutes 39 seconds |
T5- DE_instances(500-1K-2k)(CJP)+DE_instances(MAX)(DEP)
Tenant cs00000001_0003 | Starterted time | Finished time | Duration |
Test 3. Data-export . Max records. Default export profile | "2024-02-23T11:02:11.934+00:00" | "2024-02-23T11:29:32.025+00:00" | 00:27:20.091 |
Test 3. Data-export . 2000 records. Custom job profile | "2024-02-23T11:02:00.802+00:00" | "2024-02-23T11:03:30.687+00:00" | 00:01:29.885 |
Test 2. Data-export . Max records. Default export profile | "2024-02-23T10:26:59.739+00:00" | "2024-02-23T10:53:33.241+00:00" | 00:26:33.502 |
Test 2. Data-export . 1000 records. Custom job profile | "2024-02-23T10:26:43.425+00:00" | "2024-02-23T10:27:25.347+00:00" | 00:00:41.922 |
Test 1. Data-export . Max records. Default export profile | "2024-02-23T09:56:41.692+00:00" | "2024-02-23T10:23:11.426+00:00" | 00:26:29.734 |
Test 1. Data-export . 500 records. Custom job profile | "2024-02-23T09:56:20.276+00:00" | "2024-02-23T09:56:39.402+00:00" | 00:00:19.126 |
Recommendations & Jiras
Investigate Data-exports job that have “FAILED“ status and <Code>NoSuchKey</Code> <Message>The specified key does not exist.</Message>
Run data-import and data-export of authority records with default profiles to reproduce 5xx errors on okapi and mod-entity-link high memory utilization.
Test Results
T1-DE_instances(500-1K-2k)(DEP)+DE_instancesCS42(MAX)(DEP). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records) except tenant 42, on this tenant data-export run with the largest number of instances and “Default instances export profile”. Additional information about failed jobs is in the Additional Information/Error section
Tenant | 500 | 1000 | 2000 | |||||||||
Status | Started Date | Compl Date | Duration | Status | Started Date | Compl Date | Duration | Status | Started Date | Compl Date | Duration | |
cs00000001_0001 | COMPLETED | 08:41:18.306 | 08:41:42.679 | 00:00:24.373 | COMPLETED | 16:12:56.192 | 16:13:15.479 | 00:00:19.287 | COMPLETED | 20:43:32.753 | 20:44:47.318 | 00:01:14.565 |
cs00000001_0002 | COMPLETED | 08:41:25.000 | 08:41:47.885 | 00:00:22.885 | COMPLETED | 16:13:01.591 | 16:13:34.742 | 00:00:33.151 | COMPLETED | 20:43:37.932 | 20:45:58.223 | 00:02:20.291 |
cs00000001_0003 | COMPLETED | 08:41:31.646 | 08:41:53.671 | 00:00:22.025 | COMPLETED | 16:13:09.258 | 16:13:52.747 | 00:00:43.489 | COMPLETED | 20:43:46.351 | 20:47:26.980 | 00:03:40.629 |
cs00000001_0004 | COMPLETED | 08:41:36.937 | 08:41:58.192 | 00:00:21.255 | COMPLETED | 16:13:17.224 | 16:14:11.263 | 00:00:54.039 | COMPLETED | 20:43:56.513 | 20:49:11.379 | 00:05:14.866 |
cs00000001_0005 | COMPLETED | 08:41:42.698 | 08:42:03.951 | 00:00:21.253 | COMPLETED | 16:13:24.351 | 16:14:36.431 | 00:01:12.08 | COMPLETED | 20:44:03.713 | 20:50:33.894 | 00:06:30.181 |
cs00000001_0006 | COMPLETED | 08:41:47.885 | 08:42:08.914 | 00:00:21.029 | COMPLETED | 16:13:30.177 | 16:14:50.495 | 00:01:20.318 | COMPLETED | 20:44:12.660 | 20:51:55.779 | 00:07:43.119 |
cs00000001_0007 | FAIL | 08:41:53.167 | 08:42:15.317 | 00:00:22.15 | COMPLETED | 16:13:38.634 | 16:15:12.011 | 00:01:33.377 | COMPLETED | 20:44:21.387 | 20:53:27.125 | 00:09:05.738 |
cs00000001_0008 | COMPLETED | 08:41:59.692 | 08:42:25.276 | 00:00:25.584 | COMPLETED | 16:13:43.573 | 16:15:26.623 | 00:01:43.05 | COMPLETED | 20:44:29.600 | 20:54:27.147 | 00:09:57.547 |
cs00000001_0009 | COMPLETED | 08:42:04.965 | 08:42:33.644 | 00:00:28.679 | COMPLETED | 16:13:52.774 | 16:15:50.168 | 00:01:57.394 | COMPLETED | 20:44:37.979 | 20:55:25.725 | 00:10:47.746 |
cs00000001_0010 | COMPLETED | 08:42:10.940 | 08:42:42.604 | 00:00:31.664 | COMPLETED | 16:13:59.297 | 16:16:09.067 | 00:02:09.77 | COMPLETED | 20:44:46.845 | 20:56:21.134 | 00:11:34.289 |
cs00000001_0011 | FAIL | 08:42:17.625 | 08:42:50.114 | 00:00:32.489 | COMPLETED | 16:14:05.996 | 16:16:27.609 | 00:02:21.613 | COMPLETED | 20:44:54.729 | 20:57:13.442 | 00:12:18.713 |
cs00000001_0012 | COMPLETED | 08:42:25.296 | 08:43:00.791 | 00:00:35.495 | COMPLETED | 16:14:13.919 | 16:16:50.901 | 00:02:36.982 | COMPLETED | 20:45:00.316 | 20:57:40.427 | 00:12:40.111 |
cs00000001_0013 | COMPLETED | 08:42:30.529 | 08:43:09.030 | 00:00:38.501 | COMPLETED | 16:14:19.843 | 16:17:03.193 | 00:02:43.35 | COMPLETED | 20:45:09.757 | 20:58:33.223 | 00:13:23.466 |
cs00000001_0014 | COMPLETED | 08:42:36.508 | 08:43:19.801 | 00:00:43.293 | COMPLETED | 16:14:26.105 | 16:17:15.922 | 00:02:49.817 | COMPLETED | 20:45:16.057 | 20:58:58.640 | 00:13:42.583 |
cs00000001_0015 | FAIL | 08:42:43.408 | 08:43:29.083 | 00:00:45.675 | COMPLETED | 16:14:35.856 | 16:17:41.369 | 00:03:05.513 | COMPLETED | 20:45:23.453 | 20:59:23.966 | 00:14:00.513 |
cs00000001_0016 | COMPLETED | 08:42:50.139 | 08:43:35.961 | 00:00:45.822 | COMPLETED | 16:14:41.975 | 16:17:57.570 | 00:03:15.595 | COMPLETED | 20:45:34.973 | 20:59:48.715 | 00:14:13.742 |
cs00000001_0017 | COMPLETED | 08:42:56.821 | 08:43:44.103 | 00:00:47.282 | COMPLETED | 16:14:50.522 | 16:18:22.953 | 00:03:32.431 | COMPLETED | 20:45:42.889 | 21:00:13.600 | 00:14:30.711 |
cs00000001_0018 | COMPLETED | 08:43:01.562 | 08:43:55.090 | 00:00:53.528 | COMPLETED | 16:14:57.741 | 16:18:43.230 | 00:03:45.489 | COMPLETED | 20:45:50.660 | 21:00:37.526 | 00:14:46.866 |
cs00000001_0019 | FAIL | 08:43:09.901 | 08:44:00.858 | 00:00:50.957 | FAIL | 16:15:03.898 | 16:18:54.326 | 00:03:50.428 | COMPLETED | 20:45:58.742 | 21:01:00.588 | 00:15:01.846 |
cs00000001_0020 | COMPLETED | 08:43:15.211 | 08:44:12.492 | 00:00:57.281 | COMPLETED | 16:15:11.436 | 16:19:20.244 | 00:04:08.808 | COMPLETED | 20:46:07.696 | 21:01:24.518 | 00:15:16.822 |
cs00000001_0021 | COMPLETED | 08:43:24.138 | 08:44:18.231 | 00:00:54.093 | FAIL | 16:15:18.252 | 16:19:36.126 | 00:04:17.874 | COMPLETED | 20:46:17.326 | 21:01:47.076 | 00:15:29.75 |
cs00000001_0022 | COMPLETED | 08:43:29.110 | 08:44:24.802 | 00:00:55.692 | COMPLETED | 16:15:25.017 | 16:19:44.324 | 00:04:19.307 | COMPLETED | 20:46:26.716 | 21:02:09.121 | 00:15:42.405 |
cs00000001_0023 | COMPLETED | 08:43:34.515 | ||||||||||