Overview
IN PROGRESS...
- Data import create holdings job durations increased significantly in Quesnelia release. 4 times longer with 10k file. And not defined increasing in 80k file because it was stopped after 4 hours of test run with only 46 committed jobs (total for the test was 81).
- Top CPU utilization: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%. Such low resource utilization from modules side can be explained by DB queries huge average latency during INSERT and UPDATE processes which had lock on the same tuple.
- Top memory consumption: mod-inventory-storage-b - 85%, mod-data-import-b - 52%, mod-source-record-storage-b - 45%, mod-source-record-manager-b - 43%. Growing trend was defined in tests set #1 for mod-inventory-storage-b - 85%
- DI job duration for the same file size grew from test to test if to use the same instance HRID to create holdings
- DI perform faster if to use files with 1 unique instance HRID for every 1000 records. DI duration corresponds to file size with such approach. Memory utilized without growing trend. CPU and RDS utilization increased because there are less locks in DB.
Recommendations & Jiras
- Investigate growing trend for mod-inventory-storage in tests set #1 (using 1 instance HRID to create all Holdings).
Errors
- error status for the job
- Define high number of Holdings associated with one instance HRID that's still realistic
Errors
- error status for 32'd split job during 80k file importing- SNAPSHOT_UPDATE_ERROR
Test Runs
Profile used for testing - Default - Create Holdings and SRS MARC Holdings
| Set of tests № | Scenario | Test Conditions | Status |
|---|
| 1 | DI Holdings Create (previous approach) 1 instance HRID for all created holdings | 1K, 5K, 10K, 80K sequentially | Completed |
| 2 | DI Holdings Create (new approach) 1 instance HRID for every 1000 created holdings | 1K, 5K, 10K, 80K sequentially | Completed |
Test Results
Set 1 - Files used to test DI create Holdings had 1 instance HRID for all created Holdings
...
Table contains comparison between Quesnelia and Poppy
Set #1
| Test | File | Duration: Poppy | Duration: Quesnelia set #1 | Difference absolute | Difference percentage |
|---|
| 1 | 1k | 00:00:32 | 00:01:22 | 00:00:50 | 156% |
| 2 | 5k | 00:02:14 | 00:08:00 | 00:05:46 | 258% |
| 3 | 10k | 00:04:35 | 00:22:40 | 00:18:05 | 395% |
| 4 | 80k | 00:36:25 | 04:13:00 | 03:36:35 | 595% |
Set #2
| Test | File | Duration: Poppy | Duration: Quesnelia set #2 | Difference absolute | Difference percentage |
|---|
| 1 | 1k | 00:00:32 | 00:01:03 | 00:00:31 | 97% |
| 2 | 5k | 00:02:14 | 00:04:16 | 00:02:02 | 91% |
| 3 | 10k | 00:04:35 | 00:08:59 | 00:04:24 | 96% |
| 4 | 80k | 00:36:25 | 00:52:05 | 00:15:40 | 43% |
Service CPU Utilization
| Expand |
|---|
| title | CPU utilization, 1k, 5k, 10k, 80k |
|---|
|
Set #1 | Module | CPU (1k) | CPU (5k) | CPU (10k) | CPU (80k) |
|---|
| mod-inventory-b | 20.23 | 24.03 | 20.5 | 16.34 | | mod-di-converter-storage-b | 12.3 | 13.51 | 6.21 | 3.94 | | nginx-okapi | 10.67 | 13.07 | 10.37 | 5.13 | | mod-inventory-storage-b | 9.93 | 14.49 | 12.87 | 10.95 | | mod-quick-marc-b | 8.24 | 7.34 | 7.95 | 6.82 | | mod-source-record-storage-b | 7.99 | 8.42 | 6.47 | 3.79 | | mod-users-b | 7.16 | 5.95 | 5.93 | 6.58 | | okapi-b | 6.76 | 7.8 | 6.15 | 3.75 | | mod-pubsub-b | 5.69 | 5.73 | 5.78 | 5.69 | | mod-data-import-b | 5.15 | 1.27 | 1.6 | 1.23 | | mod-source-record-manager-b | 3.7 | 4.55 | 4.01 | 3.06 | | mod-authtoken-b | 3.69 | 1.26 | 1.46 | 0.99 | | mod-password-validator-b | 2.36 | 2.34 | 2.34 | 3.33 | | mod-feesfines-b | 2.34 | 2.12 | 2.2 | 2.16 | | mod-configuration-b | 2.22 | 2.08 | 1.98 | 2.71 | | mod-permissions-b | 1.66 | 0.8 | 1.11 | 0.94 | | mod-circulation-storage-b | 0.61 | 0.61 | 0.61 | 0.68 | | mod-circulation-b | 0.36 | 0.39 | 0.33 | 0.4 | | pub-okapi | 0.19 | 0.13 | 0.13 | 0.13 |
Set #2 | Module | CPU (1k) | CPU (5k) | CPU (10k) | CPU (80k) |
|---|
| mod-inventory-b | 11.93 | 28.85 | 32.97 | 33.23 | | mod-quick-marc-b | 7.19 | 8.1 | 8.19 | 7.64 | | mod-pubsub-b | 6.2 | 6.56 | 6.72 | 6.55 | | mod-users-b | 5.79 | 6.47 | 6.09 | 6.82 | | mod-configuration-b | 3.12 | 3.21 | 3.47 | 3.4 | | mod-feesfines-b | 2.39 | 2.47 | 2.55 | 2.34 | | mod-password-validator-b | 2.26 | 2.4 | 3.27 | 2.41 | | mod-di-converter-storage-b | 1.95 | 10.22 | 9.39 | 8.98 | | mod-source-record-storage-b | 1.79 | 11.45 | 11.64 | 10.84 | | mod-source-record-manager-b | 1.65 | 6.44 | 6.42 | 5.87 | | mod-data-import-b | 1.36 | 1.48 | 1.83 | 1.48 | | okapi-b | 1.28 | 13.81 | 14 | 14.93 | | mod-authtoken-b | 1.02 | 1.24 | 1.6 | 1.6 | | mod-circulation-storage-b | 0.7 | 0.71 | 0.71 | 0.73 | | nginx-okapi | 0.59 | 20.99 | 20.39 | 22.89 | | mod-permissions-b | 0.48 | 4.95 | 1.65 | 1.63 | | mod-circulation-b | 0.35 | 0.36 | 0.35 | 0.36 | | mod-inventory-storage-b | 0.33 | 14.24 | 14.54 | 13.96 | | pub-okapi | 0.17 | 0.24 | 0.24 | 0.23 |
|
Set #1: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%
...
| Expand |
|---|
|
Set #1 | Module | Memory |
|---|
| mod-inventory-storage-b | 85.62 | | mod-data-import-b | 51.63 | | mod-source-record-storage-b | 44.97 | | mod-source-record-manager-b | 42.86 | | mod-users-b | 40.38 | | mod-inventory-b | 39.47 | | mod-permissions-b | 35.82 | | okapi-b | 33.4 | | mod-di-converter-storage-b | 33.26 | | mod-feesfines-b | 32.37 | | mod-quick-marc-b | 31.46 | | mod-configuration-b | 29.41 | | mod-pubsub-b | 25.66 | | mod-authtoken-b | 20.55 | | mod-circulation-storage-b | 18.93 | | mod-circulation-b | 17.87 | | nginx-okapi | 4.8 | | pub-okapi | 4.8 |
Set #2 | Module | Memory |
|---|
| mod-inventory-storage-b | 56.04 | | mod-data-import-b | 55.45 | | mod-inventory-b | 45.63 | | mod-source-record-manager-b | 41.19 | | mod-users-b | 38.95 | | mod-source-record-storage-b | 37.37 | | mod-quick-marc-b | 33.59 | | mod-permissions-b | 33.45 | | okapi-b | 32.82 | | mod-feesfines-b | 32.65 | | mod-di-converter-storage-b | 31.91 | | mod-configuration-b | 28.49 | | mod-circulation-storage-b | 26.86 | | mod-pubsub-b | 25.83 | | mod-circulation-b | 20.14 | | mod-authtoken-b | 19.97 | | nginx-okapi | 4.69 | | pub-okapi | 4.58 |
|
Set #1
Set #2
RDS CPU Utilization
...
| Expand |
|---|
|
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
|---|
| qcon-pvt |
|
|
|
|
|
|
|
|
| | 16/06/2024 |
|
|
|
|
|
|
|
|
| | mod-inventory-b | 3 | mod-quick-marc-bnginx-okapiinventory:20.2.0 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | | mod-diquick-converter-storagemarc-bokapi-b | 1 | mod-source-record-quick-marc:5.1.0 | 2 | 2288 | 2176 | 128 | 1664 | 384 | 512 | | nginx-okapi | 1 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 |
|
|
| | mod-di-converter-storage-b | 2 | mod-sourcedi-record-manager-b | mod-inventory-storage-b | mod-pubsub-b | mod-users-b | mod-data-import-b | mod-organizations-storage-b | mod-notes-b | mod-gobi-b | mod-permissions-b | mod-search-b | mod-circulation-storage-b | mod-circulation-b | pub-okapiconverter-storage:2.2.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | | okapi-b | 1 | okapi:5.3.0 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | | mod-source-record-storage-b | 1 | mod-source-record-storage:5.8.0 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | | mod-source-record-manager-b | 1 | mod-source-record-manager:3.8.0 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | | mod-inventory-storage-b | 2 | mod-inventory-storage:27.2.0-SNAPSHOT.738 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | | mod-pubsub-b | 1 | mod-pubsub:2.13.0 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 922 | | mod-users-b | 1 | mod-users:19.3.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | | mod-data-import-b | 1 | mod-data-import:3.1.0 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | | mod-organizations-storage-b | 1 | mod-organizations-storage:4.7.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | | mod-notes-b | 1 | mod-notes:5.2.0 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 | | mod-gobi-b | 1 | mod-gobi:2.8.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | | mod-permissions-b | 2 | mod-permissions:6.5.0 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | | mod-search-b | 9 | mod-search:3.3.0-SNAPSHOT.224 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | | mod-circulation-storage-b | 1 | mod-circulation-storage:17.2.0 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | | mod-circulation-b | 2 | mod-circulation:24.2.1 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | | pub-okapi | 1 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 |
|
|
|