Data Import MARC Authorities [Poppy]
Overview
In the scope of PERF-698 - Getting issue details... STATUS - * Poppy Release | Data import | Create MARC authority Records it's needed to run tests to answer questions:
- Determine the time it takes to complete Data-import on PCP1 env
- Determine the main modules that are involved in the process (if obvious or if known)
- Test specific settings or items or scenarios:
Check-in and Checkout (CICO) is in progress and there are 5 concurrent users, and run DI for 1K - 5K- 10K-25K-50K
Test concurrent DI jobs with multi-tenants in the same cluster.
Summary
- 1K, 5K, 10K, 50K data import jobs finished successfully without errors or issues. 22K completed with 67 errors, problem in data, because same errors were on 3 tenants.
- General summary. Data Import MARC Authorities work on Poppy like on the previous releases. DI duration is about the same and resource utilization has the same behavior as in the previous releases.
- The environment is stable and can process 5 DI jobs in parallel on 3 tenants.
Test 1. DI Primary tenant 1k-5K-10K-22K-50K
- DI performance for:
- 1K is faster by Orchid but on the same level as Morning Glory and Nolana
- 5K and 10K DI duration is about the same as Orchid, Morning Glory, and Nolana
- 23K and 50K DI duration is longer on Poppy in comparison to Orchid, but faster than on Morning Glory and Nolana
- Resource Utilization during data imports was stable during the test.
- Maximal CPU consumption was for mod-quick-marc service about 55-60% of CPU and spiked up to 80% on mod-data-import-b, when the 50K records file was uploading;
- No memory leaks;
- RDS utilization was about 95%;
- Maximal CPU consumption was for mod-quick-marc service about 55-60% of CPU and spiked up to 80% on mod-data-import-b, when the 50K records file was uploading;
Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU
- Average response time is slower from 2 up to 18% for small files(1K, 5K, and 10K ), but for big files (23K, 50K);
- The duration of DI during CICO is similar to results "without CICO";
- Memory and CPU utilization were without problems. The most consuming CPU service is mod-inventory-b about 80%. RDS utilization was about 95%;
Test 3. Multitenant DI 50K-22K-10K-5K-1K
All DI finished successfully(except 23K, DI file has 67 broken records), with no memory leaks.
Test Runs & Results
Job Profile Default - Create SRS MARC Authority
Test # | # of records | % creates | File | DI duration | DI | DI | DI |
|---|---|---|---|---|---|---|---|
| 1 | 1,000 | 100 | 1k_marc_authority.mrc?api=v2 | 24 s | 27 s | 41 sec | 29 sec |
| 2 | 5,000 | 100 | LC_SUBJ_msplit00000000.mrc?api=v2 | 1 min 21 s | 1 min 15 s | 1min 21s | 1 min 38 sec |
| 3 | 10,000 | 100 | msplit00000000.mrc?api=v2 | 2 min 32 s | 2 min 31 s | 2min 53s | 2 min 53 sec |
| 4 | 22778 | 100 | msplit00000013.mrc?api=v2 | 11 min 14 s | 7 min 7 s | 5 min 42s | 6 min 24 sec |
| 5 | 50,000 | 100 | 50000_authorityrecords.mrc?api=v2 | 22 min | 11 min 24 s | 11 min 11s | 13 min 48 sec |
Test with CICO 5 concurrent users and DI 1K and 5K and 50K
| CICO, Median time without DI | CICO, 95% time without DI | |
|---|---|---|
| Check-In | 516 ms | 567 ms |
| Check-Out | 910 ms | 2094 ms |
Test # | # of records | DI Duration with CICO | CI time Avg | Baseline CI delta | CI time 95th pct | Baseline CI delta | CO time Avg | Baseline CO Avg Delta | CO time 95th pct | Baseline CO delta |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1,000 | 35 sec | 0.525 | +2% | 0.576 | +1% | 1.078 | +18% | 1.326 | -27% |
| 2 | 5,000 | 1 min 41 sec | 0.513 | -1% | 0.612 | +8% | 0.9 | -0.001% | 1.019 | -52% |
| 3 | 10,000 | 3 min 4 sec | 0.581 | +12% | 0.685 | +20% | 1.016 | +11% | 1.321 | -27% |
| 4 | 22,778 | 6 min 32 sec | 0.598 | +16% | 1.542 | +171% | 1.244 | +36% | 1.729 | -18% |
| 5 | 50,000 | 13 min 48 sec | 0.671 | +30% | 1.953 | +244% | 1.51 | +65% | 2.09 | -0.01% |
Multitenant testing
File with 50K, 23K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3
| Test | Num of records | Tenant 1 pcp1-00 duration | Tenant 2 pcp1-01 duration | Tenant 3 pcp1-02 duration |
|---|---|---|---|---|
| 1. | 1000 | 66 min | 75 min | 74 min |
| 2. | 5000 | 67 min | 76 min | 75 min |
| 3. | 10,000 | 67 min | 76 min | 74 min |
| 4. | 22,778 | 67 min* | 73 min* | 71 min* |
| 5. | 50,000 | 62 min | 56 min | 54 min |
*Completed with errors
And on all of the tenants test 4 finished with 67 errors, with the same messages
Error message:io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"JSON parse error: Cannot deserialize value of type `java.util.UUID` from String \"76c74801-afec-45a0-aad7-3ff23591e147 76c74801-afec-45a0-aad7-3ff23591e147\": UUID has to be represented by standard 36-char representation","type":"HttpMessageNotReadableException","code":"validation"}],"total_records":1}
Resource utilization
Test 1. DI Primary tenant 1k-5K-10K-22K-50K
Data-import 17:43 → 1000 records import; 17:50 → 5000 records import; 17:55-18:00 →10K records import; 18:05- 18:10 → 23K records import; 18:20- 18:35→ 50K records import.
Service CPU Utilization
CPU Utilization during data-imports was stable during the test, mod-quick-marc-b consumed about 55-60% of CPU and Spikes up to 80% on mod-data-import-b, but we saw the same behavior when large files were uploaded for data-import.
CPU utilization after tests returned to its before-test statements.
Memory utilization
RDS utilization was about 95%
Database load 
Top SQL
Test 2. DI Primary tenant 1k-5K-10K-22K-50K + CI/CO 5VU
CICO Responce time graph
Data import 1k- 23:53,15 - 23:53,50; 5k- 23:57,17 - 23:59; 10k- 00:04 - 00:07; 23k- 00:09 - 00:15; 3k- 00:18 - 00:32; 
Service CPU Utilization
CPU Utilization during data imports was stable during the test, and after the tests finished returned to its before-test statements.
Memory utilization was stable during the tests.
RDS utilization was about 95%
Number of DB connections
Database load
Top SQL
Test 3. Multitenant DI 50K-22K-10K-5K-1K.
Service CPU Utilization
CPU Utilization during data imports was stable during the test, except 2 spikes on mod-permission-b services to about 110%, and after the tests finished returned to its before-test statements.
Memory utilization was stable during the tests.
Maximal RDS utilization was about 96%
Number of DB connections
Database Load
Appendix
Infrastructure
PTF -environment pcp1
- 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
| pcp1-pvt | ||||||||||
| mod-entities-links | 13 | mod-entities-links:2.0.3 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | FALSE |
| mod-data-import | 11 | mod-data-import:3.0.3 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | FALSE |
| mod-permissions | 32 | mod-permissions:6.4.0 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | FALSE |
| mod-inventory-storage | 12 | mod-inventory-storage:27.0.3 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE |
| mod-source-record-storage | 15 | mod-source-record-storage:5.7.3 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE |
| mod-inventory | 11 | mod-inventory:20.1.3 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | FALSE |
| mod-di-converter-storage | 15 | mod-di-converter-storage:2.1.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE |
| mod-circulation | 12 | mod-circulation:24.0.8 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | FALSE |
| mod-source-record-manager | 14 | mod-source-record-manager:3.7.4 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE |
| mod-quick-marc | 9 | mod-quick-marc:5.0.0 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | FALSE |
| nginx-okapi | 9 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | FALSE |
| okapi-b | 11 | okapi:5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | FALSE |