Table of Contents outline true
...
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Summary
- Data import tests finished successfully, only Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files. Duration of DI grew in correspondence with the number of records in files.
- Check-in and Check-out with 5 virtual users was performed during DI Create new MARC authority records jobs for non-matches No issues.
- Data Import in Quesnelia without CICO perform faster than with it.
- Comparing Poppy and Quesnelia releases
- Check-in / Check-out perform better in Quesnelia. Response time improved during Create jobs for long period of work time on 15% in
...
- Average.
- DI durations improved - 11%-
...
- 14% in Average.
- During testing, we noticed spikes in the mod permissions module. To mitigate this issue and prevent system slowdowns, we adjusted the order of loading files, starting with Tenant 3 (qcp1-02)
...
- , followed by Tenant 2 (qcp1-01)
...
- , and finally Tenant 1 (qcp1-00)
...
- .
Test Results and Comparison
...
Test with 1k, 10k, 25k and 50k records files DI started on one tenant only(qcp1-00), and comparative results between Poppy and Quesnelia.
...
- Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.
CICO, Median time without | CICO, 95% time without DI (Poppy) | CICO, Median time without DI (Quesnelia) | CICO, 95% time without DI (Quesnelia) | CICO, Avg time without DI (Quesnelia) | |
|---|---|---|---|---|---|
| Check-In | 516 ms | 567 ms | 503 ms -2.5% | 593 ms | 511 ms |
| Check-Out | 910 ms | 2094 ms | 836 ms -8% | 1117 ms -46% | 876 ms |
...
- Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.
# of records | DI Duration with CICO | CI time Avg | CI time 95th pct | CO time Avg | CO time 95th pct | # of records (Quesnelia) | DI Duration with CICO | CI time Avg (Quesnelia) | CI time 95th pct (Quesnelia) | CO time Avg (Quesnelia) | CO time 95th pct (Quesnelia) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1,000 | 35 sec | 0.525 | 0.576 | 1.078 | 1.326 | 1,000 | 20 sec | 0.560 +6% | 0.754 +30% | 1.164 +8% | 1.313 -1% |
| 5,000 | 1 min 41 sec | 0.513 | 0.612 | 0.9 | 1.019 | 5,000 | 1 min 19 sec -21.7% | 0.701 +36% | 1.171 +91% | 1.141 +26% | 1.790 +75% |
| 10,000 | 3 min 4 sec | 0.581 | 0.685 | 1.016 | 1.321 | 10,000 | 2 min 35 sec -15.7% | 0.723 +24% | 1.024 +49% | 1.179 +16% | 1.494 +13% |
| 22,778 | 6 min 32 sec | 0.598 | 1.542 | 1.244 | 1.729 | 25,000 | 6 min 26 sec -1.5% | 0.722 +20% | 1.024 -33% | 1.180 -5% | 1.494 -13% |
| 50,000 | 13 min 48 sec | 0.671 | 1.953 | 1.51 | 2.09 | 50,000 | 12 min 16 sec | 0.777 +15% | 1.045 -46% | 1.265 | 1.550 -25% |
...
PTF - environment Quesnelia (qcp1)
- 10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer
Name Memory GIB vCPUs db.r6g.xlarge
32 GiB 4 vCPUs - MSK ptf-mobius-testing2
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
- 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
- 09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
- 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []
Link to Jira ticket: https://folio-org.atlassian.net/browse/PERF-801
Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and system stacked.
...
CPU Utilization when mod-permissions spiked and system stacked.
Recommendations & Jiras (Optional)
Link to Jira ticket: https://folio-org.atlassian.net/browse/PERF-801
Methodology/Approach
DI tests scenario a data import job profile that creates new MARC authority records for non-matches (Job Profile: KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q) were started from UI on Quesnelia (qcp1) env with file splitting features enabled on a non-ecs environment..
...
- The above files are all stored here - MARC Resources
- 22k file what was provided from MARC Resources does nor work, so 50k file was split to file with 25k records and used instead of 22k file. - At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl file, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).
Test set
- Test 1: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only.
- Test 2: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only plus Check-in and Checkout (CICO) for 5 concurrent users.
- Test 3: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)
- Test 4: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file with pause between files: 50k, 25k, 10k, 5k, and 1k for order tenants : Tenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)
- Test 5: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file without pause between files: 1k, 5k, 10k, 25k and 50k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)
...