| Table of Contents |
|---|
...
Ticket:
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Summary
Data import Import duration approximately doubling with of 10k and 25k jobs approximately doubles when increasing the number of concurrent jobs on different tenants. This trend is consistent across the central main/first tenant and other tenants.
Smoke test with 50k didn't reveal some issues. Duration for 3 three concurrent DI Create jobs was three times 3x higher than one DI on central the main tenant that ; this only confirm previous statement about the concurrency affecteffect.
Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.
...
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|
| 1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 |
| tenants |
Test Results
Data import
As the number of concurrent Data Import jobs increases and file size grow grows, the duration of DI jobs grow grows proportionally.
Smoke tests Test finished successfully for 3 concurrent DI Create jobs with 50kof 50K each.
| DI Create | # of test | Number of concurrent jobs | Central Main tenant (fs09000000) | Second tenant | Third tenant |
|---|---|---|---|---|---|
| 10k10K | Baseline | 1 | 00:04:56 | ||
| 1 | 2 | 00:10:43 | 00:10:37 | ||
| 2 | 3 | 00:21:12 | 00:21:06 | 00:20:57 * | |
| 25k25K | Baseline | 1 | 00:11:24 | ||
| 3 | 2 | 00:23:44 | 00:23:30 | ||
| 4 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | |
| DI Update | |||||
| 10k10K | Baseline | 1 | 00:06:32 | ||
| 5 | 2 | 00:09:47 | 00:11:26 | ||
| 6 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | |
| 25k25K | Baseline | 1 | 00:15:13 | ||
| 7 | 2 | 00:30:49 | 00:30:52 | ||
| 8 | 3 | 00:47:47 | 00:48:17 | 00:47:54 | |
| DI Create (smoke Smoke test) | |||||
| 50k50K | 9 | 1 | 00:22:31 | ||
| 10 | 3 | 01:12:54 | 01:12:44 | 01:12:35 |
* - Errors occurred only in 10k 10K DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
...
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...