| Table of Contents |
|---|
...
Ticket:Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-474
Previous test report: Data Import Create MARC holdings records [Nolana]
Summary
- Data import jobs duration significantly increased compared to the Nolana results, see TestResults.
- Maximum CPU utilization was observed for mod-inventory-storage module during 80k DI job - 88%, see ServiceCPUUtilization.
- There is a significant memory utilization increase for some modules during 10k and 80k DI jobs, see MemoryUtilization.
- Maximum service memory utilization reached 97% for mod-inventory-storage during 80k DI job, see MemoryUtilization.
- During 80k DI job there is a switch of instrances for reader and writer role. This might happen because of overload and restart of the writer instance, see DBCPUUtilization.
- Data import jobs duration significantly increased compared to the Nolana results, see TestResults. It can be caused by added index on the ncp5 environment on Orchid release to make it production like.
...
It should be noted that trigger process_marc_records_lb_insert_update_trigger was enabled for mod_source_record_storage.marc_records_lb table to make the environment production-like. It could influence the results in current release. At the same time tests showed that DI jobs duration with and without the trigger stays the same.
Recommendations & Jiras
Issue with failing 80K DI jobs was reproduced in 100% (tested several times). This issues requires further investigation.
Ticket:
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Test Results
Profile used for testing - Default - Create Holdings and SRS MARC Holdings
Test | File | Duration: Morning_Glory | Duration: Nolana | Duration: Orchid | Diff_absolute | Diff_percentage |
|---|---|---|---|---|---|---|
1 | 1k | 28s | 32s | 45s | 13s | +40% |
| 2 | 5k | 1m 48s | 4m 20s | 7m 47s | 3m 27s | +79% |
| 3 | 10k | 4m 4s | 3m 24s | 19m 46s | 16m 22s | +481% |
| 4 | 80k | 29m 6s | 21m 22s | 20m (error*) | - | - |
...
Infrastructure
PTF -environment ncp5
- 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731 - number of connections for mod-source-record-manager and mod-source-record-storage - 30 connections.
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Kafka topics partitioning: - 2 partitions for DI topics
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage | 26.0.0 | 12 | 2 | 2048 | 4096 | 3690 | 512 | 3076 |
| mod-inventory | 20.0.6 | 12 | 2 | 1024 | 2880 | 2592 | 512 | 1814 |
| mod-source-record-storage | 5.6.7 | 27 | 2 | 2048 | 5600 | 5000 | 512 | 3500 |
| mod-quick-marc | 3.0.0 | 6 | 1 | 128 | 2288 | 2176 | 512 | 1664 |
| mod-source-record-manager | 3.6.4 | 18 | 2 | 2048 | 5600 | 5000 | 512 | 3500 |
| mod-di-converter-storage | 2.0.5 | 6 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-data-import | 2.7.1 | 10 | 1 | 256 | 2048 | 1844 | 512 | 1292 |
| okapi | 5.0.1 | 8 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2022.06.14 | 7 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2022.06.14 | 7 | 2 | 128 | 1024 | 896 | - | 768 |
...