...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
| Table of Contents |
|---|
This document contains the results of testing Fiscal year rollover in scope of ticket
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Summary
- Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum. More details...
- DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%). More details...
- Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover. More details...
- "Out of shared memory" error occurred during 100K orders rollover with 90K open orders. This issue was reported previously in
.Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-428 - "504 Gateway Time-out" error occurred during 100K orders rollover with 50K open orders. This issue was reported and marked as fixed previously (
) but should be investigated furhterfurther.Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODORDERS-865 - A set of improvements was prepared by dev team:
. When this these fixes are implemented, tests will be conducted again to assess performance and compare results with current ones.Jira Legacy server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-439
Recommendations and Jiras
Investigate application behavior during fiscal year rollover:
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Investigate "Out of shared memory" and "504 Gateway Time-out" errors in scope of previously created tickets.
Conduct another set of tests when all the fixes are implemented.
Test Runs and Results
Test # | Scenario | Load level | Data quantity | Fiscal year rollover duration | Comments |
|---|---|---|---|---|---|
| 1. | Fiscal year rollover | 1 rollover at a time | 10K orders: 5K - open 5K - closed | 3 hours (2 hours financial rollover, 1 hour orders rollover) | No errors |
| 2. | 50K orders: 40K - open 10K - closed | 2 days 6 hours (1 day 8 hours financial rollover, 22 hours orders rollover) | No errors | ||
| 3. | 100K orders: 90K - open 10K - closed | 8 days (6 days 6 hours financial rollover, 1 day 18 hours orders rollover) | Rollover was stoped stopped manually, "Out of shared memory" error occurred. | ||
| 4. | 100K orders: 50K - open 50K - closed | 6 days 13 hours (3 days 9 hours financial rollover, 3 days 4 hours orders rollover) | Rollover finished by itself, "504 Gateway Time-out" error occurred. |
Service CPU Utilization
Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum.
10K orders
According to the graph, mod-orders-storage is about 40-60% during orders rollover stage.
50K orders
According to the graph, mod-orders-storage is about 15-20% during orders rollover stage.
Also, there is a spike between 06/28 and 06/29 for mod-finance-storage module.
100K orders (90K - open, 10K - closed)
There is a spike for mod-orders and mod-orders-storage at the end of the rollover.
100K orders (50K - open, 50K - closed)
According to the graph, mod-orders-storage cpu CPU utilization is about 10% during orders rollover stage.
Memory Utilization
10K orders
50K orders
There is an increase of service memory usage from 44% to 65% for mod-finance-storage.
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
Srvice Service memory usage increase by services:
- mod-finance-storage - increase from 37% to 59%;
- okapi - increase from 44% to 57%;
- mod-orders-storage - increase from 42% to 54%;
- mod-orders - increase from 59% to 64%;
- mod-rinance finance - increase from 40% to 44%.
DB CPU Utilization
DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%).
10K orders
According to the graph, CPU utilization during finance part was about 31%, during orders part - 83%.
50K orders
According to the graph, CPU utilization during finance part was 63%, during orders part - 85%.
100K orders (90K - open, 10K - closed)
According to the graph, CPU utilization during finance part was 65%, during orders part - 30%.
100K orders (50K - open, 50K - closed)
According to the graph, CPU utilization during finance part was 67%, during orders part - 40%.
DB Connections
10K orders
50K orders
100K orders
100K orders (50K - open, 50K - closed)
Database load
Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover.
...
SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE10K orders
50K orders
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
Top SQL
10K orders
50K orders
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
Appendix
Infrastructure
PTF -environment ncp5
- 8 9 m6i.2xlarge EC2 instances located in in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-orders | 12.6.6 | 10 | 2 | 1024 | 2048 | 1440 | 512 | 1024 |
| mod-orders-storage | 13.5.0 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance | 4.7.1 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance-storage | 8.4.1 | 8 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 5.0.1 | 6 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | 768 |
Methodology/Approach
Before each fiscal year rollover data should be preapredprepared.
Before data importing next schemas should be truncated:
...
For other orders quantity and data structure JMeter script should be used (perf-testing/workflows-scripts/fiscal_year/FY_rollover_preparation on the GitHub). But in this case data generation will take a lot of time.
Details on data preparation can be found at the link: Steps for testing process#Fiscalyearrollover