Overview
This document contains the results of testing Fiscal year rollover in scope of ticket
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
...
- Service CPU utilization for mod-orders-storage increased up to 40- 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum.
- DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%).
- Lock wait type wait was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover.
- "Out of shared memory" error occurred during 100K orders rollover with 90K open orders. This issue was reported previously in
.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-428 - "504 Gateway Time-out" error occurred during 100K orders rollover with 50K open orders. This issue was reported and marked as fixed previously (
) but should be investigated furhter.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODORDERS-865 - A set of improvements was prepared by dev team:
. When this fixes are implemented, tests will be conducted again to assess performance and compare results with current ones.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-439
...
Test # | Scenario | Load level | Data quantity | Fiscal year rollover duration | Comments |
|---|---|---|---|---|---|
| 1. | Fiscal year rollover | 1 rollover at a time | 10K orders: 5K - open 5K - closed | 3 hours (2 hours financial rollover, 1 hour orders rollover) | No errors |
| 2. | 50K orders: 40K - open 10K - closed | 2 days 6 hours (1 day 8 hours financial rollover, 22 hours orders rollover) | No errors | ||
| 3. | 100K orders: 90K - open 10K - closed | 8 days (6 days 6 hours financial rollover, 1 day 18 hours orders rollover) | Rollover was stoped manually, "Out of shared memory" error occurred. | ||
| 4. | 100K orders: 50K - open 50K - closed | 6 days 13 hours (3 days 9 hours financial rollover, 3 days 4 hours orders rollover) | Rollover finished by itself, "504 Gateway Time-out" error occurred. |
Service CPU Utilization
Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum.
10K orders
According to the graph, mod-orders-storage is about 40-60% during orders rollover stage.
50K orders
spike - mod-finance-storage
According to the graph, mod-orders-storage is about 15-20% during orders rollover stage stage.
Also, there is a spike between 06/28 and 06/29 for mod-finance-storage module.
100K orders (90K - open, 10K - closed)
There is a spike - for mod-orders and mod-orders-storage at the end of the rollover.
100K orders (50K - open, 50K - closed)
According to the graph, mod-orders-storage cpu utilization is about 10% during orders rollover stage stage.
Memory Utilization
10K orders
50K orders
There is an increase of service memory usage from 44% to 65% for mod-finance-storage - increase from 44% to 65%.
100K orders (90K - open, 10K - closed)
...
100K orders (50K - open, 50K - closed)
Srvice memory usage increase by services:
- mod-finance-storage - increase from 37% to 59%;
- okapi - increase from 44% to 57%;
- mod-orders-storage - increase from 42% to 54%;
- mod-orders - increase from 59% to 64%;
- mod-rinance - increase from 40% to 44%.
DB CPU Utilization
DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%).
10K orders
COU According to the graph, CPU utilization during finance part - was about 31%, during orders part - 83%.
50K orders
COU According to the graph, CPU utilization during finance part - was 63%, during orders part - 85%.
100K orders (90K - open, 10K - closed)
COU According to the graph, CPU utilization during finance part - was 65%, during orders part - 30%.
100K orders (50K - open, 50K - closed)
COU According to the graph, CPU utilization during finance part - was 67%, during orders part - 40%.
DB Connections
10K orders
50K orders
...
100K orders (50K - open, 50K - closed)
Database load
Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover.
10K orders
50K orders
100K orders (90K - open, 10K - closed)
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-orders | 12.6.6 | 10 | 2 | 1024 | 2048 | 1440 | 512 | 1024 |
| mod-orders-storage | 13.5.0 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance | 4.7.1 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance-storage | 8.4.1 | 8 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 5.0.1 | 6 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | 768 |
...
For other orders quantity and data structure JMeter script should be used (perf-testing/workflows-scripts/fiscal_year/FY_rollover_preparation on the GitHub). But in this case data generation will take a lot of time.
...