| Table of Contents |
|---|
...
- Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum. More details...
- DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%). More details...
- Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover. More details...
- "Out of shared memory" error occurred during 100K orders rollover with 90K open orders. This issue was reported previously in
.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-428 - "504 Gateway Time-out" error occurred during 100K orders rollover with 50K open orders. This issue was reported and marked as fixed previously (
) but should be investigated furhter.Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODORDERS-865
Recommendations and Jiras
Investigate appliocation behavior during rollover:
A set of improvements was prepared by dev team:
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Test Runs and Results
Test # | Scenario | Load level | Data quantity | Fiscal year rollover duration | Comments |
|---|---|---|---|---|---|
| 1. | Fiscal year rollover | 1 rollover at a time | 10K orders: 5K - open 5K - closed | 3 hours (2 hours financial rollover, 1 hour orders rollover) | No errors |
| 2. | 50K orders: 40K - open 10K - closed | 2 days 6 hours (1 day 8 hours financial rollover, 22 hours orders rollover) | No errors | ||
| 3. | 100K orders: 90K - open 10K - closed | 8 days (6 days 6 hours financial rollover, 1 day 18 hours orders rollover) | Rollover was stoped manually, "Out of shared memory" error occurred. | ||
| 4. | 100K orders: 50K - open 50K - closed | 6 days 13 hours (3 days 9 hours financial rollover, 3 days 4 hours orders rollover) | Rollover finished by itself, "504 Gateway Time-out" error occurred. |
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-orders | 12.6.6 | 10 | 2 | 1024 | 2048 | 1440 | 512 | 1024 |
| mod-orders-storage | 13.5.0 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance | 4.7.1 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance-storage | 8.4.1 | 8 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 5.0.1 | 6 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | 768 |
...
For other orders quantity and data structure JMeter script should be used (perf-testing/workflows-scripts/fiscal_year/FY_rollover_preparation on the GitHub). But in this case data generation will take a lot of time.
...