Overview
This document contains the results of testing Fiscal year rollover in scope of ticket PERF-289 - Getting issue details... STATUS .
Summary
- Service CPU utilization for mod-orders-storage increased up to 40-60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum.
- DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%).
- Lock type wait was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover.
- "Out of shared memory" error occurred during 100K orders rollover with 90K open orders. This issue was reported previously in MODFISTO-428 - Getting issue details... STATUS .
- "504 Gateway Time-out" error occurred during 100K orders rollover with 50K open orders. This issue was reported and marked as fixed previously ( MODORDERS-865 - Getting issue details... STATUS ) but should be investigated furhter.
- A set of improvements was prepared by dev team: MODFISTO-439 - Getting issue details... STATUS . When this fixes are implemented, tests will be conducted again to assess performance and compare results with current ones.
Test Runs and Results
Test # | Scenario | Load level | Data quantity | Fiscal year rollover duration | Comments |
|---|---|---|---|---|---|
| 1. | Fiscal year rollover | 1 rollover at a time | 10K orders: 5K - open 5K - closed | 3 hours (2 hours financial rollover, 1 hour orders rollover) | No errors |
| 2. | 50K orders: 40K - open 10K - closed | 2 days 6 hours (1 day 8 hours financial rollover, 22 hours orders rollover) | No errors | ||
| 3. | 100K orders: 90K - open 10K - closed | 8 days (6 days 6 hours financial rollover, 1 day 18 hours orders rollover) | Rollover was stoped manually, "Out of shared memory" error occurred. | ||
| 4. | 100K orders: 50K - open 50K - closed | 6 days 13 hours (3 days 9 hours financial rollover, 3 days 4 hours orders rollover) | Rollover finished by itself, "504 Gateway Time-out" error occurred. |
Service CPU Utilization
10K orders
mod-orders-storage is about 40-60% during orders rollover stage
50K orders
spike - mod-finance-storage
mod-orders-storage is about 15-20% during orders rollover stage
100K orders (90K - open, 10K - closed)
spike - mod-orders and mod-orders-storage
100K orders (50K - open, 50K - closed)
mod-orders-storage is about 10% during orders rollover stage
Memory Utilization
10K orders
50K orders
mod-finance-storage - increase from 44% to 65%
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
mod-finance-storage - increase from 37% to 59%
okapi - increase from 44% to 57%
mod-orders-storage - increase from 42% to 54%
mod-orders - increase from 59% to 64%
mod-rinance - increase from 40% to 44%
DB CPU Utilization
10K orders
COU utilization during finance part - 31%, orders part - 83%
50K orders
COU utilization during finance part - 63%, orders part - 85%
100K orders (90K - open, 10K - closed)
COU utilization during finance part - 65%, orders part - 30%
100K orders (50K - open, 50K - closed)
COU utilization during finance part - 67%, orders part - 40%
DB Connections
10K orders
50K orders
100K orders
100K orders (50K - open, 50K - closed)
Database load
10K orders
50K orders
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
Top SQL
10K orders
50K orders
100K orders (90K - open, 10K - closed)
100K orders (50K - open, 50K - closed)
Appendix
Infrastructure
PTF -environment ncp5
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-orders | 12.6.6 | 10 | 2 | 1024 | 2048 | 1440 | 512 | 1024 |
| mod-orders-storage | 13.5.0 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance | 4.7.1 | 5 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-finance-storage | 8.4.1 | 8 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 5.0.1 | 6 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2022.03.02 | 6 | 2 | 128 | 1024 | 896 | - | 768 |
Methodology/Approach
Before each fiscal year rollover data should be preapred.
Before data importing next schemas should be truncated:
- mod_finance_storage.budget
- mod_finance_storage.transaction
- mod_finance_storage.budget_expense_class
- mod_finance_storage.expense_class
- mod_finance_storage.fund
- mod_finance_storage.fund_type
- mod_finance_storage.fiscal_year
- mod_finance_storage.ledger
- mod_finance_storage.group_fund_fiscal_year
- mod_orders_storage.purchase_order
- mod_orders_storage.po_line
- mod_orders_storage.titles
- mod_notes.note
To import data (100K orders: 50% open, 50% closed) backup of DB schemas was created. It can be found in "Finance" folder on the host.
For other orders quantity and data structure JMeter script should be used (perf-testing/workflows-scripts/fiscal_year/FY_rollover_preparation on the GitHub). But in this case data generation will take a lot of time.