Overview
This document contains the results of testing Fiscal year rollover in scope of ticket PERF-1016 - Getting issue details... STATUS .
Summary
- Test rollover duration was 3 hours 35 minutes, finished successfully.
- Fiscal year rollover duration was 8 hours and failed with Order rollover Error.
- Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, except a spike at the beginning of test rollover (mod-invoice reached 10%, mod-invoice-storage - 8%). Some modules were steady - mod-inventory - 11%, mod-pubsub - 8%.
- DB CPU usage was on level of 9% during the test rollover and there was growing trend from 9% to 15% during the rollover itself.
- No memory leaks were found.
Test Runs and Results
Test # | Scenario | Load level | Data quantity** | Fiscal year rollover duration Orchid* | Fiscal year rollover duration Poppy* | FYR duration Quesnelia | Comments | ||
|---|---|---|---|---|---|---|---|---|---|
| 1 | Test Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - pending 200K order lines 400K transactions | - | 3.5 hours | 3.5 hours | Test rollover finished successfully. | ||
| 2 | Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - pending 200K order lines 400K transactions | 6 days (failed due to 504 Gateway Time-out" error) | 29 hours | 8 hours (failed due | Rollover failed. |
*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing
*Results are taken from the previous test report. Details can be found at the link: [Poppy] Fiscal year rollover testing
**Data structure can be found at the link: Fiscal year rollover testing
Instance CPU Utilization
Test FY rollover
no graph
FY rollover
no graph
Service CPU Utilization
Test FY rollover
Maximum CPU utilization by modules:
mod-finance storage - 16% (spike at the beginning)
mod-inventory - 10%
mod-orders, mod-orders-storage - 1,5%, 3%
mod-finance - under 1%
FY rollover
Maximum CPU utilization by modules:
mod-inventory - 11%
mod-orders - 1.5%
mod-orders-storage - 3%
mod-finance - 0.15%
mod-finance storage - 1.2%
Memory Utilization
Test FY rollover
Memory usage by modules:
mod-finance-storage - 85%
mod-finance - 41%
mod-orders - 57%
mod-orders-storage - 46%
FY rollover
Memory usage by modules:
mod-finance-storage - 85%
mod-finance - 42%
mod-orders - 58%
mod-orders-storage - 46%
DB CPU Utilization
Test FY rollover
Maximum DB CPU usage - 9 - 20%
FY rollover
Maximum DB CPU usage - 99%
DB Connections
Test FY rollover
FY rollover
DB Load
Test FY rollover
FY rollover
Top SQL
Test FY rollover
FY rollover
autovacuum: VACUUM ANALYZE fs09000000_mod_inventory_storage.instance (to prevent wraparound)
Appendix
Infrastructure
PTF -environment qelc2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.4xlarge
32 GiB 4 vCPUs - - MSK fse-tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Methodology/Approach
Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover
The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.