| Table of Contents |
|---|
...
- Eureka - failed
- Test fiscal year rollover duration was 3 hours 35 minutes, finished successfully.
- Fiscal year rollover duration was 8 hours and failed with Order rollover Error.
- Non-eureka - successful
- Test FYR - 3 hours 20 minutes.
- FYR - 9 hours 7 minutes
- Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, except a spike at the beginning of test rollover (mod-invoice reached 10%, mod-invoice-storage - 8%). Some modules were steady - mod-inventory - 11%, mod-pubsub - 8%.
- DB CPU usage was on level of 9% during the test rollover and there was growing trend from 9% to 15% during the rollover itself.
- No memory leaks were found.
Recommendations
- Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
Test Runs and Results
Test # | Scenario | Load level | Data quantity** | Fiscal year rollover duration Orchid* | Fiscal year rollover duration Poppy* | FYR duration Quesnelia (Eureka) | FYR duration Quesnelia (non-Eureka) | Comments | ||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Test Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - pending 200K order lines 400K transactions | - | 3.5 hours | 3.5 hours | 3 hours 20 minutes | Test rollover finished successfully. | ||
| 2 | Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - pending 200K order lines 400K transactions | 6 days (failed due to 504 Gateway Time-out" error) | 29 hours | 8 hours (failed due | 9 hours 7 minutes | On eureka Rollover failed. On non-eureka Rollover succeeded. |
...
mod-orders-storage - 46%
DB CPU Utilization
Test FY rollover (qelc2)
Maximum DB CPU usage - 9 - 20%
Test FY rollover (qcon)
FY rollover (qelc2)
Maximum DB CPU usage - 99%
...
...
FY rollover
...
FY rollover
(qcon)
Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY cs00000int_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.
DB Connections
Test FY rollover (qelc2)
Test FY rollover (qcon)
FY rollover (qelc2)
FY rollover (qcon)
mod-serials was not paused for the test (because of that we see such connection number increase)
DB Load
Test FY rollover (qcon)
FY rollover (qcon)
Top SQL
Test FY rollover (qcon)
FY rollover
...
(qcon)
Appendix
Infrastructure
PTF -environment qelc2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.4xlarge
32 GiB 4 vCPUs - - MSK fse-tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
PTF -environment qcon
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs -
...
Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover
- Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
- Consider using prepared scripts and queries in S3 bucket - Buckets/fse-ptf/FYR/
The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.
...