Overview

This document contains the results of testing Fiscal year rollover (FYR) in scope of ticket PERF-1016 - Getting issue details... STATUS .

Summary

• Eureka - failed 
  • Test fiscal year rollover duration was 3 hours 35 minutes, finished successfully.
  • Fiscal year rollover duration was 8 hours and failed with Order rollover Error.
• Non-eureka - successful
  • Test FYR  - 3 hours 20 minutes.
  • FYR - 9 hours 7 minutes
• Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, except a spike at the beginning of test rollover (mod-invoice reached 10%, mod-invoice-storage - 8%). Some modules were steady - mod-inventory - 11%, mod-pubsub - 8%.
• DB CPU usage was on level of 9% during the test rollover and there was growing trend from 9% to 15% during the rollover itself.
• No memory leaks were found.

Recommendations

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
• Investigation of issues on Eureka based environment MODFISTO-495 - Getting issue details... STATUS - Time out issue  EUREKA-518 - Getting issue details... STATUS - UI pending issue

Errors

• 127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with this amount of data. Probably it may cause fail of FYR. 

Test Runs and Results

*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing

*Results are taken from the previous test report. Details can be found at the link: [Poppy] Fiscal year rollover testing

**Data structure can be found at the link: Fiscal year rollover testing

Instance CPU Utilization

Test FY rollover

no graph

FY rollover

no graph

Service CPU Utilization

Test FY rollover

Maximum CPU utilization by modules:

mod-finance storage - 16% (spike at the beginning)

mod-inventory - 10%

mod-orders, mod-orders-storage - 1,5%, 3%

mod-finance - under 1%

FY rollover

Maximum CPU utilization by modules:

mod-inventory - 11%

mod-orders - 1.5%

mod-orders-storage - 3%

mod-finance - 0.15%

mod-finance storage - 1.2%

Memory Utilization

Test FY rollover

Memory usage by modules:

mod-finance-storage - 85%

mod-finance - 41%

mod-orders - 57%

mod-orders-storage - 46%

FY rollover

Memory usage by modules:

mod-finance-storage - 85%

mod-finance - 42%

mod-orders - 58%

mod-orders-storage - 46%

DB CPU Utilization

Test FY rollover (qelc2)

Maximum DB CPU usage - 9 - 20%

Test FY rollover (qcon)

FY rollover (qelc2)

Maximum DB CPU usage - 99%

FY rollover (qcon)

Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY cs00000int_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.

DB Connections

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

mod-serials was not paused for the test (because of that we see such connection number increase)

DB Load

Test FY rollover (qcon)

FY rollover (qcon)

Top SQL

Test FY rollover (qcon)

FY rollover (qcon)

Appendix

Infrastructure

PTF -environment qelc2

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.4xlarge	32 GiB	4 vCPUs	-

• MSK fse-tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

PTF -environment qcon

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	-

Methodology/Approach

Files

Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
• Consider using prepared scripts and queries in S3 bucket - Buckets/fse-ptf/FYR/

The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.