Test status: COMPLETED WITH ERRORS

Table of Contents

Overview IN PROGRESS

...

Test status: COMPLETED WITH ERRORS

Table of Contents

Overview IN PROGRESS

• Regression testing of Fiscal Year Rollover (FYR) on Eureka based environment in quesnelia release. 
  • The testing triggered from UI for the ledger with generated amount of defined data set (Data quantity**: 50000 orders with Open status and 50000 orders with Pending status).
• Testing include two different phases of the fiscal year transition process:
  • #1 phase: Test Fiscal Year Rollover - is a risk free simulation pre-step to #2 phase which includes only Finance part of the data (to preview how the rollover settings will affect funds, budgets, and encumbrances).
  • #2 phase: Fiscal Year Rollover - perform a permanent changes. It is the actual process which move financial data, budgets, and encumbrances into the new fiscal year. It includes not only Finance but Orders parts of the data too.
• The purpose of testing is to define duration time of those 2 phases, to do comparison between current and previous results of each phase, to find any trends for resource utilization. To recommend improvements how to make it work better if any confirmed.
  • Previous successful testing for #2 phase took 29 hours

• Jiras/ links:

  • The previous results report: [Poppy] Fiscal year rollover testing
  • The current ticket:

    Jira Legacy
    server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-1016

Summary

• Eureka - completed with errorsRunning FYR on eureka completed with errors. Tests failed multiple times on overall order rollover with 504 Gateway Time-out issue.
• The problem confirmed on okapi based environment. In scope of investigation of the problem the Time-out was solved changing parameter max_locks_per_transaction in DB configuration from 64 to 1024.
• Eureka based environment (after changes were applied): 
  • #1 Phase Test FYR took 3 hours 35 minutes, completed successfully.
  • #2 Phase actual FYR took 8 hours and completed with errors (Order rollover Error) .
  Non-eureka - successful
  • - 
• Okapi based environment:
  
  • Test FYR  - 3 hours 20 minutes.
  • FYR - 9 hours 7 minutes
• Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, except a spike at the beginning of test rollover (mod-invoice reached 10%, mod-invoice-storage - 8%). Some modules were steady - mod-inventory - 11%, mod-pubsub - 8%.
• DB CPU usage was on level of 9% during the test rollover and there was growing trend from 9% to 15% during the rollover itself.
• No memory leaks were found.

Recommendations

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
• Investigation of issues on Eureka based environment

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-495

  - Time out issue 

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key EUREKA-518

  - UI pending issue

Errors

• 127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with this amount of data. Probably it may cause fail of FYR. 

Test Runs and Results

...

Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY cs00000int_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.

DB Connections

Test FY rollover (qelc2)

...

PTF -environment qelc2

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.4xlarge	32 GiB	4 vCPUs	-

  
  

• MSK fse-tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

PTF -environment qcon

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	-

  
  

...

Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
• Consider using prepared scripts and queries in S3 bucket - Buckets/fse-ptf/FYR/

The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.

...