Test status: COMPLETED WITH ERRORS

...

• Running FYR on eureka completed with errors. Tests failed multiple times on overall order rollover with 504 Gateway Time-out issue.
• The problem confirmed on okapi based environment. In scope of investigation of the problem the Time-out was solved changing parameter max_locks_per_transaction in DB configuration from 64 to 1024.
• Eureka based environment (after changes were applied): 
  • #1 Phase Test FYR took 3 hours 35 minutes, completed successfully.
  • #2 Phase actual FYR took 8 hours and completed with errors* (Order rollover Error).
• Okapi based environment (after changes were applied, testing is carried out in scope of investigation to check functionality of FYR):
  
  • #1 Phase Test FYR took 3 hours 20 minutes. Completed successfully.
  • #2 Phase actual FYR took 9 hours 7 minutes. Completed successfully.
• Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, both phases was under 5% on eureka environment except a spike at the beginning of test rollover (mod-invoice reached 10%-finance-storage - 16%, mod-invoice - storage - 8%10%). Some modules were steady - Module mod-inventory - 11% , mod-pubsub - 8%.during whole test. 
• DB CPU usage was on level of 9% during the test rollover #1 Phase and there was growing trend from 9% to 15% during the rollover itself on Eureka environment with 4xlarge database. On xlarge database - 39% without growth.
• No memory leaks were found.

Recommendations & Jiras

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRolloverTime-out issue during actual rollover (#2 Phase).
• Investigation of issues on Eureka based environment

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODFISTO-495

  - Time out issue.

Errors

• 127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with this amount of data. Probably it may cause fail of FYR. 
• * Error message: io.vertx.core.impl.NoStackTraceThrowable: Response status code 401 is not between 200 and 299,qelc2/mod-finance-storage/9044d47f773b48a99cc6c8420864b724 which indicate possible RTR issue.
• WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-4) Egress routing entry was not found for the request's path. Forwarding request to the Gateway:moduleId = sc-mod-finance-storage, path = /orders/rollover, destination = https://kong-qelc2.int.aws.folio.org, x-okapi-module-id = null",qelc2/sidecar-mod-finance-storage/9044d47f773b48a99cc6c8420864b724,054267740449:qelc2-folio-eis
  
• ERROR edgerRolloverService handleOrderRolloverError:: Orders rollover failed for ledger f38a29f5-feaa-4363-98a1-1eba79557c9c

...

Note: Instance level CPU screenshots is not included in this report. Taking into account that Fiscal Year Rollover process is mostly happening on DB side - it's barely visible on service CPU, so it's invisible on instance CPU. 

Test

...

FYR (qelc2)

Low CPU usage during FYR reflects that it can handle large fiscal year transitions efficiently. As the process runs mostly on database level. 

Image Added

Maximum CPU utilization by modules:

...

mod-orders, mod-orders-storage - 1,5%, 3%

mod-finance - under 1%

...

FYR (qelc2)

Image Added

Maximum CPU utilization by modules:

...

mod-finance storage - 1.2%

Memory Utilization

Test

...

FYR (qelc2)

Image Added

Memory usage by modules:

mod-finance-storage - 85%

...

mod-orders - 57%

mod-orders-storage - 46%

...

FYR (qelc2)

Memory usage by modules graph does not show any memory leaks trends.

Image Added

Memory usage by modules:

mod-finance-storage - 85%

...

mod-orders-storage - 46%

DB CPU Utilization

Test

...

FYR (qelc2)

...

Maximum DB CPU usage range: 9% - 9 - 20%

Image Added

Test FY rollover (qcon)

...

FYR (qelc2)

Maximum DB CPU usage - 99%

...

FYR (qcon)

Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY [tenant]_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.

DB Connections

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

On qcon module mod-serials was not paused for the test (because of that we see such connection number increase)spiking affect DB connections growth from 700 to 1400.

DB Load

Test FY rollover (qcon)

FY rollover (qcon)

Top SQL

Test FY rollover (qcon)

FY rollover (qcon)

Appendix

Infrastructure

PTF -environment qelc2

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.4xlarge	32 128 GiB	4 16 vCPUs-	4000
  db.r6g.xlarge	32 GB	4vCPUs	2000

  
  

• MSK fse-tenant
  
  • 4 m5kafka.m7g.2xlarge xlarge brokers in 2 zones (2 brokers per zone)

  • Apache Kafka version 3.7.x, metadata mode - KRaft

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

PTF -environment qcon

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	-2000

  
  

...

To create encumbrances linked to the current budget orders should be with open status. To change the status from pending to open select needed data for the jmeter script. Save to use in the PUT request orders/composite-orders/${po_id}. Call Run needed number of requests to prepare open orders.

...

{"id":"${po_id}","approved":true,"notes":[],"poNumber":"${po_number}","orderType":"One-Time","reEncumber":true,"vendor":"${vendor_id}","workflowStatus":"Open","acqUnitIds":[],"needReEncumber":false}

Before each fiscal year rollover data should be prepared. Common information Legacy approach about data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

• Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
• Additional info for testing (scripts, queries) saved in S3 bucket - Buckets/fse-ptf/FYR/