[Quesnelia][Eureka] Fiscal year rollover testing
Test status: COMPLETED WITH ERRORS
Overview
- Regression testing of Fiscal Year Rollover (FYR) on Eureka based environment in quesnelia release.
- The testing triggered from UI for the ledger with generated amount of defined data set (Data quantity**: 50000 orders with Open status and 50000 orders with Pending status).
- Testing include two different phases of the fiscal year transition process:
- #1 phase: Test Fiscal Year Rollover - is a risk free simulation pre-step to #2 phase which includes only Finance part of the data (to preview how the rollover settings will affect funds, budgets, and encumbrances).
- #2 phase: Fiscal Year Rollover - perform a permanent changes. It is the actual process which move financial data, budgets, and encumbrances into the new fiscal year. It includes not only Finance but Orders parts of the data too.
- The purpose of testing is to define duration time of those 2 phases, to do comparison between current and previous results of each phase, to find any trends for resource utilization. To recommend improvements how to make it work better if any confirmed.
- Previous successful testing for #2 phase took 29 hours
Jiras/ links:
Summary
- Running FYR on eureka completed with errors. Tests failed multiple times on overall order rollover with 504 Gateway Time-out issue.
- The problem confirmed on okapi based environment. In scope of investigation of the problem the Time-out was solved changing parameter max_locks_per_transaction in DB configuration from 64 to 1024.
- Eureka based environment (after changes were applied):
- #1 Phase Test FYR took 3 hours 35 minutes, completed successfully.
- #2 Phase actual FYR took 8 hours and completed with errors* (Order rollover Error).
- Okapi based environment (after changes were applied, testing is carried out in scope of investigation to check functionality of FYR):
- #1 Phase Test FYR took 3 hours 20 minutes. Completed successfully.
- #2 Phase actual FYR took 9 hours 7 minutes. Completed successfully.
- Service CPU utilization for both phases was under 5% on eureka environment except a spike at the beginning of test rollover (mod-finance-storage - 16%, mod-invoice - 10%). Module mod-inventory - 11% during whole test.
- DB CPU usage was on level of 9% during #1 Phase and there was growing trend from 9% to 15% during the rollover itself on Eureka environment with 4xlarge database. On xlarge database - 39% without growth.
- No memory leaks were found.
Recommendations & Jiras
- Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid Time-out issue during actual rollover (#2 Phase).
- Investigation of issues on Eureka based environment - MODFISTO-495Getting issue details... STATUS - Time out issue.
Errors
- 127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with open orders >= 42000 after max_locks_per_transaction parameter was changed from 64 to 1024. Probably it may cause fail of FYR.
- * Error message: io.vertx.core.impl.NoStackTraceThrowable: Response status code 401 is not between 200 and 299,qelc2/mod-finance-storage/9044d47f773b48a99cc6c8420864b724 which indicate possible RTR issue.
- WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-4) Egress routing entry was not found for the request's path. Forwarding request to the Gateway:moduleId = sc-mod-finance-storage, path = /orders/rollover, destination = https://kong-qelc2.int.aws.folio.org, x-okapi-module-id = null",qelc2/sidecar-mod-finance-storage/9044d47f773b48a99cc6c8420864b724,054267740449:qelc2-folio-eis
- ERROR edgerRolloverService handleOrderRolloverError:: Orders rollover failed for ledger f38a29f5-feaa-4363-98a1-1eba79557c9c
Test Runs and Results
Data size: 100K orders (50K orders in open status, 50K - pending), 200K order lines, 400K transactions.
Test # | Scenario | Load level | Fiscal year rollover duration Orchid* | Fiscal year rollover duration Poppy* | FYR duration Quesnelia (Eureka) | FYR duration Quesnelia (non-Eureka) | Comments |
---|---|---|---|---|---|---|---|
1 | Test Fiscal year rollover | 1 rollover at a time | - | 3.5 hours | 3.5 hours | 3 hours 20 minutes | Test rollover finished successfully. |
2 | Fiscal year rollover | 1 rollover at a time | 6 days (failed with 504 error) | 29 hours | 8 hours, failed due to 500(server error) | 9 hours 7 minutes | On eureka Rollover competed with error. On non-eureka Rollover completed successfully. |
*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing
*Results are taken from the previous test report. Details can be found at the link: [Poppy] Fiscal year rollover testing
**Data structure can be found at the link: Fiscal year rollover testing
Service CPU Utilization
Note: Instance level CPU screenshots is not included in this report. Taking into account that Fiscal Year Rollover process is mostly happening on DB side - it's barely visible on service CPU, so it's invisible on instance CPU.
Test FYR (Eureka)
Low CPU usage during FYR reflects that it can handle large fiscal year transitions efficiently. As the process runs mostly on database level.
Maximum CPU utilization by modules:
mod-finance storage - 16% (spike at the beginning)
mod-inventory - 10%
mod-orders, mod-orders-storage - 1,5%, 3%
mod-finance - under 1%
FYR (Eureka)
Maximum CPU utilization by modules on eureka:
mod-inventory - 11%
mod-orders - 1.5%
mod-orders-storage - 3%
mod-finance - 0.15%
mod-finance storage - 1.2%
Memory Utilization
Test FYR (Eureka)
This graph shows memory usage during #1 phase of FYR. No signs of memory leaks trends.
Memory usage by modules:
mod-finance-storage - 85%
mod-finance - 41%
mod-orders - 57%
mod-orders-storage - 46%
FYR (Eureka)
This graph shows memory usage during #2 phase of FYR. No signs of memory leaks trends too. Consumption is on the same level as #1 phase.
Memory usage by modules:
mod-finance-storage - 85%
mod-finance - 42%
mod-orders - 58%
mod-orders-storage - 46%
DB CPU Utilization
Test FYR (Eureka)
DB CPU usage range: 9% - 20%. Utilization was stable during #1 phase except of spikes to 20%. It competed successfully so it didn't affect the test.
Test FYR (non-Eureka)
The #1 phase of FYR was carried out on xlarge DB instance. That's the reason why we see 34% of DB CPU utilization. The utilization was stable - 34%. This phase completed successfully.
FYR (Eureka)
Maximum DB CPU usage - 99%
FYR (non-Eureka)
Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY [tenant]_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.
The #2 phase completed successfully in non-eureka. It utilized 35% which is almost the same as in #1 phase.
DB Connections
DB connections stay on the same level during all phases of FYR. Connections in Eureka - 690 and non-Eureka 1380.. The difference can be explained by enabled mod-serials-management module on qcon environment.
Test FYR (Eureka)
Test FYR (non-Eureka)
FYR (Eureka)
FYR (non-Eureka)
On qcon module mod-serials spiking affect DB connections growth from 700 to 1400.
DB Load
Test FY rollover (non-Eureka)
On this graphs sql statement responsible for budget encumbrances rollover consume most of CPU.
FY rollover
Top SQL
Test FY rollover (non-Eureka)
On this graphs sql statement responsible for budget encumbrances rollover consume most of CPU.
FY rollover
Appendix
Infrastructure
PTF -environment qelc2
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer.
Name Memory GIB vCPUs max_connections db.r6g.4xlarge
128 GiB 16 vCPUs 4000 db.r6g.xlarge 32 GB 4vCPUs 2000 database cluster and instance parameters max_locks_per_transaction - 64, changed to 1024
- MSK fse-tenant
- 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
Apache Kafka version 3.7.x, metadata mode - KRaft
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
PTF -environment qcon
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2000 database cluster and instance parameters max_locks_per_transaction - 64, changed to 1024
Methodology/Approach
In order to run FYR:
truncate tables with query
truncate table [tenant]_mod_organizations_storage.organizations cascade; truncate table [tenant]_mod_orders_storage.purchase_order cascade; truncate table [tenant]_mod_invoice_storage.invoices cascade;
Create organization (Vendor) to reuse its id as access provider in the next query which generates needed data set.
Replace FYR id, access provider, fund id, fund name in accordance with FYR ledger info. Also check all instance related fields which should be replaced with actual data. The script prepared to generate needed amount of organizations, orders, po lines, invoices.
To create encumbrances linked to the current budget orders should be with open status. To change the status from pending to open select needed data for the jmeter script. Save to use in the PUT request orders/composite-orders/${po_id}. Run needed number of requests to prepare open orders.
SELECT id as po_id, jsonb->>'vendor' as vendor_id, jsonb->>'poNumber' as po_number select count(id) FROM cs00000int_mod_orders_storage.purchase_order where jsonb->>'workflowStatus'='Pending'
{"id":"${po_id}","approved":true,"notes":[],"poNumber":"${po_number}","orderType":"One-Time","reEncumber":true,"vendor":"${vendor_id}","workflowStatus":"Open","acqUnitIds":[],"needReEncumber":false}
Legacy approach about data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover Use this link to get exact number of organizations, orders, POLined, invoices
- Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
- Additional info for testing (scripts, queries) saved in S3 bucket - Buckets/fse-ptf/FYR/
- envs_data_gen_scripts - txt files with sql which populate DB with data
- qelc2 and qcon jmeter scripts to change status from pending to open