Test status: completed

Overview

• This document includes testing of Fiscal Year Rollover (FYR) in sunflower release. 

  • The testing triggered from UI for the ledger with generated amount of defined data set (Data quantity**: 50000 orders with Open status and 50000 orders with Pending status).

• Testing include two different phases of the fiscal year transition process:

  • #1 phase: Test Fiscal Year Rollover - is a risk free simulation pre-step to #2 phase which includes only Finance part of the data (to preview how the rollover settings will affect funds, budgets, and encumbrances).

  • #2 phase: Fiscal Year Rollover - perform a permanent changes. It is the actual process which move financial data, budgets, and encumbrances into the new fiscal year. It includes not only Finance but Orders parts of the data too.

• The purpose of testing is to define duration of those two phases, to do comparison between current and previous results of each phase, to find any trends for resource utilization. To recommend improvements how to make it work better if any confirmed.

  • Previous successful testing for #2 phase took 3 hours and 30 minutes

• Jiras/ links:

  • The previous results reports:

    • https://folio-org.atlassian.net/wiki/x/EIBGIQ

  • The current ticket: https://folio-org.atlassian.net/browse/PERF-1272

Summary

• Running FYR completed successfully.

• Eureka based environment (with CSP 6 v13.0.10 - https://folio-org.atlassian.net/browse/MODORDERS-1388 ): 

  • #1 Phase Test FYR took 1 hours 18 minutes, completed successfully.

  • #2 Phase actual FYR took 2 hours 15 minutes, completed successfully. During this phase budgetsClosingRollover and finanacialRollover successed in 2 hours and 7 minutes and ordersRollover took 8 minutes.

• Service CPU utilization for both phases was under 8% on eureka environment except a spike at ordersRollover stage of test rollover (mod-orders-storage - 51%, mod-finance-storage 39%).

• DB CPU usage was on level of 34% (db.r7g.xlarge) during both phases of FYR. Previously on quesnelia release xlarge db used 39%.

• No memory leaks were found.

• FYR retesting with default memory for mod-finance, mod-finance-storage completed successfully with the same phases durations.

Recommendations & Jiras

• To solve UI issues on budget and transaction tabs memory allocated to:

  • mod-finance and mod-finance-storage: "memory": 2048, "memoryReservation": 1740, -Xmx1536m

  • sidecar-mod-finance, mod-finance-storage: "memoryReservation": 870, -Xmx768m

Errors

• "POST /finance-storage/ledger-rollovers HTTP_1_1" 504 0 rt=400.006 uct="" This error happen when FYR triggered from UI and doesn’t affect FYR itself. To get status for the processes use Logs tab.

• finance/budget/{budget_UUID}/view 2026-03-17 14:45:15,365 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-0) Connection reset {
  "errors" : [ {
  "message" : "Failed to proxy request: response timeout",
  "type" : "InternalServerErrorException",
  "code" : "service_error",
  "parameters" : [ {
  "key" : "cause",
  "value" : "Connection reset"
  } ]
  } ],
  "total_records" : 1
  }

Test Runs and Results

Data size: 100K orders (50K orders in open status, 50K - pending), 200K order lines, 400K transactions. 

*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing

*Results are taken from the previous test report. Details can be found at the link: [Poppy] Fiscal year rollover testing

*Results are taken from the previous test report. Details can be found at the link: https://folio-org.atlassian.net/wiki/x/EIBGIQ

**Data structure can be found at the link: Fiscal year rollover testing

*** Tests carried out with raised memory for mod-finance and mod-finance-storage to solve UI issues on Budget and Transactions tabs.

Instance CPU Utilization

This graph shows how instances utilize CPU during the tests.

Open

Service CPU Utilization

Test FYR & FYR

Data transition process happen on database side and it explains low cpu utilization from services. Maximum CPU utilization by services during tests were not over 8% in all phases except of spikes on ordersRollover process mod-finance-storage - 39%, mod-orders-storage - 51%

Open

Memory Utilization

Test FYR & FYR

This graph shows memory usage during all phases of FYR. After tests completed no signs of memory growing trends for mod-finance-storage.

Open

Memory usage stayed on the same level after tests stopped.

Open

Database metrics

DB CPU Utilization

Test FYR & FYR

DB CPU usage during all phases: 34%. Except of spike to 78% at the last ordersRollover step.

Open

DB Connections

Test FYR & FYR

DB connections stay stable on 1300 level during all phases of FYR.

Open

DB Load

Test FYR & FYR

Open

Top SQL

Test FYR & FYR

Open

DB metrics (additional)

Transactions per second - under 5, spike at the last ordersRollover step - 19.

Open

CPU utilization

• (08:15–11:55) Dominated by nice (~30%)

  • Instance is moderately loaded

  • Most CPU is from low-priority/background processes

• Spike at ~12:00

  • ordersRollover process

Open

This graph show there’re no deadlocks

Open

Appendix

Infrastructure

PTF -environment SELCTLS2 - TLS

Folio

• 13 r7i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• db.r7g.xlarge database instance, writer

• MSK - ptf-loc-1

  • 6 kafka.m7g.large brokers in 2 zones (3 per zone)

  • Apache Kafka version 3.9.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=2

• Open Search ptf-loc:

  • Data node - r7g.2xlarge.search (4)

  • Master node - m7g.large.search (3)

• DB instance parameter group

  fse-aurora-postgresql16-7k-db-connection - 7000 db connections instead of 5000 in default parameter group (the only difference).

Dataset

Finance data prepared in accordance to recommendation

• oganizations - 1000

• orders - 100000

• polines - 200000

• pieces - 400000

Other data

Modules