...

Test status: COMPLETED WITH ERRORS

Table of Contents
Overview

...

IN PROGRESS

Regression testing of Fiscal Year Rollover (FYR)

...

on Eureka based environment in quesnelia release.
- The testing triggered from UI for the ledger with generated amount of defined data set (Data quantity).
Testing include two different phases of the fiscal year transition process:
- #1 phase: Test Fiscal Year Rollover - is a risk free simulation pre-step to #2 phase which includes only Finance part of the data (to preview how the rollover settings will affect funds, budgets, and encumbrances).
- #2 phase: Fiscal Year Rollover - perform a permanent changes. It is the actual process which move financial data, budgets, and encumbrances into the new fiscal year. It includes not only Finance but Orders parts of the data too.
The purpose of testing is to define duration time of those 2 phases, to do comparison between current and previous results of each phase, to find any trends for resource utilization. To recommend improvements how to make it work better if any confirmed.
- Previous successful testing for #2 phase took 29 hours
Jiras/ links:
- The previous results report: [Poppy] Fiscal year rollover testing
- The current ticket:
  Jira Legacy
  server System Jira
  serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
  key PERF-1016

...

Summary

Eureka - failed completed with errors
- Test fiscal year rollover duration was #1 Phase Test FYR took 3 hours 35 minutes, finished completed successfully.
- Fiscal year rollover duration was #2 Phase actual FYR took 8 hours and failed completed with errors (Order rollover Error).
Non-eureka - successful
- Test FYR - 3 hours 20 minutes.
- FYR - 9 hours 7 minutes
Service CPU utilization for pre-strop to Fiscal Year Rollover (FYR) - Test FYR was less than 5% for all the modules, except a spike at the beginning of test rollover (mod-invoice reached 10%, mod-invoice-storage - 8%). Some modules were steady - mod-inventory - 11%, mod-pubsub - 8%.
DB CPU usage was on level of 9% during the test rollover and there was growing trend from 9% to 15% during the rollover itself.
No memory leaks were found.

Recommendations

Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
Investigation of issues on Eureka based environment
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODFISTO-495
- Time out issue
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key EUREKA-518
- UI pending issue

Errors

127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with this amount of data. Probably it may cause fail of FYR.

Test Runs and Results

Test #

Scenario

Load level

Data quantity**

Fiscal year rollover duration

Orchid*

Fiscal year rollover duration

Poppy*

FYR duration

Quesnelia (Eureka)

FYR duration

Quesnelia (non-Eureka)

Comments

1

Test Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - pending

200K order lines

400K transactions

-

3.5 hours

3 hours 20 minutes

Test rollover finished successfully.

2

Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - pending

200K order lines

400K transactions

6 days (failed due to 504 Gateway Time-out" error)

29 hours

8 hours

(failed due

500 Internal Server Error)

9 hours 7 minutes

On eureka Rollover failed.

On non-eureka Rollover succeeded.

...

**Data structure can be found at the link: Fiscal year rollover testing

Instance CPU Utilization

Test FY rollover

no graph

FY rollover

no graph

Service CPU Utilization

Test FY rollover

Maximum CPU utilization by modules:

...

mod-orders, mod-orders-storage - 1,5%, 3%

mod-finance - under 1%

FY rollover

Maximum CPU utilization by modules:

...

mod-finance storage - 1.2%

Memory Utilization

Test FY rollover

Memory usage by modules:

mod-finance-storage - 85%

...

mod-orders - 57%

mod-orders-storage - 46%

FY rollover

Memory usage by modules:

mod-finance-storage - 85%

...

mod-orders - 58%

mod-orders-storage - 46%

DB CPU Utilization

Test FY rollover (qelc2)

Maximum DB CPU usage - 9 - 20%

Test FY rollover (qcon)

FY rollover (qelc2)

Maximum DB CPU usage - 99%

FY rollover (qcon)

Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY cs00000int_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.

DB Connections

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

mod-serials was not paused for the test (because of that we see such connection number increase)

DB Load

Test FY rollover (qcon)

FY rollover (qcon)

Top SQL

Test FY rollover (qcon)

FY rollover (qcon)

Appendix

Infrastructure

PTF -environment qelc2

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.4xlarge
32 GiB 4 vCPUs -
MSK fse-tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 3.7.x
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

PTF -environment qcon

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs -

...

Expand

title	Module QCON

Cluster Resources - qcon-pvt (Wed Nov 06 11:28:29 UTC 2024)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-finance-storage	1	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-ncip	1	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-agreements	1	mod-agreements:7.0.0	2	1592	1488	128	0	0	0
mod-ebsconet	1	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-consortia	1	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
mod-organizations	1	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-serials-management	1	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512
mod-settings	2	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-search	9	mod-search:3.3.0-SNAPSHOT.224	2	2592	2480	2048	1440	512	1024
edge-dematic	1	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-data-import	1	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
mod-tags	1	mod-tags:2.2.0	2	1024	896	128	768	88	128
mod-authtoken	6	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-inventory-update	1	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	1	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	1	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	3	mod-orders-storage:13.7.4	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	1	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-password-validator	1	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-licenses	1	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	1	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-bulk-operations	1	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	1	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	1	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	1	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	1	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	1	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	1	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	1	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	1	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	1	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	4	mod-permissions:6.6.0-SNAPSHOT.379	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	3	mod-orders:12.8.8	2	2048	1740	1024	1024	384	512
edge-patron	1	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	1	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-invoice	5	mod-invoice:5.8.2	2	2880	2304	2	2048	88	128
mod-inventory-storage	2	mod-inventory-storage:27.2.0-SNAPSHOT.738	2	4096	3690	2048	3076	384	512
mod-user-import	1	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	1	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-source-record-storage	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-calendar	1	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-event-config	1	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	1	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	3	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.2.2	2	1024	896	128	768	88	128
edge-orders	1	edge-orders:3.0.0	2	1024	896	128	768	88	128
edge-rtac	1	edge-rtac:2.7.1	2	1024	896	128	768	88	128
mod-users	3	mod-users:19.3.1	2	2048	1792	128	1024	88	256
mod-template-engine	1	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	1	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	4	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	2	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	3	mod-invoice-storage:5.8.1	2	1872	1536	2	1024	384	512
mod-reading-room	7	mod-reading-room:1.0.0-SNAPSHOT.11	2	1024	896	128	768	88	128
mod-service-interaction	1	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-patron	1	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-data-export	1	mod-data-export:5.0.0	1	2048	1524	1024	0	0	0
mod-oai-pmh	1	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-notes	1	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-login	2	mod-login:7.11.0	2	2048	1792	1024	1024	88	256
mod-organizations-storage	1	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	1	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Methodology/Approach

Files

View file
name PERF_865_FYR_results.xlsx
height 250

Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
Consider using prepared scripts and queries in S3 bucket - Buckets/fse-ptf/FYR/

The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.

...

Version	Old Version 7	New Version 8
Changes made by	Leonid Kolesnykov	Leonid Kolesnykov
Saved on	Nov 11, 2024	Nov 14, 2024

Versions Compared

Key

Test status: COMPLETED WITH ERRORS

Table of Contents
Overview

IN PROGRESS

Summary

Recommendations

Errors

Test Runs and Results

Instance CPU Utilization

Test FY rollover

FY rollover

Service CPU Utilization

Test FY rollover

FY rollover

Memory Utilization

Test FY rollover

FY rollover

DB CPU Utilization

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

DB Connections

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

DB Load

Test FY rollover (qcon)

FY rollover (qcon)

Top SQL

Test FY rollover (qcon)

FY rollover (qcon)

Appendix

Infrastructure

Methodology/Approach

Name	Memory GIB	vCPUs	max_connections
db.r6g.4xlarge	32 GiB	4 vCPUs	-

Page Comparison

Versions Compared

Key

<span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-0">[data-colorid=</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-0">mvickmjl1r</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-1">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-2">mvickmjl1r</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-3">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-4">ly2s53pguk</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-5">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-6">ly2s53pguk</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-7">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-8">gj6znaa5l4</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-9">]{color:</span>

Table of ContentsOverview

IN PROGRESS

Summary

Recommendations

Errors

Test Runs and Results

Instance CPU Utilization

Test FY rollover

FY rollover

Service CPU Utilization

Test FY rollover

FY rollover

Memory Utilization

Test FY rollover

FY rollover

DB CPU Utilization

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

DB Connections

Test FY rollover (qelc2)

Test FY rollover (qcon)

FY rollover (qelc2)

FY rollover (qcon)

DB Load

Test FY rollover (qcon)

FY rollover (qcon)

Top SQL

Test FY rollover (qcon)

FY rollover (qcon)

Appendix

Infrastructure

Methodology/Approach

Table of Contents
Overview