Overview

This document contains the results of testing Fiscal year rollover in scope of ticket PERF-289 - Getting issue details... STATUS .

Summary

Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum. More details...
DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%). More details...
Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover. More details...
"Out of shared memory" error occurred during 100K orders rollover with 90K open orders. This issue was reported previously in MODFISTO-428 - Getting issue details... STATUS .
"504 Gateway Time-out" error occurred during 100K orders rollover with 50K open orders. This issue was reported and marked as fixed previously ( MODORDERS-865 - Getting issue details... STATUS ) but should be investigated further.
A set of improvements was prepared by dev team: MODFISTO-439 - Getting issue details... STATUS . When these fixes are implemented, tests will be conducted again to assess performance and compare results with current ones.

Recommendations and Jiras

Investigate application behavior during fiscal year rollover: MODORDERS-916 - Getting issue details... STATUS

Investigate "Out of shared memory" and "504 Gateway Time-out" errors in scope of previously created tickets.

Conduct another set of tests when all the fixes are implemented.

Test Runs and Results

Test #	Scenario	Load level	Data quantity	Fiscal year rollover duration	Comments
1.	Fiscal year rollover	1 rollover at a time	10K orders: 5K - open 5K - closed	3 hours (2 hours financial rollover, 1 hour orders rollover)	No errors
2.			50K orders: 40K - open 10K - closed	2 days 6 hours (1 day 8 hours financial rollover, 22 hours orders rollover)	No errors
3.			100K orders: 90K - open 10K - closed	8 days (6 days 6 hours financial rollover, 1 day 18 hours orders rollover)	Rollover was stopped manually, "Out of shared memory" error occurred.
4.			100K orders: 50K - open 50K - closed	6 days 13 hours (3 days 9 hours financial rollover, 3 days 4 hours orders rollover)	Rollover finished by itself, "504 Gateway Time-out" error occurred.

Service CPU Utilization

Service CPU utilization for mod-orders-storage increased up to 60% during orders phase of 10K rollover, while during 50K and 100K orders rollover mod-orders-storage CPU utilization reaches only 20% maximum.

10K orders

According to the graph, mod-orders-storage is about 40-60% during orders rollover stage.

50K orders

According to the graph, mod-orders-storage is about 15-20% during orders rollover stage.

Also, there is a spike between 06/28 and 06/29 for mod-finance-storage module.

100K orders (90K - open, 10K - closed)

There is a spike for mod-orders and mod-orders-storage at the end of the rollover.

100K orders (50K - open, 50K - closed)

According to the graph, mod-orders-storage CPU utilization is about 10% during orders rollover stage.

Memory Utilization

10K orders

50K orders

There is an increase of service memory usage from 44% to 65% for mod-finance-storage.

100K orders (90K - open, 10K - closed)

100K orders (50K - open, 50K - closed)

Service memory usage increase by services:

mod-finance-storage - increase from 37% to 59%;
okapi - increase from 44% to 57%;
mod-orders-storage - increase from 42% to 54%;
mod-orders - increase from 59% to 64%;
mod-finance - increase from 40% to 44%.

DB CPU Utilization

DB CPU utilization shows different behavior for rollovers of different data quantity. During 10K and 50k rollover DB CPU utilization increased during orders phase (31% → 83%, 61% → 85%), while during 100K rollover DB CPU utilization decreased during orders phase (65% → 30%, 67% → 40%).

10K orders

According to the graph, CPU utilization during finance part was about 31%, during orders part - 83%.

50K orders

According to the graph, CPU utilization during finance part was 63%, during orders part - 85%.

100K orders (90K - open, 10K - closed)

According to the graph, CPU utilization during finance part was 65%, during orders part - 30%.

100K orders (50K - open, 50K - closed)

According to the graph, CPU utilization during finance part was 67%, during orders part - 40%.

DB Connections

10K orders

50K orders

100K orders

100K orders (50K - open, 50K - closed)

Database load

Lock wait type was observed during orders phase of rollover. Lock waiting time is higher during 10K orders rollover than during 50K and 100K orders rollover.

SQL query causing locking:

SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE

10K orders

50K orders

100K orders (90K - open, 10K - closed)

100K orders (50K - open, 50K - closed)

Top SQL

10K orders

50K orders

100K orders (90K - open, 10K - closed)

100K orders (50K - open, 50K - closed)

Appendix

Infrastructure

PTF -environment ncp5

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters:

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-orders	12.6.6	10	2	1024	2048	1440	512	1024
mod-orders-storage	13.5.0	5	2	128	1024	896	128	768
mod-finance	4.7.1	5	2	128	1024	896	128	768
mod-finance-storage	8.4.1	8	2	128	1024	896	128	700
okapi	5.0.1	6	3	1024	1684	1440	512	922
nginx-okapi	2022.03.02	6	2	128	1024	896	-	-
pub-okapi	2022.03.02	6	2	128	1024	896	-	768

Methodology/Approach

Before each fiscal year rollover data should be prepared.

Before data importing next schemas should be truncated:

mod_finance_storage.budget
mod_finance_storage.transaction
mod_finance_storage.budget_expense_class
mod_finance_storage.expense_class
mod_finance_storage.fund
mod_finance_storage.fund_type
mod_finance_storage.fiscal_year
mod_finance_storage.ledger
mod_finance_storage.group_fund_fiscal_year
mod_orders_storage.purchase_order
mod_orders_storage.po_line
mod_orders_storage.titles
mod_notes.note

To import data (100K orders: 50% open, 50% closed) backup of DB schemas was created. It can be found in "Finance" folder on the host.

For other orders quantity and data structure JMeter script should be used (perf-testing/workflows-scripts/fiscal_year/FY_rollover_preparation on the GitHub). But in this case data generation will take a lot of time.

Details on data preparation can be found at the link: Steps for testing process#Fiscalyearrollover

Browser not supported