Overview

This document contains the results of testing Fiscal year rollover in scope of ticket PERF-657 - Getting issue details... STATUS .

Summary

Test rollover duration was 3.5 hours, finished successfully.
Fiscal year rollover duration was 29 hours, finished successfully.
Service CPU utilization for both tests was less than 10% for all the modules, except a spike at the beginning of test rollover (mod-finance CPU load reached 42%).
DB CPU usage reached 60% during the test rollover and 99% during the rollover itself.
No memory leaks were found.

Test Runs and Results

Test #

Scenario

Load level

Data quantity**

Fiscal year rollover duration

Orchid*

Fiscal year rollover duration

Poppy

Comments

Test Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - closed

200K order lines

400K transactions

3.5 hours

Test rollover finished successfully.

Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - closed

200K order lines

400K transactions

6 days (failed due to 504 Gateway Time-out" error)

29 hours

Rollover finished successfully.

*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing

**Data structure can be found at the link: Fiscal year rollover testing

Instance CPU Utilization

Test FY rollover

FY rollover

Service CPU Utilization

Test FY rollover

Maximum CPU utilization by modules:

mod-orders - 1.5%

mod-orders-storage - 1.5%

mod-finance - 42% (spike at the beginning)

mod-finance storage - 8%

FY rollover

Maximum CPU utilization by modules:

mod-orders - 2%

mod-orders-storage - 6%

mod-finance - 7%

mod-finance storage - 2%

Memory Utilization

Test FY rollover

Memory usage by modules:

mod-finance-storage - 27% → 50%

mod-finance - 29% → 33%

FY rollover

Memory usage by modules:

mod-orders-storage - 34% → 42%

mod-finance storage - 30% → 51%

DB CPU Utilization

Test FY rollover

Maximum DB CPU usage - 60%

FY rollover

Maximum DB CPU usage - 99%

DB Connections

Test FY rollover

FY rollover

DB Load

Test FY rollover

FY rollover

Top SQL

Test FY rollover

autovacuum: VACUUM fs09000000_mod_inventory_storage.instance (to prevent wraparound)

DO
$$
begin
 PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"1d92b38f-214a-4a9c-bbf1-3aaaeb5e8659","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Preview","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAl

FY rollover

autovacuum: VACUUM ANALYZE fs09000000_mod_inventory_storage.instance (to prevent wraparound)

 SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN  ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent('Ongoing'))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE lower(f_unaccent('Ongoing')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE left(lower(f_unaccent('Ongoing')),600) AND lower(f_unaccent(purchase_order.jsonb->>'orderType')) LIKE lower(f_unaccent('Ongoing')) END)

 SELECT fs09000000_mod_orders_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN  ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent(''Ongoing''))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE lower(f_unaccent(''Ongoing'')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE left(lower(f_unaccent(''Ongoing'')),600) AND lower(f_unaccent(purchase_ord

DO
$$
begin
 PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"006b9433-28a9-4e14-a4fb-2e62fcd3edcd","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Commit","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAll

 SELECT jsonb FROM fs09000000_mod_permissions.permissions

 INSERT INTO fs09000000_mod_audit.acquisition_order_line_log (id, action, order_id, order_line_id, user_id, event_date, action_date, modified_content_snapshot) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)

 UPDATE fs09000000_mod_orders_storage.po_line SET jsonb = $1 WHERE id = ($1::jsonb->>?)::uuid

 autovacuum: VACUUM fs09000000_mod_orders_storage.outbox_event_log

 autovacuum: VACUUM fs09000000_mod_orders_storage.po_line

 SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module pcp1-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-finance-storage	9	8.5.0	2	1024	896	1024	700	88	128	false
mod-orders-storage	9	13.6.0	2	1024	896	512	700	88	128	false
mod-finance	9	4.8.0	2	1024	896	128	700	88	128	false
mod-orders	9	12.7.0	2	2048	1440	1024	1024	384	512	false
nginx-okapi	9	2023.06.14	2	1024	896	128	0	0	0	false
okapi-b	11	5.1.2	3	1684	1440	1024	922	384	512	false
pub-okapi	9	2023.06.14	2	1024	896	128	768	0	0	false

Methodology/Approach

Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Browser not supported