/
[Poppy] Fiscal year rollover testing

[Poppy] Fiscal year rollover testing

Dec 14, 2023

Overview

This document contains the results of testing Fiscal year rollover in scope of ticket PERF-657: [Poppy] Fiscal year rollover re-testingClosed.

Summary

  • Test rollover duration was 3.5 hours, finished successfully.

  • Fiscal year rollover duration was 29 hours, finished successfully.

  • Service CPU utilization for both tests was less than 10% for all the modules, except a spike at the beginning of test rollover (mod-finance CPU load reached 42%).

  • DB CPU usage reached 60% during the test rollover and 99% during the rollover itself.

  • No memory leaks were found.

Test Runs and Results

Test #

Scenario

Load level

Data quantity**

Fiscal year rollover duration

Orchid*

Fiscal year rollover duration

Poppy

Comments

Test #

Scenario

Load level

Data quantity**

Fiscal year rollover duration

Orchid*

Fiscal year rollover duration

Poppy

Comments

1

Test Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - closed

200K order lines

400K transactions

-

3.5 hours

Test rollover finished successfully.

2

Fiscal year rollover

1 rollover at a time

100K orders:

50K - open

50K - closed

200K order lines

400K transactions

6 days (failed due to 504 Gateway Time-out" error)

29 hours

Rollover finished successfully.

*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing

**Data structure can be found at the link: Fiscal year rollover testing

Instance CPU Utilization

Test FY rollover

FY rollover

Service CPU Utilization

Test FY rollover

Maximum CPU utilization by modules:

mod-orders - 1.5%

mod-orders-storage - 1.5%

mod-finance - 42% (spike at the beginning)

mod-finance storage - 8%

FY rollover

Maximum CPU utilization by modules:

mod-orders - 2%

mod-orders-storage - 6%

mod-finance - 7%

mod-finance storage - 2%

Memory Utilization

Test FY rollover

Memory usage by modules:

mod-finance-storage - 27% → 50%

mod-finance - 29% → 33%

FY rollover

Memory usage by modules:

mod-orders-storage - 34% → 42%

mod-finance storage - 30% → 51%

DB CPU Utilization

Test FY rollover

Maximum DB CPU usage - 60%

FY rollover

Maximum DB CPU usage - 99%

DB Connections

Test FY rollover

FY rollover

DB Load

Test FY rollover

FY rollover

Top SQL

Test FY rollover

autovacuum: VACUUM fs09000000_mod_inventory_storage.instance (to prevent wraparound)
DO $$ begin PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"1d92b38f-214a-4a9c-bbf1-3aaaeb5e8659","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Preview","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAl

FY rollover

autovacuum: VACUUM ANALYZE fs09000000_mod_inventory_storage.instance (to prevent wraparound)
SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent('Ongoing'))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE lower(f_unaccent('Ongoing')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE left(lower(f_unaccent('Ongoing')),600) AND lower(f_unaccent(purchase_order.jsonb->>'orderType')) LIKE lower(f_unaccent('Ongoing')) END)
SELECT fs09000000_mod_orders_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent(''Ongoing''))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE lower(f_unaccent(''Ongoing'')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE left(lower(f_unaccent(''Ongoing'')),600) AND lower(f_unaccent(purchase_ord
DO $$ begin PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"006b9433-28a9-4e14-a4fb-2e62fcd3edcd","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Commit","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAll
SELECT jsonb FROM fs09000000_mod_permissions.permissions
INSERT INTO fs09000000_mod_audit.acquisition_order_line_log (id, action, order_id, order_line_id, user_id, event_date, action_date, modified_content_snapshot) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
UPDATE fs09000000_mod_orders_storage.po_line SET jsonb = $1 WHERE id = ($1::jsonb->>?)::uuid
autovacuum: VACUUM fs09000000_mod_orders_storage.outbox_event_log
autovacuum: VACUUM fs09000000_mod_orders_storage.po_line
SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, writer/reader

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3



Module
pcp1-pvt

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

Module
pcp1-pvt

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mod-finance-storage

9

8.5.0

2

1024

896

1024

700

88

128

false

mod-orders-storage

9

13.6.0

2

1024

896

512

700

88

128

false

mod-finance

9

4.8.0

2

1024

896

128

700

88

128

false

mod-orders

9

12.7.0

2

2048

1440

1024

1024

384

512

false

nginx-okapi

9

2023.06.14

2

1024

896

128

0

0

0

false

okapi-b

11

5.1.2

3

1684

1440

1024

922

384

512

false

pub-okapi

9

2023.06.14

2

1024

896

128

768

0

0

false

Methodology/Approach

Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover

The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.