[Poppy] Fiscal year rollover testing
- 1 Overview
- 2 Summary
- 3 Test Runs and Results
- 4 Instance CPU Utilization
- 4.1 Test FY rollover
- 4.2 FY rollover
- 5 Service CPU Utilization
- 5.1 Test FY rollover
- 5.2 FY rollover
- 6 Memory Utilization
- 6.1 Test FY rollover
- 6.2 FY rollover
- 7 DB CPU Utilization
- 7.1 Test FY rollover
- 7.2 FY rollover
- 8 DB Connections
- 8.1 Test FY rollover
- 8.2 FY rollover
- 9 DB Load
- 9.1 Test FY rollover
- 9.2 FY rollover
- 10 Top SQL
- 10.1 Test FY rollover
- 10.2 FY rollover
- 11 Appendix
- 11.1 Infrastructure
- 11.2 Methodology/Approach
Overview
This document contains the results of testing Fiscal year rollover in scope of ticket 
Summary
Test rollover duration was 3.5 hours, finished successfully.
Fiscal year rollover duration was 29 hours, finished successfully.
Service CPU utilization for both tests was less than 10% for all the modules, except a spike at the beginning of test rollover (mod-finance CPU load reached 42%).
DB CPU usage reached 60% during the test rollover and 99% during the rollover itself.
No memory leaks were found.
Test Runs and Results
Test # | Scenario | Load level | Data quantity** | Fiscal year rollover duration Orchid* | Fiscal year rollover duration Poppy | Comments |
|---|---|---|---|---|---|---|
1 | Test Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - closed 200K order lines 400K transactions | - | 3.5 hours | Test rollover finished successfully. |
2 | Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - closed 200K order lines 400K transactions | 6 days (failed due to 504 Gateway Time-out" error) | 29 hours | Rollover finished successfully. |
*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing
**Data structure can be found at the link: Fiscal year rollover testing
Instance CPU Utilization
Test FY rollover
FY rollover
Service CPU Utilization
Test FY rollover
Maximum CPU utilization by modules:
mod-orders - 1.5%
mod-orders-storage - 1.5%
mod-finance - 42% (spike at the beginning)
mod-finance storage - 8%
FY rollover
Maximum CPU utilization by modules:
mod-orders - 2%
mod-orders-storage - 6%
mod-finance - 7%
mod-finance storage - 2%
Memory Utilization
Test FY rollover
Memory usage by modules:
mod-finance-storage - 27% → 50%
mod-finance - 29% → 33%
FY rollover
Memory usage by modules:
mod-orders-storage - 34% → 42%
mod-finance storage - 30% → 51%
DB CPU Utilization
Test FY rollover
Maximum DB CPU usage - 60%
FY rollover
Maximum DB CPU usage - 99%
DB Connections
Test FY rollover
FY rollover
DB Load
Test FY rollover
FY rollover
Top SQL
Test FY rollover
autovacuum: VACUUM fs09000000_mod_inventory_storage.instance (to prevent wraparound)
DO
$$
begin
PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"1d92b38f-214a-4a9c-bbf1-3aaaeb5e8659","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Preview","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAlFY rollover
autovacuum: VACUUM ANALYZE fs09000000_mod_inventory_storage.instance (to prevent wraparound)
SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent('Ongoing'))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE lower(f_unaccent('Ongoing')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE left(lower(f_unaccent('Ongoing')),600) AND lower(f_unaccent(purchase_order.jsonb->>'orderType')) LIKE lower(f_unaccent('Ongoing')) END)
SELECT fs09000000_mod_orders_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent(''Ongoing''))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE lower(f_unaccent(''Ongoing'')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE left(lower(f_unaccent(''Ongoing'')),600) AND lower(f_unaccent(purchase_ord
DO
$$
begin
PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"006b9433-28a9-4e14-a4fb-2e62fcd3edcd","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Commit","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAll SELECT jsonb FROM fs09000000_mod_permissions.permissions INSERT INTO fs09000000_mod_audit.acquisition_order_line_log (id, action, order_id, order_line_id, user_id, event_date, action_date, modified_content_snapshot) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) UPDATE fs09000000_mod_orders_storage.po_line SET jsonb = $1 WHERE id = ($1::jsonb->>?)::uuid autovacuum: VACUUM fs09000000_mod_orders_storage.outbox_event_log autovacuum: VACUUM fs09000000_mod_orders_storage.po_line SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATEAppendix
Infrastructure
PTF -environment pcp1
10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
MSK tenant
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
|---|---|---|---|---|---|---|---|---|---|---|
mod-finance-storage | 9 | 2 | 1024 | 896 | 1024 | 700 | 88 | 128 | false | |
mod-orders-storage | 9 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 | false | |
mod-finance | 9 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | false | |
mod-orders | 9 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 | false | |
nginx-okapi | 9 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | false | |
okapi-b | 11 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | false | |
pub-okapi | 9 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | false |
Methodology/Approach
Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover
The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.