[Poppy] Fiscal year rollover testing
- Hanna Tuhuzbayeva
Overview
This document contains the results of testing Fiscal year rollover in scope of ticket PERF-657 - Getting issue details... STATUS .
Summary
- Test rollover duration was 3.5 hours, finished successfully.
- Fiscal year rollover duration was 29 hours, finished successfully.
- Service CPU utilization for both tests was less than 10% for all the modules, except a spike at the beginning of test rollover (mod-finance CPU load reached 42%).
- DB CPU usage reached 60% during the test rollover and 99% during the rollover itself.
- No memory leaks were found.
Test Runs and Results
Test # | Scenario | Load level | Data quantity** | Fiscal year rollover duration Orchid* | Fiscal year rollover duration Poppy | Comments |
|---|---|---|---|---|---|---|
| 1 | Test Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - closed 200K order lines 400K transactions | - | 3.5 hours | Test rollover finished successfully. |
| 2 | Fiscal year rollover | 1 rollover at a time | 100K orders: 50K - open 50K - closed 200K order lines 400K transactions | 6 days (failed due to 504 Gateway Time-out" error) | 29 hours | Rollover finished successfully. |
*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing
**Data structure can be found at the link: Fiscal year rollover testing
Instance CPU Utilization
Test FY rollover
FY rollover
Service CPU Utilization
Test FY rollover
Maximum CPU utilization by modules:
mod-orders - 1.5%
mod-orders-storage - 1.5%
mod-finance - 42% (spike at the beginning)
mod-finance storage - 8%
FY rollover
Maximum CPU utilization by modules:
mod-orders - 2%
mod-orders-storage - 6%
mod-finance - 7%
mod-finance storage - 2%
Memory Utilization
Test FY rollover
Memory usage by modules:
mod-finance-storage - 27% → 50%
mod-finance - 29% → 33%
FY rollover
Memory usage by modules:
mod-orders-storage - 34% → 42%
mod-finance storage - 30% → 51%
DB CPU Utilization
Test FY rollover
Maximum DB CPU usage - 60%
FY rollover
Maximum DB CPU usage - 99%
DB Connections
Test FY rollover
FY rollover
DB Load
Test FY rollover
FY rollover
Top SQL
Test FY rollover
autovacuum: VACUUM fs09000000_mod_inventory_storage.instance (to prevent wraparound)
DO
$$
begin
PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"1d92b38f-214a-4a9c-bbf1-3aaaeb5e8659","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Preview","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAl
FY rollover
autovacuum: VACUUM ANALYZE fs09000000_mod_inventory_storage.instance (to prevent wraparound)
SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent('Ongoing'))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE lower(f_unaccent('Ongoing')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>'orderType')),600) LIKE left(lower(f_unaccent('Ongoing')),600) AND lower(f_unaccent(purchase_order.jsonb->>'orderType')) LIKE lower(f_unaccent('Ongoing')) END)
SELECT fs09000000_mod_orders_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_orders_storage.po_line WHERE (((po_line.purchaseOrderId IN ( SELECT id FROM purchase_order WHERE CASE WHEN length(lower(f_unaccent(''Ongoing''))) <= 600 THEN left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE lower(f_unaccent(''Ongoing'')) ELSE left(lower(f_unaccent(purchase_order.jsonb->>''orderType'')),600) LIKE left(lower(f_unaccent(''Ongoing'')),600) AND lower(f_unaccent(purchase_ord
DO
$$
begin
PERFORM fs09000000_mod_finance_storage.budget_encumbrances_rollover('{"id":"006b9433-28a9-4e14-a4fb-2e62fcd3edcd","ledgerId":"52b500b3-4757-4d06-9fde-be125cc5e738","rolloverType":"Commit","fromFiscalYearId":"66ef2756-cb3c-4773-8c6a-24dd52e9aad8","toFiscalYearId":"74af00ef-4809-4dd1-a56c-c26f1102db37","restrictEncumbrance":true,"restrictExpenditures":true,"needCloseBudgets":true,"currencyFactor":2,"budgetsRollover":[{"rolloverAllocation":true,"rolloverBudgetValue":"Available","setAll
SELECT jsonb FROM fs09000000_mod_permissions.permissions
INSERT INTO fs09000000_mod_audit.acquisition_order_line_log (id, action, order_id, order_line_id, user_id, event_date, action_date, modified_content_snapshot) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
UPDATE fs09000000_mod_orders_storage.po_line SET jsonb = $1 WHERE id = ($1::jsonb->>?)::uuid
autovacuum: VACUUM fs09000000_mod_orders_storage.outbox_event_log
autovacuum: VACUUM fs09000000_mod_orders_storage.po_line
SELECT * FROM fs09000000_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE
Appendix
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
| Module pcp1-pvt | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-finance-storage | 9 | 8.5.0 | 2 | 1024 | 896 | 1024 | 700 | 88 | 128 | false |
| mod-orders-storage | 9 | 13.6.0 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 | false |
| mod-finance | 9 | 4.8.0 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | false |
| mod-orders | 9 | 12.7.0 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 | false |
| nginx-okapi | 9 | 2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | false |
| okapi-b | 11 | 5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | false |
| pub-okapi | 9 | 2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | false |
Methodology/Approach
Before each fiscal year rollover data should be prepared. Data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover
The difference between Fiscal Year Rollover and Test rollover is that test rollover includes only Finance part of the data. Usual rollover includes also Orders part of the data.