PTF-Test Hybrid DB Deployment on multiple tenants[Galileo environment]. Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarg. Part 3

PTF-Test Hybrid DB Deployment on multiple tenants[Galileo environment]. Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarg. Part 3

 

Overview

This report details the performance evaluation of a Dual-Provisioned Database architecture within a multi-tenant FOLIO environment (Galileo release). The tested architecture features a Provisioned Primary DB (db.r7g.2xlarge) paired with a Dedicated Offload DB (db.r8g.2xlarge). 7 highly intensive, I/O-heavy modules - mod-source-record-manager, mod-source-record-storage, mod-inventory-storage, mod-orders-storage, mod-data-export, mod-lists, and mod-fqm-manager were routed to the Dedicated Offload DB(db.r8g.2xlarge). The objective of this testing phase is to determine if a dual-provisioned hardware baseline canuccessfullytabilize heavy Data Import, Data Export) without degrading real-time Circulation (CICO) and ListApp workflows across 60 tenants.
Ticket: https://folio-org.atlassian.net/browse/PERF-1334

 

Summary:

Test 3 -tandard Load: Under thetandard load profile, the Dual-Provisionedetup easily outperforms the previouserverless baseline. Data Import (50K) matched its optimal 18-minute benchmark, Data Export completed in 17utes, and CICO operations reduced by about 1econd. This confirms that the r8g.2xlarge instance can handle the concurrent load of all 7 heavy modules under normal operating conditions.

Test 4 applied a 3x load multiplier totress-test the architecture's breaking point. CICO achieved its fastest times yet (Check-In at 2.40econds; Check-Out at 3.50econds). Data Import execution degraded to 57utes. Based on the Performance Insight graph foregcon-perf1334-2nddb (Dedicated Offload DB, db.r8g.2xlarge), the primary bottleneck identified is IO: DataFileRead, which dominated the wait events throughout the test, accounting for approximately 70–80% of Average Activeessions (AAS). CPU utilization remained negligible throughout, confirming that compute was not the limiting factor. The degradation observed in Test 4 results — particularly the Data Import regression (18 → 57) is consistent with this I/Oaturation pattern on the Offload DB.

Test 5 applied a verticallycaled Dedicated Offload DB (db.r8g.4xlarge) under a high load profile (60 tenants) to determine if doubling the hardware capacity could mitigate the bottlenecks observed in Test 4. Despite the increased CPU and memory, the verticalcaling yielded no meaningful performance recovery. CICO remainedtable (Check-In at 3.12econds; Check-Out at 4.37econds), also Data Import duration 59utes, and ListApp didin`t fail and latency reached 8utes. This confirms that throwing more physical hardware at the Offload DB does not resolve the degradation.

 

Recommendations & Jiras

Test Runs 

The first two tests (Test 1 and Test 2),hown in the test results table, are from previous testing. Link to the report:

Test 1- Report
Test 2 - Report

Test 1. Baseline r7g 4xl;

Test 2. Baseline r7g 2xl +L 5-64ACU;

Test 3.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarge.

  • Load model: Testcenario 1: Medium load.

  • Routing: 7 modules (srm,rs, inv-strg, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8g.2xlarge.

  • Number of tenants: 60;

Test 4.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarge.

  • Load model: Testcenario 2: High load.

  • Routing: 7 modules (srm,rs, inv-strg, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8g.2xlarge.

  • Number of tenants: 60;

Test 5.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarge.

  • Load model: Testcenario 2: High load.

  • Routing: 7 modules (srm,rs, inv-strg, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8g.2xlarge.

  • Number of tenants: 60;

Test 6.

  • Architecture: Provisioned primary DB db.r7g.4xlarge

  • Load model: Testcenario 3: MOBIUS Test.

Test 7.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarge.

  • Load model: Testcenario 3: MOBIUS Test.

  • Routing: 7 modules (srm,rs, inv-strg, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8g.2xlarge.

Test 8.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8g.2xlarge.

  • Load model: Testcenario 3: MOBIUS Test.

  • Routing: 7 modules (srm,rs,earch, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8g.2xlarge.

Test 9.

  • Architecture: Provisioned primary DB db.r7g.2xlarge paired with Dedicated Offload DB db.r8gd.2xlarge.

  • Load model: Testcenario 3: MOBIUS Test.

  • Routing: 7 modules (srm,rs, inv-strg, ord-strg, data-export, lists, and fqm-manager) routed to Dedicated Offload DB db.r8gd.2xlarge.