This performance testing initiative is designed to assess the system's stability and performance degradation when handling multiple concurrent, resource-intensive workflows across an increasing number of tenants. The primary goal is to identify the saturation point where system resources become critical, leading to unacceptable performance or instability. All test were performed on Sunflower Eureka Galileo environment- SEGCON.
Ticket:PTF-Test Heavy Workflows on multiple tenants concurrently[Galileo enviroment].Preview

1. The performance assessment involved executing a set of Heavy Workflows (Data Import, Data Export, Bulk Edit, and OAI-PMH) with number of Virtual User from 1 to 9 alongside 8VU doing CICO on configurations that scaled both the DB (from 2xl to 4xl) and the Kafka Broker (from xlarge to 2xlarge). To imprope performance we increased the load limits (x4 and x8 from defaults values), the shared buffer (+30%), and 8 tasks to main modules mod-inventory,mod-inventory-storage, mod-srm and mod-srs. With growing number of VU, each VU was working on another tenant as showed in the table test runs.

• Data Import Performance

  • With increasing workload (HW1 → HW6), execution time grew from 10 min to ~90 min. Thus, we can see a correlation: a twofold increase in the number of DI causes a twofold increase in duration.

  • Tests HW6 and HW7 indicated a bottleneck in the Kafka broker, as its CPU utilization reached 70%. Consequently, we decided to increase the instance type in the subsequent test to address this limitation.

  • After upgrading Kafka broker and DB instances (HW7–HW9), execution stabilized on the level of HW5 and was equal to DI dration of 6 VU

  • Significant performance improvements were observed after: increasing DB instance sizes and increasing kafka broker instance sizes

  • raising loadLimit ×4 and x8, increasing shared_buffers by 30%, enabling 8 parallel tasks didn`t show improvements

• The Check-In/Check-Out (CICO) workflow, consistently executed with 8 virtual users (VU) across all testing phases (HW1 through HW11), served as the critical metric for measuring the impact of heavy background workflows on key workflow. CICO responce times experienced a degradation as the concurrency of the Heavy Workflows (DI, DE, BE, OAI-PMH) increased.

  • Baseline (HW1): CICO transactions had very good responce times, with Check-In at 1.9 sec and Check-Out at 2.4 sec.

  • Saturation Point (HW6-HW8): As the number of concurrent heavy workflows reached the maximum of 9 per type across multiple tenants, CICO latency increase in 4 times: Check-In peaked at 7.0 sec Check-Out peaked at 10.0 sec (HW8).

  • DB Bottleneck (HW8): The peak latencies of 7.0 sec (Check-In) and 10.0 sec (Check-Out) occurred in HW8, where the system was under maximum load (9 tenants) with a scaled Kafka Broker (2xl) but the initial DB size (2xl). This reinforces the finding that database load (Bulk Edit, Data export, OAI-PMH and Data Import) was the primary constraint, significantly impacting CICO performance.

  • Scaling and Tuning (HW9-HW11): CICO performance saw a substantial recovery when both the DB and application were optimized:

    • In HW9 (DB 4xl, Kafka 2xl), latencies slightly improved to 1.7 sec (Check-In) and 3.4 sec (Check-Out).

    • In the final, most optimized tests (HW10/HW11), where application limits and task allocation were tuned, CICO stabilized at a near-baseline level: 1.7–2.1 sec (Check -In) and 3.7–4.7 sec (Check-Out).

  • The Bulk Edit (BE) workflow, particularly the Holdings Edit operation, proved to be the most demanding process, exhibiting a nearly 13-fold increase in latency (up to 1027 seconds). Data Export (DE) performance severely degraded, with Custom Profile times surging five-fold and eventually failing to start, underscoring its vulnerability to resource exhaustion(we think this is because 1000% and higher CPU utilization on data-export module). Meanwhile, the OAI-PMH process, though not experiencing extreme latency, however we can see 3 times degradation on key requests

• This perf.testing focuses on the impact of infrastructure scaling and application tuning under the high load, on the execution time of 50K Data Imports concurrently across 9 tenants (DI-9P-T1 through DI-9P-T6).

  Initial Infrastructure Baseline (DI-9P-T1 & T2)

Test setup Heavy Workflows #1

Test setup Data Import #2
Data Import (50K create imports) in parralel on 9 tenants

This table contains testing results for “Test setup Heavy Workflows #1“