Overview

This test aims to validate the stability and High Availability (HA) capabilities of the folio-keycloak module running in clustered mode. Specifically, we are testing the transition from the custom JDBC_PING2 discovery protocol to the standard jdbc-ping. This is an experimental and regression test utilizing an unreleased folio-keycloak image built from a development branch.

The testing will be conducted on the MOBIUS environment, configured with 60 tenants and the MOBIUS dataset, to simulate a realistic multi-tenant Keycloak/Eureka architecture.

Testing Goals:

• Baseline Establishment: Run a mixed-workload scenario (Authorization and roles creation) on the existing JDBC_PING2 setup to capture baseline metrics.

• A/B Comparison: Deploy the custom branch (KEYCLOAK-111) with jdbc-ping and execute the same load. The goal is to verify cluster mode stability, confirm the new mechanism is active, and ensure no undesirable side effects are introduced.

• Failover & Scaling Verification: Observe how Keycloak behaves during dynamic scaling under load. This includes scaling up (adding nodes), scaling down (gracefully removing nodes), and failover testing (forcefully stopping/killing nodes) to verify HA functionality.

Defined SLAs:

• Performance: 0 regression. Response times (RT) with the new jdbc-ping configuration must not degrade compared to the baseline.

• Failover Reliability: Aiming for zero dropped connections (0% errors) or minimal, acceptable momentary spikes in 5xx errors/RT during container termination and cluster recovery.

Jira Ticket: https://folio-org.atlassian.net/browse/PERF-1387

Summary

1. High Availability (HA) & Failover: During the dynamic scaling and failover tests, the jdbc-ping cluster met the SLA. When forcefully terminating a Keycloak node, the system recovered successfully, resulting in an error rate of 0% before traffic.

2. In test 2,3,4,5 Folio-keycloak works in cluster mode as expected.

3. Error rate were 0% in all FailOver tests.

4. We conducted three test runs for each configuration and analyzed the results. The requirement of 0 regression was met. The implementation of the Keycloak-111 branch introduces no systemic degradation to response times. Furthermore, high-volume core transactions demonstrated noticeable improvements in both average response times and 90th percentile (pct90) metrics compared to the baseline.

Service Level Agreement (SLA) Compliance & A/B Comparison

Goal: 0 Regression Response times (90th percentile) on the custom branch (jdbc-ping) must not degrade compared to the JDBC_PING2 baseline.

1. Core Authentication & Role Management Lifecycle

2. Failover Reliability

Goal: Minimal disruption during dynamic scaling and failover. Expectation is 0 dropped connections, or only momentary, acceptable spikes in 5xx errors while the cluster recovers.

Test Runs

Load Model

Workflow 1. Authorization

• Total Concurrency: 10 threads (Virtual Users).

• Multi-Tenant Distribution: The load is distributed evenly across 30 distinct tenants to simulate a realistic, multi-tenant environment.

• User Account: A predefined dataset of 300 unique user accounts is utilized during the test execution.

Workflow 2. Role & Access Management

• Total Concurrency: 1 thread (Virtual Users).

• Multi-Tenant Distribution: The load is distributed evenly across 60 distinct tenants to simulate a realistic, multi-tenant environment.

The transaction flow includes the following steps:

1. Create Role: Generating a new user role within the system.

2. Select Application: Linking the role to a specific application context.

3. Assign Capability-sets: Attaching specific permissions and capability-sets to the newly created role.

4. Assign Role to User: Mapping the configured role to a random user from the account pool.

5. Delete Role: Removing the role to clean up the environment and complete the lifecycle loop.

Test results