BF-970 Sidecar Performance Testing

Purpose

Evaluate sidecar performance under different types of loads:

Ingress requests with user JWT token
Ingress requests with system JWT token
Ingress requests with user and system JWT tokens
Unauthorized egress requests

Approach

Testing Framework

Automatic token issuing with specified delay for each pair of username + password in ingress-requests.file

Ingress and egress requests file is formed based on available user capabilities and has the following format

{"username":"nENjhUV6_vDJptu3f30DB","password":"yAW0AUyvFOZuUGOdICel","method":"POST","url":"/inventory-hierarchy/items-and-holdings"}
{"username":"giLofMi8_QmkHFrjEFKXd","password":"iYJWfBVJ5bit3v6MZCd1","method":"DELETE","url":"/identifier-types/{id}"}
{"username":"N1lgC0Th_XL6DhYuVJqJA","password":"W9vpGUOwddz2oE3nwtIk","method":"GET","url":"/item-storage/items"}

Each test can be configured using a profile in .conf

ingress-10u-300s-50s {
  baseUrl = "http://localhost:19021"
  tracingEnabled = true
  rampUpUsers = 10
  rampUpDuration = 1s
  testDuration = 300s
  tokenRefreshInterval = 60s
  config {
    type = "ingress"
    includeUserJwt = true
    includeSystemJwt = true
    requestsFile = "igress-requests.jsonl"
  }
}

Before each test to warm-up sidecar, a load is given for 1 minute for 10 simultaneous users

folio-module-sidecar

A sidecar image with maximum isolation from other system services, all outside requests are mocked using Wiremock, including:

mgr-applications
mgr-tenant-entitlements
mgr-tenants
AWS SSM service calls

Ingress requests are simulated using bootstrap information for mod-inventory-storage-28.1.0-SNAPSHOT.1064
Egress requests are simulated using bootstrap information for mod-bulk-operations-2.2.0-SNAPSHOT.348

MODULE_URL points to the mock-server
SIDECAR_FORWARD_UNKNOWN_REQUESTS_DESTINATION points to the mock-server
all locations for egress modules point to the mock-server

keycloak

It contains a tenant and 100 auth users with 10 random permissions per user based on mod-inventory-storage capabilities.

mock-server

A native image based on Quarkus can respond to any request quickly with a specified delay.

mock-server should not be a bottleneck for sidecar performance testing

Performance test configuration

Property	Value

Property	Value
Environment	Local (Windows 11, Docker Desktop) Client: Version: 27.4.0 Context: desktop-linux Debug Mode: false Server Version: 27.4.0 Storage Driver: overlayfs driver-type: io.containerd.snapshotter.v1 Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog CDI spec directories: /etc/cdi /var/run/cdi Swarm: inactive Runtimes: io.containerd.runc.v2 nvidia runc Default Runtime: runc Init Binary: docker-init containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111 runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: seccomp Profile: unconfined Kernel Version: 5.15.153.1-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.93GiB Name: docker-desktop ID: 626567ea-afe5-4475-889f-b23fe4dcdf36 Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=npipe://\\.\pipe\docker_cli Experimental: false Insecure Registries: hubproxy.docker.internal:5555 127.0.0.0/8 Live Restore Enabled: false
sidecar	folio-module-sidecar https://hub.docker.com/layers/folioorg/folio-module-sidecar/2.0.5/images/sha256-addbf9e7e8a5f4475c19f89a51193281578dd5e76cb6f59ee2c54340e8941a93 `CPU Reservation: 128 / 256 / 512 / 1024 (0.125 / 0.25 / 0.5 / 1.0 vCPU) CPU Limit: 128 / 256 / 512 / 1024 (0.125 / 0.25 / 0.5 / 1.0 vCPU) Memory Reservation: 192 MB Memory Limit: 224 MB`
folio-module	A native image based on Quarkus that accepts any request and returns it with all requested data back, including the request body `CPU limit: 1024 (1.0 vCPU) Memory Limit: 400 MB Response Delay: 50 / 200 / 1000 ms Request Body: jsonObject with key: 10 random chars, value: 100 random chars), number of keys = 50 / 500 / 5000`
keycloak	`CPU Limit: 3072 (3.0 vCPU) CPU Reservation: 2048 (2.0 vCPU) Memory Limit: 1200 MB Memory Reservation: 1200 MB`
Performance Test duration	600 sec
V_USERS	500
RAMP_UP	550 sec

Performance test results

Test results as archive:

50 ms delay / 0.1 vCPU / 1 Event Loop / 5 Thread workers

Ingress requests

Requests	Executions					Response Time (ms)
Requests	Total	OK	KO	% KO	Cnt/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Mean	Std Dev
All Requests	27713	27713	0	0	46.19	19	5360	8230	11596	12309	13851	5779	3293
Authenticate user	1141	1141	0	0	1.9	19	25	39	70	123	164	34	20
GET request	6096	6096	0	0	10.16	63	5611	8511	11605	12241	13354	6072	3149
POST request	4821	4821	0	0	8.04	56	5577	8385	11580	12204	13305	6013	3139
DELETE request	10703	10703	0	0	17.84	55	5534	8436	11508	12165	13851	6004	3132
PUT request	4952	4952	0	0	8.25	55	5616	8335	11607	12156	13305	6030	3123

Ingress requests with module token

Requests	Executions					Response Time (ms)
Requests	Total	OK	KO	% KO	Cnt/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Mean	Std Dev
All Requests	26975	26975	0	0	45.11	2	3729	7289	20655	26419	34906	5791	6254
Authenticate user	1188	1188	0	0	1.99	19	38	62	149	221	237	53	44
Authenticate module client	44	44	0	0	0.07	2	7	9	10	11	11	6	3
GET request	5896	5896	0	0	9.86	55	3987	7458	20701	25350	34906	6013	6159
POST request	4662	4662	0	0	7.8	54	4106	7681	21165	27303	33899	6270	6466
DELETE request	10372	10372	0	0	17.34	54	4004	7485	20879	26516	33001	6050	6283
PUT request	4813	4813	0	0	8.05	55	3976	7359	20652	26481	32881	5965	6189

Egress requests

Requests	Executions					Response Time (ms)
Requests	Total	OK	KO	% KO	Cnt/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Mean	Std Dev
All Requests	87511	87511	0	0	145.85	54	1661	2469	3772	4575	6697	1854	1006
GET request	35719	35719	0	0	59.53	54	1624	2444	3709	4487	6637	1830	992
POST request	22916	22916	0	0	38.19	54