Check-in-check-out Test Report Trillium [non-ECS]
Overview
Regression testing of Circulation Check-in Check-out (CICO) workflow in Trillium release in scope of https://folio-org.atlassian.net/browse/PERF-1371 ;
The purpose of CICO testing:
Define system behavior (ECS resource usage, DB resource usage, queries) with different load;
Measure response times of Trillium release and compare them with Sunflower to define trends;
Find anomalies and performance issues if any. If possible recommend improvements
Previous report: Check-in-check-out Test Report Sunflower [non-ECS]
Summary
All five Check-In/Check-Out tests (8, 20, 30, 75 vUsers at 45 min + 30 vUsers longevity at 12 hours) completed successfully with only a few failed requests caused by data issues, not system problems.
Tests were performed sequentially for 8, 20, 30, and 75 virtual users. No signs of memory leaks were detected across related modules; memory usage showed stable trends with no spikes or growth.
Trillium outperforms Sunflower across all load levels. Performance improvements scale with load:
Check-Out Controller improved by 21.7% (8 vUsers), 26.6% (20 vUsers), 27.2% (30 vUsers), 43.4% (75 vUsers, from 1629ms to 922ms).
Check-In Controller improved by 13.6% (8 vUsers), 17.8% (20 vUsers), 16.4% (30 vUsers), 38.4% (75 vUsers, from 782ms to 482ms).
Longevity test (30 vUsers, 12 hours) also shows improvement: Check-Out avg decreased by 15.2% (1080ms → 916ms) and Check-In by 3.4% (414ms → 400ms).
CPU utilization in the highest-load test (#4, 75 vUsers) was dominated by mod-circulation-b (345%), kong-b (223%), and mod-circulation-storage-b (210%).
Memory remained stable throughout all tests. The only anomaly was mod-data-export-worker-b, which showed an unexpected drop from 52% to 39% during the 75 vUsers test.
Database: One long-running query was identified in mod-search (call_number CTE join) averaging 5640.92ms per call, though its throughput was negligible at 0.04 calls/sec
Test Runs
Test # | vUsers | Ramp-up, sec | Duration, sec |
|---|---|---|---|
1 | 8 | 80 | 2700 |
2 | 20 | 200 | 2700 |
3 | 30 | 300 | 2700 |
4 | 75 | 750 | 2700 |
5 | 30 | 300 | 43200 (12 hours) |
Results
Test results
Label | Response times (ms) | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | 30 vUsers (test #5) | |||||||||||
#Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | |
Check-Out Controller | 2651 | 1345 | 779 | 6658 | 1309 | 734 | 9943 | 1348 | 757 | 24143 | 1734 | 922 | 157033 | 1536 | 916 |
Check-In Controller | 2026 | 685 | 389 | 4964 | 689 | 375 | 7457 | 744 | 388 | 18168 | 908 | 482 | 118928 | 656 | 400 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 2651 | 374 | 240 | 6659 | 355 | 223 | 9943 | 402 | 229 | 24153 | 518 | 287 | 157038 | 406 | 263 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 2027 | 268 | 173 | 4964 | 249 | 172 | 7458 | 365 | 175 | 18170 | 413 | 228 | 118928 | 301 | 183 |
GET_circulation/loans (Submit_barcode_checkout) | 2651 | 219 | 127 | 6658 | 192 | 118 | 9943 | 214 | 122 | 24146 | 304 | 155 | 157033 | 188 | 122 |
Comparisons vs Sunflower
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | T | S |
|
| T | S |
|
| T | S |
|
| T | S |
|
|
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta,% | ||||
Check-Out Controller | 779 | 995 | -216 | -21.70% | 734 | 1000 | -266 | -26.60% | 757 | 1040 | -283 | -27.20% | 922 | 1629 | -707 | -43.40% |
Check-In Controller | 389 | 450 | -61 | -13.60% | 375 | 456 | -81 | -17.80% | 388 | 464 | -76 | -16.40% | 482 | 782 | -300 | -38.40% |
Comparison of longevity test
30 vUsers Longevity (test #5) | |||
|---|---|---|---|
Response Times, milliseconds avg. | Eureka comparison | ||
Sunflower | Trillium | Delta,ms | Delta,% |
1080 | 916 | -164 | -15.20% |
414 | 400 | -14 | -3.40% |
Service CPU utilization
most used modules in test #4:
# | Service | Value |
|---|---|---|
1 | mod-circulation-b | 345 |
2 | kong-b | 223 |
3 | mod-circulation-storage-b | 210 |
4 | mod-users-b | 133 |
5 | mod-remote-storage-b | 106 |
6 | mod-inventory-storage-b | 88 |
7 | mod-feesfines-b | 66 |
8 | mod-search-b | 59 |
9 | mod-audit-b | 50 |
10 | mod-patron-blocks-b | 43 |
11 | mod-inventory-b | 37 |
12 | mod-source-record-storage-b | 31 |
13 | mod-inn-reach-b | 30 |
14 | mod-settings-b | 28 |
15 | mod-users-keycloak-b | 28 |
16 | mod-orders-storage-b | 25 |
17 | mod-configuration-b | 24 |
18 | mod-roles-keycloak-b | 24 |
Service memory usage
Memory usage showed stable trends with no spikes and growths. The only service that behaved weird was mod-data-export-worker-b with dropdown of memory consumption from 52% to 39% while test #4 75vUsers (highest load on CI/CO flows)
Database metrics
Long running queries
WITH cte AS ( SELECT id, call_number, call_number_pr... 5640.92 ms/per call but throughput was just 0.04 per sec.
WITH cte AS (
SELECT
id,
call_number,
call_number_prefix,
call_number_suffix,
call_number_type_id,
last_updated_date
FROM fs09000000_mod_search.call_number
WHERE (last_updated_date, id) > ($1, $2)
ORDER BY last_updated_date, id
LIMIT $3
)
SELECT
c.id,
c.call_number,
c.call_number_prefix,
c.call_number_suffix,
c.call_number_type_id,
c.last_updated_date,
json_agg(
CASE
WHEN sub.instance_ids IS NULL THEN NULL
ELSE json_build_object(
'instanceId', sub.instance_ids,
'tenantId', sub.tenant_id,
'shared', sub.shared,
'locationId', sub.location_id
)
END
) AS instances
FROM cte c
LEFT JOIN (
SELECT
cte.id,
ins.tenant_id,
i.shared,
ins.location_id,
array_agg(DISTINCT i.id) AS instance_ids
FROM fs09000000_mod_search.instance_call_number ins
INNER JOIN cte ON ins.call_number_id = cte.id
INNER JOIN fs09000000_mod_search.instance i ON i.id = ins.instance_id
GROUP BY
cte.id,
ins.tenant_id,
i.shared,
ins.location_id
) sub ON c.id = sub.id
GROUP BY
c.id,
c.call_number,
c.call_number_prefix,
c.call_number_suffix,
c.call_number_type_id,
c.last_updated_date
ORDER BY last_updated_date ASC, id ASCAppendix
Infrastructure
PTF -environment tecp1 |
|---|
|