Check-in-check-out Test Report (Ramsons) [non-ECS] [Eureka]
- Leonid Kolesnykov
Test status: PASSED
Overview
- Regression testing of Check-In/Check-Out (CI/CO) fixed load tests on eureka based environment in Ramsons non-ECS release.
- Test was carried out on central tenant.
- The purposes of CI/CO testing:
- To test how different load from CI/CO flow affect the system
- To define response times of Check-In and Check-Out
- To find any trends for resource utilization and recommend improvements
- To check how system behaves over extended period during longevity test (SSO Session Max limit in keycloak is 10 hours)
- Compare results (current and previous)
- The current ticket:
PERF-1065 - [Ramsons] [non-ECS] [Eureka] CI/CO
Summary
- Common results:
- The Ramsons release shows good performance up to 30 virtual users (vUsers) without notable performance degradation. The significant reduction in RTs across all tests observed compared to Quesnelia. The slight RT increase after 5 hours in the CO flow in longevity test should be investigated.
- Tests #1, #2, #3, #4
- Average response times (RT) in tests with 8, 20, 30 vUsers were the same in average (delta - 80 ms.). Average RT in 20 vUsers test CI - 480 ms, CO - 862 ms.
- Average RT in test with 75 vUsers response times grew +40% compared to 20 vUsers. Average RT in 75 vUsers test CI - 933 ms, CO - 1472 ms.
- Test #5
- Average RT in test #5 with 30 vUsers. CI - 442 ms, CO - 971 ms.. There's no expected degradation during 10 hours test in Average. But we can observe RT growing after 5 hours of test + 200 ms in CO flow.
- No visible memory leaks found. Accumulating affect in longevity test sh
- Comparison with Quesnelia results:
- CI RT decreased 17% for 8 vUsers, 25% - 20 vUsers, 33% - 30 vUsers, 79% - 75 vUsers.
- CO RT decreased 36% for 8 vUsers, 36% - 20 vUsers, 42% - 30 vUsers, 88% - 75 vUsers
- Longevity test: CI RT decreased 66%, CO RT decreased 78%
Resources
- CPU utilization
- CPU utilization depends on number of concurrent virtual users and increasing accordingly. Top used modules (75 vUsers): mod-remote-storage - 102%, mod-feesfines - 55%, mod-roles-keycloak - 51%, mod-inventory-storage - 48%, mod-users - 45%
- Memory consumption
Top module memory usage: folio-keycloak - 83%, mod-agreements - 73% in 75 vUsers test. Folio-keycloak used 90% in longevity.
- RDS CPU utilization average
- RDS CPU utilized in average: 8 vUsers - 15%, 20 vUsers - 23%, 30 vUsers - 30%, 75 vUsers - 66% During longevity test CPU was growing from 32% to 42%.
- CPU (User) usage by broker
- 8 vUsers - 15%, 20 vUsers - 16%, 30 vUsers - 17%, 75 vUsers - 22%. During Longevity test - 17%
Recommendations & Jiras
- To avoid Internal server errors during testing on eureka environment new values were applied for mod-users-keycloak and mod-users-keycloak - Sidecar 1.
| Module | Mem Hard Limit | Mem Soft Limit | CPU Units | Xmx |
|---|---|---|---|---|
| mod-users-keycloak | 1536 | 1480 | 256 | 1024 |
| mod-users-keycloak - Sidecar 1 | 1280 | 768 | 256 | 512 |
Test Runs
The following table contains tests configuration information
Test # | vUsers | Ramp-up, sec | Duration, sec |
1 | 8 | 80 | 2700 |
| 2 | 20 | 200 | 2700 |
| 3 | 30 | 300 | 2700 |
| 4 | 75 | 750 | 2700 |
| 5 | 30 | 300 | 36000 |
Results
Response time
The table contains results of Check-in, Check-out tests in Ramsons release (with modified mod-users-keycloak revision).
Test #1, #2, #3, #4
This table has comparison between average values of response times of Ramsons and Quesnelia releases (default mod-users-keycloak values)
| 8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Requests | Response Times (ms) | Response Times (ms) | Response Times (ms) | Response Times (ms) | ||||||||
| Label | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average |
| Check-Out Controller | 1853 | 1104 | 924 | 4554 | 952 | 862 | 6447 | 986 | 946 | 13891 | 1449 | 1472 |
| Check-In Controller | 1334 | 578 | 496 | 3321 | 480 | 467 | 4942 | 518 | 495 | 10449 | 1029 | 933 |
| POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 1854 | 477 | 402 | 4555 | 381 | 375 | 6449 | 396 | 398 | 13898 | 632 | 657 |
| POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 1336 | 293 | 260 | 3329 | 248 | 250 | 4951 | 265 | 269 | 10472 | 425 | 449 |
| GET_circulation/loans (Submit_barcode_checkout) | 1854 | 223 | 173 | 4554 | 183 | 162 | 6447 | 188 | 189 | 13893 | 279 | 304 |
| GET_users (Get_check_in_page) | 10477 | 385 | 225 | |||||||||
| GET_inventory/items (Submit_barcode_checkout) | 13898 | 100 | 113 | |||||||||
This table has comparison between average values of response times of Ramsons and Quesnelia releases (modified mod-users-keycloak values)
| 8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Requests | Response Times (ms) | Response Times (ms) | Response Times (ms) | Response Times (ms) | ||||||||
| Label | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average |
| Check-Out Controller | 1772 | 1175 | 965 | 4509 | 987 | 860 | 6593 | 1006 | 873 | 14632 | 1177 | 1000 |
| Check-In Controller | 1380 | 765 | 608 | 3293 | 609 | 550 | 4908 | 613 | 524 | 11177 | 732 | 576 |
| POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 1773 | 523 | 433 | 4509 | 408 | 365 | 6597 | 419 | 370 | 14635 | 532 | 432 |
| POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 1383 | 463 | 364 | 3297 | 366 | 326 | 4917 | 368 | 297 | 11198 | 424 | 309 |
| GET_circulation/loans (Submit_barcode_checkout) | 1772 | 239 | 180 | 4509 | 201 | 169 | 6594 | 209 | 173 | 14635 | 246 | 195 |
Test #5
| 30 vUsers Longevity test (test #5) | |||
|---|---|---|---|
| Requests | Samples, Response Times (ms) | ||
| Label | #Samples | 95th pct | Average |
| Check-Out Controller | 90038 | 1224 | 971 |
| Check-In Controller | 68133 | 533 | 442 |
| POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 90040 | 440 | 383 |
| POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 68138 | 306 | 221 |
| GET_circulation/loans (Submit_barcode_checkout) | 90040 | 191 | 164 |
Comparisons
This table has comparison between average values of response times of Ramsons and Quesnelia releases (default mod-users-keycloak values)
| 8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Requests | Response Times, milliseconds | |||||||||||||||
| Quesnelia | Ramsons | Quesnelia | Ramsons | Quesnelia | Ramsons | Quesnelia | Ramsons | |||||||||
| Label | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | ||||
| Check-Out Controller | 1460 | 924 | -536 | -36.70% | 1357 | 862 | -494.82 | -36.47% | 1647 | 946 | -701 | -42.56% | 12589 | 1472 | -11116.8 | -88.31% |
| Check-In Controller | 604 | 496 | -108 | -17.87% | 626 | 467 | -158.78 | -25.37% | 748 | 495 | -253 | -33.85% | 4610 | 933 | -3677.03 | -79.76% |
This table has comparison between average values of response times of Ramsons and Quesnelia releases (modified mod-users-keycloak values)
| 8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Requests | Response Times, milliseconds | |||||||||||||||
| Quesnelia | Ramsons | Quesnelia | Ramsons | Quesnelia | Ramsons | Quesnelia | Ramsons | |||||||||
| Label | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | Average | Delta,ms | Delta,% | ||||
| Check-Out Controller | 1460 | 965.1 | -495 | -33.88% | 1357 | 859.64 | -497.18 | -36.64% | 1647 | 872.7 | -774 | -47.01% | 12589 | 1000 | -11588.6 | -92.05% |
| Check-In Controller | 604 | 608.1 | 4.19 | 0.69% | 626 | 550.09 | -75.69 | -12.10% | 748 | 524.4 | -224 | -29.92% | 4610 | 576 | -4033.99 | -87.51% |
Comparison of longevity test
| 30 vUsers Longevity (test #5) | |||
|---|---|---|---|
| Response Times, milliseconds | |||
| Quesnelia | Ramsons | ||
| Average | Average | Delta,ms | Delta,% |
| 4541 | 971 | -3570 | -78.62% |
| 1311 | 442 | -869 | -66.29% |
API requests where response times >= 100 milliseconds
API | 75 vUsers Ramsons Average, ms |
|---|---|
| POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 657 |
| POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 449 |
| GET_circulation/loans (Submit_barcode_checkout) | 304 |
| GET_users (Get_check_in_page) | 225 |
| GET_inventory/items (Submit_barcode_checkout) | 113 |
Cluster Resources Utilization
CPU Utilization
CPU utilization depends on number of concurrent virtual users and increasing accordingly. Top used modules (75 vUsers): mod-remote-storage - 102%, mod-feesfines - 55%, mod-roles-keycloak - 51%, mod-inventory-storage - 48%, mod-users - 45%.
For exact numbers use the chart below.
Tests #1, #2, #3, #4
Test #5
Memory Consumption
Modules which consumed most of memory: folio-keycloak - 83%, mod-agreements - 73% in 75 vUsers test. Folio-keycloak used 90% in longevity.
Tests #1, #2, #3, #4
Test #5
RDS CPU Utilization
RDS CPU utilized in average: 8 vUsers - 15%, 20 vUsers - 23%, 30 vUsers - 30%, 75 vUsers - 66% During longevity test CPU was growing from 32% to 42%.
Tests #1, #2, #3, #4
Test #5
RDS Database Connections
For 45 minute and longevity tests RDS used max 1100 connections.
Tests #1, #2, #3, #4
Test #5
Database load
INSERT INTO fs09000000_mod_pubsub.audit_message (id, event_id, event_type, tenant_id, audit_date, state, published_by, correlation_id, created_by, error_message) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
WITH prefixes AS (
SELECT prefix || '%' AS pattern
FROM UNNEST(CAST($1 AS text[])) AS prefix
),
permission_names AS (
SELECT permission_name
FROM UNNEST(CAST($2 AS text[])) AS permission_name
),
user_capabilities AS (
SELECT uc.user_id, uc.capability_id FROM user_capability uc
UNION ALL
SELECT ucs.user_id, csc.capability_id
FROM user_capability_set ucs
INNER JOIN capability_set_capability csc
ON ucs.capability_set_id = csc.capability_set_id
UNION ALL
SELECT ur.user_id, rc.capability_id
FROM user_role ur
INNER JOIN (
SELECT rc.role_id, rc.capability_id FROM role_capability rc
UNION ALL
SELECT rcs.role_id, csc.capability_id
FROM role_capability_set rcs
INNER JOIN capability_set_capability csc
ON rcs.capability_set_id = csc.capability_set_id
) rc ON rc.role_id = ur.role_id
),
user_permissions AS (
SELECT DISTINCT c.folio_permission
FROM user_capabilities uc
INNER JOIN capability c ON uc.capability_id = c.id
WHERE uc.user_id = $3
AND (
c.folio_permission IN (SELECT permission_name FROM permission_names)
OR EXISTS (
SELECT 1 FROM prefixes p
WHERE c.folio_permission LIKE p.pattern
)
)
),
replaced_permissions AS (
SELECT UNNEST(p.replaces) AS folio_permission
FROM user_permissions up
INNER JOIN permission p
ON p.name = up.folio_permission
)
SELECT DISTINCT folio_permission
FROM (
SELECT folio_permission FROM user_permissions
UNION ALL
SELECT folio_permission FROM replaced_permissions
) t
UPDATE fs09000000_mod_inventory_storage.item SET jsonb=$1 WHERE id=$2 RETURNING jsonb::text
SELECT fs09000000_mod_inventory_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.service_point WHERE ((true) AND ( (get_tsvector(f_unaccent(service_point.jsonb->>''ecsRequestRouting'')) @@ tsquery_phrase(f_unaccent(''true''))) IS NOT TRUE)) AND (id=''3a7de149-f17d-4602-adfe-0b09bf8e884a'')')
SELECT upsert('circulation_logs', $1::uuid, $2::jsonb)
SELECT COUNT(*) FROM fs09000000_mod_users.users
SELECT fs09000000_mod_circulation_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_circulation_storage.loan_policy WHERE id=''2be97fb5-eb89-46b3-a8b4-776cea57a99e''')
SELECT fs09000000_mod_patron_blocks.count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>''userId'') = ''aa054aab-73c3-4764-941e-41fdff14900b''')
SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>'userId') = 'cd980484-e41b-4334-8ade-1f31a067422e'
Tests #1, #2, #3, #4
Test #5
MSK resources utilization
CPU (User) usage by broker
8 vUsers - 15%, 20 vUsers - 16%, 30 vUsers - 17%, 75 vUsers - 22%. During Longevity test - 17%
Tests #1, #2, #3, #4
Test #5
Errors:
Test | #Samples | #Errors | Error | #Errors | Error | #Errors | Error | #Errors | Error | #Errors | Error | #Errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| #5, Longevity | 842459 | 319 | 500/Internal Server Error | 306 | 422/Unprocessable Entity | 7 | 400/Bad Request | 3 | 401/Unauthorized | 2 | 504/Gateway Time-out | 1 |
Appendix
Infrastructure
| PTF -environment recp1 |
|---|
|
DB table records size:
|
|---|
Modules
Methodology/Approach
Description
Testing includes data preparation step and testing itself
- Data preparation for each test takes up to 20 minutes and consists of truncating involved in testing tables, populating data and updating statuses of items.
- Test itself depends on duration and virtual users number creating necessary load.
In Ramsons token expiration set to 10 minutes by default so to run any tests use new login implementation from the script. Pay attention to Backend Listener. Replace value of application parameter to make the results visible in Grafana dashboard.
Module configuration recommended setup
New approach implemented in SRS module to run marc_indexers (mi) WITH deleted_rows every 30 minutes
{
"name": "srs.marcIndexers.delete.interval.seconds",
"value": "1800"
},
{
"name": "srs.marcIndexers.delete.dirtyBatchSize",
"value": "1000"
}
Update mod-serials module. Set number of task with 0 to exclude significant database connection growth.
DB trigger setup in Ramsons
Usual PTF CI/CO data preparation script won’t work in Ramsons. To solve that disable trigger updatecompleteupdateddate_item_insert_update before data preparation for the tenant and enable it before test start.
The sql file was updated to do that step from the script.
Data preparation
- To prepare data establish connection by AWS key
- Run CICO_db_preparation.sh script located in /scripts folder. Before use the file tenats.csv to edit the list of tenants to restore the database.
- Files location: Buckets/fse-ptf/Scripts/CICO/Ramsons/
To start test from AWS instance (load generator) use template for the command. Test locally before start.
nohup jmeter -n -t /home/ptf/testdata/recp1/circulation_checkInCheckOut_recp1.jmx -l recp1_8vusers.jtl -e -o /home/ptf/testdata/recp1/results/recp1_8vusers -JGlobal_duration=2700 -JCICO_vusers=8 -JCICO_rampup=80
Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
Test CI/CO with 30 users for 24 hours to detect any trends in memory.
To create widgets in AWS dashboard to monitor and collect CI/CO related modules parameters (service CPU and Memory) use these json:
Additional Screenshots of graphs or charts
Response times
Folio-module-sidecar connected wiki page:
To define the response times for requests that take longer than 100 milliseconds