Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests. PERF-879 - Getting issue details... STATUS

Summary IN-PROGRESS

Check-In/Check-Out tests on Quesnelia release (Eureka) in QECP1 environment carried out with 8, 20, 30, 75 virtual users during 45 minutes and with 30 virtual users during longevity test (24 hours) to detect any trends in resource utilization

45 minute tests

Response times in tests with 8, 20, 30 virtual users delta was 200 ms. The lowest average response times were with 20 virtual users test: Check-In - 625 ms, average Check-Out - 1356 ms
The highest average response times showed 75 vUsers test: Check-In - 4 sec 610 ms, Check-Out - 12 sec 589 ms (whole test), Check-In - 622 ms, Check-Out - 1540 ms (before saturation point with 33 vUsers)

Longevity test

Longevity test was running 10 hours and began failing because of token expiration. Possible reason here is SSO Session Max limitation 10 hours (fs09000000/realm-settings/sessions).
- Average response times in 10 hours test for Check-In - 1003 ms, Check-Out - 2870 ms
- First 5 hours 30 minutes average response times Check-In - 770 ms, Check-Out - 1773 ms, the next 4 hours 30 minutes - Check-In - 1295 ms, Check-Out - 4280 ms

Comparison with previous Quesnelia release (qcp1 environment) results:

8, 20, 30 vUsers tests were stable. And 75 vUsers test revealed saturation point during ramp-up period with 33 vUsers. After this point response times increased significantly with the same throughput.
There's a significant performance degradation compared to previous results from tests in quesnelia release on qcp1 environment (non-eureka):
- 8 vUsers - Check-out - 96%, Check-in - 48%
- 20 vUsers - Check-out - 61%, Check-in - 33%
- 30 vUsers - Check-out - 89%, Check-in - 55%
- 75 vUsers - Check-out - 1111%, Check-in - 695% (during whole test)
- 75 vUsers - Check-out - %, Check-in - % (before saturation point was found)

Resources

CPU utilization

Memory consumption

RDS CPU utilization average

CPU (User) usage by broker

Common notes

Recommendations

Test Runs

The following table contains tests configuration information

Test #	vUsers	Ramp-up, sec	Duration, sec
1	8	80	2700
2	20	200	2700
3	30	300	2700
4	75	750	2700
5	30	300	86400

Results

Errors:

Response time

The table contains results of Check-in, Check-out tests in Quesnelia release (Eureka). All requests that are >= 100 ms (in average response times)

45 minute tests

	8 vUsers			20 vUsers			30 vUsers			75 vUsers
Requests	Response Times (ms)			Response Times (ms)			Response Times (ms)			Response Times (ms)
Label	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct
Check-In Controller	1745	1459.64	1694	4140	1356.82	1569.95	6047	1646.96	2725.6	6567	12589.1	19031
Check-In Controller	1288	603.95	706	3297	625.78	665	4560	748.3	1340.05	4957	4610.3	8229.2
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	1746	661.9	755.95	4142	613.14	659	6047	729.59	1242.8	6594	3868.57	7169.25
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	1289	240.43	287.5	3305	277.17	293	4572	286.61	363	4969	1642.04	3832.5
GET_circulation/loans (Submit_barcode_checkout)	1745	285.6	340	4141	275.47	307	6047	347.89	640.6	6571	2528.16	5109
GET_circulation/loans (Submit_patron_barcode)	1746	36.23	262							6599	1548.85	3773
GET_circulation/requests (Submit_barcode_checkin)										4960	1171.58	3366.5
GET_circulation/requests_status_Open (Submit_patron_barcode)										6596	1120.86	3117.2
GET_circulation/requests (Submit_barcode_checkout)										6565	997.17	3095.7
GET_users (Submit_patron_barcode)	1747	121.36	129				6050	136.53	368.45	6603	872.72	2508.6
GET_inventory/items (Submit_barcode_checkin)	1289	107.55	130.5	3305	111.12	136	4576	165.7	469.15	4970	848.62	2773
GET_users (Get_check_in_page)	1290	120.27	140				4577	165.61	506.3	4977	836.34	2577.1
GET_inventory/items (Submit_barcode_checkout)	1746	107.88	130	4143	110.18	136	6048	160.44	468.55	6594	757.24	2602
GET_circulation/requests_status_openAwaitingPickup (Submit_patron_barcode)										6594	718.98	2636

Longevity test

30 vUsers Longevity
Requests	Samples, Response Times
Label	#Samples	Average	95th pct
Check-Out Controller	77197	2870	8510
Check-In Controller	58806	1003	3320
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	77205	1100	4050
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	58817	290	390

Comparisons

This table has comparison between average values of response times of Quesnelia (Check-in-check-out Test Report (Quesnelia)) and Quesnelia (Eureka) releases

	8 Users				20 Users				30 Users				75 Users				75 Users, upto 33 vUsers rampup
Requests	Response Times				Response Times				Response Times				Response Times				Response Times
	Quesnelia	Eureka			Quesnelia	Eureka			Quesnelia	Eureka			Quesnelia	Eureka			Quesnelia	Eureka
Label	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%
Check-Out Controller	741	1459	718	96.90%	842.65	1356	513.35	60.92%	870.2	1646	775.8	89.15%	1039	12589	11550	1111.65%	1039	1541	502	48.32%
Check-In Controller	408	603	195	47.79%	467.71	625	157.29	33.63%	480.62	748	267.38	55.63%	580	4610	4030	694.83%	580	623	43	7.41%

Comparison of Longevity test

30 vUsers Longevity
Response Times
Quesnelia	Eureka
Average		Delta,ms	Difference,%
1065	2870	1805	169.48%
454	1003	549	120.93%

Resources Utilization

CPU Utilization

45 minute and longevity tests used CPU in correspondence with number of vUsers but there's module that used about 16% - mod-inventory-storage-b. We do not observe expected growth for module CPU utilization in 75 vUsers.

45 minute tests

CPU by modules

CPU parameter in revisions set to 0 (containers use all available resources)

Service Name	CPU 8 vUsers	CPU 20 vUsers	CPU 30 vUsers	CPU 75 vUsers
mod-inventory-storage-b				16.8
kong-b				5.27
mod-inventory-b				4.7
mod-circulation-b				3.57
mod-circulation-storage-b				3.36
mod-users-keycloak-b				2.38
mod-permissions-b				2.2
mod-pubsub-b				2.08
mod-users-b				2.01
mod-feesfines-b				1.64
mod-roles-keycloak-b				1.4
mod-configuration-b				1.12
edge-patron-b				1.11
mod-remote-storage-b				1.07
mod-patron-blocks-b				0.83
mod-audit-b				0.6
mgr-tenant-entitlements-b				0.6
mgr-tenants-b				0.58
mgr-applications-b				0.56
mod-source-record-manager-b				0.54

Longevity test

Memory Consumption

45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%

Memory usage by modules

Service Name	Memory 75 vUsers	Service Name	Memory 30 vUsers Longevity
mod-inventory-b	81.19	mod-search-b	90.1
mod-search-b	68.23	mod-inventory-b	70.6
mod-agreements-b	65.43	mod-scheduler-b	69.72
mod-users-keycloak-b	64.15	mod-agreements-b	67.9
mgr-applications-b	63.73	mod-users-keycloak-b	64.95
mod-scheduler-b	63.36	folio-keycloak-b	64.87
mgr-tenant-entitlements-b	61.72	mgr-applications-b	64.62
mod-roles-keycloak-b	61.5	mgr-tenant-entitlements-b	63.84
folio-keycloak-b	57.62	mod-roles-keycloak-b	62.91
mod-notes-b	56.01	mod-notes-b	59.3
mod-lists-b	55.4	mod-lists-b	57.03
mod-data-export-worker-b	54.31	mod-data-export-worker-b	56.41
mod-users-b	53.11	mod-patron-blocks-b	55.2
mgr-tenants-b	53.01	mgr-tenants-b	54.02
mod-feesfines-b	52.16	mod-users-b	53.98
mod-login-keycloak-b	51.31	mod-feesfines-b	53.8
mod-service-interaction-b	50.98	mod-invoice-b	53.13
mod-orders-b	50.98	mod-serials-management-b	53
mod-invoice-b	50.74	mod-login-keycloak-b	52.52
mod-audit-b	50.59	mod-service-interaction-b	52.26

45 minute tests

Longevity test

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 14%, 20 vUsers - 26%, 30 vUsers - 35%, 75 vUsers - 36% During longevity test CPU stayed on the same level - 38%

45 minute tests

Longevity test

RDS Database Connections

For both 45 minute and longevity tests RDS used max 867 connections. Without test it was 812 connections.

45 minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 18, 15, 15, 13 % accordingly to broker distribution in 45 min tests

Common CPU utilization by broker during longevity was 18, 18, 16, 14% accordingly to broker distribution. Resource utilization decreased significantly after 6 hours 20 minutes (17:40 UTC) to 14, 12, 12, 11%

45 minute tests

Longevity test

Database load

45 minute tests

Longevity

Appendix

Infrastructure

PTF -environment qecp1
11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 1 instance of db.r6.xlarge database instance: Writer instance MSK tenant [ kafka configurations] 4 kafka.m5.2xlarge brokers in 2 zones Apache Kafka version 2.8.0 EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=3

DB table records size:

fs09000000
- instances - 24493326
- items - 25662862
- holding records - 24459683

Modules

Check-In / Check-Out related modules

All modules

Methodology/Approach

Run data preparation script before each CI/CO test
Update .jmx file script for Quesnelia release
Create artefact and upload to carrier-io
Use Jenkins job to change parameters and run tests
Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
Test CI/CO with 30 users for 24 hours to detect any trends in memory.

[Eureka] Check-in-check-out Test Report (Quesnelia)

Overview

Summary IN-PROGRESS

Recommendations

Test Runs

Results

Errors:

Comparisons

Resources Utilization

CPU Utilization

Memory Consumption

RDS CPU Utilization

RDS Database Connections

CPU (User) usage by broker

Database load

Appendix

Infrastructure

Methodology/Approach