Overview

The test's target is to investigate the behavior of the Check-In/Check-Out activities during fixed load tests.

PERF-857 - Getting issue details... STATUS

Summary

Check-In/Check-Out tests on Quesnelia were carried out with 8, 20, 30, and 75 virtual users for 45 minutes and 30 virtual users for 24 hours to detect any trends in resource utilization (only 16 hours of results available).

45-minute tests

Response times in tests with 8, 20, and 30 virtual users didn't differ a lot but the lowest response times were with 20 virtual users. Average Check-In - 441 ms, average Check-Out - 893 ms
The highest response time results were with 75 virtual users. Average Check-In - 546 ms, Check-Out - 1176 ms
Reports: reports_test_results_8.zip, reports_test_results_20.zip, reports_test_results_30.zip, reports_test_results_75.zip

Longevity test

Average Check-In - 458 ms, Check-Out - 1258 ms (the results from the test 16 hours)
Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.
RDS CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.
Grafana results

Resources

CPU utilization
- 45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.

Memory consumption
- Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

Average RDS CPU utilization
- 8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

CPU (User) usage by broker
- Common CPU utilization by the broker during all tests was up to 12% with equal distribution between brokers

Common notes

Longevity test revealed Database CPU Utilization growing trend and CI and CO response time increasing.

Recommendations

Add more CPU to nginx-okapi to exclude module fluctuations

Test Runs

The following table contains test configuration information

Test #	Test artefact	vUsers	Ramp-up, sec	Duration, sec
1	circulation_checkInCheckOut_qcon	8	80	2700
2		20	200	2700
3		30	300	2700
4		75	750	2700
5		30	300	86400

Results

Errors:

Error messages: Response 401 due to changed login process and token expiration time 10 min (0.04%).

Response time

The table contains results of Check-in, Check-out tests in the Quesnelia release.

45-minute tests

	8 vUsers			20 vUsers			30 vUsers			75 vUsers
Requests	Response Times (ms)			Response Times (ms)			Response Times (ms)			Response Times (ms)
Label	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct
Check-Out Controller	1815	910.51	1102.40	4324	893.26	1054.00	6626	899.13	1058.00	10986	1176.40	1569.00
Check-In Controller	1391	476.67	598.80	3527	441.99	551.60	4911	435.40	544.00	14564	546.74	760.00
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	1798	275.86	340.00	4275	268.25	315.00	6556	268.31	316.00	14434	331.68	491.00
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	1388	243.06	313.55	3518	214.64	275.00	4906	205.53	267.00	11159	233.01	348.00
GET_circulation/loans (Submit_patron_barcode)	1805	143.41	173.00	4292	141.66	160.00	6586	140.52	161.00	14493	177.03	273.00
GET_circulation/loans (Submit_barcode_checkout)	1795	140.59	163.20	4275	139.57	158.00	6550	140.00	159.00	14418	174.65	265.00

Longevity test

30 vUsers Longevity
Requests	Samples, Response Times
Label	#Samples	95th pct	Average
Check-Out Controller	144129	1.998	1.258
Check-In Controller	108447	0.645	0.458
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	142488	0.613	0.383
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	107907	0.301	0.200

Resources Utilization

CPU Utilization

45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.

45 minute tests

Longevity test

Memory Consumption

45-minute tests didn't reveal some problems with memory usage by modules. Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

45-minute tests

Longevity test

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

45-minute tests

Longevity test

RDS Database Connections

For both 45-minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.

45-minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 12% with equal distribution between brokers

45 minute tests

Longevity test

Appendix

Infrastructure

PTF -environment qcon
11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 1 instance of db.r6.xlarge database instance: Writer instance MSK - tenant 4 kafka.m5.2xlarge brokers in 2 zones Apache Kafka version 2.8.0 EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=3

DB table records size:

fs09000000
- instances - 24493326
- items - 25662862
- holding records - 24459683

Modules

Check-In / Check-Out related modules

Module qcon-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
mod-agreements	1	mod-agreements:7.0.0	2	1592	1488	128	0	0	0	FALSE
mod-authtoken	1	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128	FALSE
mod-bulk-operations	1	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512	FALSE
mod-calendar	1	mod-calendar:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512	FALSE
mod-circulation-item	1	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0	FALSE
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512	FALSE
mod-configuration	1	mod-configuration:5.10.0	2	1024	896	128	768	88	128	FALSE
mod-dcb	1	mod-dcb:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-feesfines	1	mod-feesfines:19.1.0	2	1024	896	128	768	88	128	FALSE
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512	FALSE
mod-login	1	mod-login:7.11.0	2	1440	1298	1024	768	384	512	FALSE
mod-notes	1	mod-notes:5.2.0	2	1024	896	128	952	384	512	FALSE
mod-patron-blocks	1	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128	FALSE
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512	FALSE
mod-remote-storage	1	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512	FALSE
mod-search	1	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024	FALSE
mod-users	1	mod-users:19.3.1	2	1024	896	128	768	88	128	FALSE
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512	FALSE
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE

All modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
qcon-pvt
Fri Apr 05 15:19:05 UTC 2024
mod-remote-storage	1	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512	FALSE
mod-finance-storage	1	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128	FALSE
mod-ncip	1	mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-agreements	1	mod-agreements:7.0.0	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	1	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256	FALSE
mod-consortia	1	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024	FALSE
mod-organizations	1	mod-organizations:1.9.0	2	1024	896	128	700	88	128	FALSE
mod-serials-management	1	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512	FALSE
mod-settings	1	mod-settings:1.0.3	2	1024	896	200	768	88	128	FALSE
mod-search	1	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024	FALSE
edge-dematic	1	edge-dematic:2.2.0	1	1024	896	128	768	88	128	FALSE
mod-data-import	1	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512	FALSE
mod-tags	1	mod-tags:2.2.0	2	1024	896	128	768	88	128	FALSE
mod-authtoken	1	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128	FALSE
edge-courses	1	edge-courses:1.4.0	2	1024	896	128	768	88	128	FALSE
mod-inventory-update	1	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128	FALSE
mod-notify	1	mod-notify:3.2.0	2	1024	896	128	768	88	128	FALSE
mod-configuration	1	mod-configuration:5.10.0	2	1024	896	128	768	88	128	FALSE
mod-orders-storage	1	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128	FALSE
edge-caiasoft	1	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128	FALSE
mod-login-saml	1	mod-login-saml:2.8.0	2	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	1	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128	FALSE
mod-password-validator	1	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512	FALSE
mod-licenses	1	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512	FALSE
mod-gobi	1	mod-gobi:2.8.0	2	1024	896	128	700	88	128	FALSE
mod-bulk-operations	1	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512	FALSE
mod-fqm-manager	1	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512	TRUE
edge-dcb	1	edge-dcb:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-graphql	1	mod-graphql:1.12.1	2	1024	896	128	768	88	128	FALSE
mod-finance	1	mod-finance:4.9.0	2	1024	896	128	700	88	128	FALSE
mod-erm-usage	1	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128	FALSE
mod-batch-print	1	mod-batch-print:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-copycat	1	mod-copycat:1.6.0	2	1024	512	128	768	88	128	FALSE
mod-lists	1	mod-lists:2.0.0	2	3000	2600	128	2048	384	512	FALSE
mod-entities-links	1	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024	FALSE
mod-permissions	2	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512	FALSE
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-orders	1	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512	FALSE
edge-patron	1	edge-patron:5.1.0	2	1024	896	256	768	88	128	FALSE
edge-ncip	1	edge-ncip:1.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	1	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128	FALSE
mod-invoice	1	mod-invoice:5.8.0	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512	FALSE
mod-user-import	1	mod-user-import:3.8.0	2	1024	896	128	768	88	128	FALSE
mod-sender	1	mod-sender:1.12.0	2	1024	896	128	768	88	128	FALSE
edge-oai-pmh	1	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	1	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512	FALSE
mod-rtac	1	mod-rtac:3.6.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512	FALSE
mod-calendar	1	mod-calendar:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-event-config	1	mod-event-config:2.7.0	2	1024	896	128	768	88	128	FALSE
mod-courses	1	mod-courses:1.4.10	2	1024	896	128	768	88	128	FALSE
mod-circulation-item	1	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0	FALSE
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512	FALSE
mod-email	1	mod-email:1.17.0	2	1024	896	128	768	88	128	FALSE
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512	FALSE
mod-di-converter-storage	1	mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128	FALSE
edge-orders	1	edge-orders:3.0.0	2	1024	896	128	768	88	128	FALSE
edge-rtac	1	edge-rtac:2.7.1	2	1024	896	128	768	88	128	FALSE
mod-users	1	mod-users:19.3.1	2	1024	896	128	768	88	128	FALSE
mod-template-engine	1	mod-template-engine:1.20.0	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	1	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128	FALSE
mod-audit	1	mod-audit:2.9.0	2	1024	896	128	768	88	128	FALSE
edge-fqm	1	edge-fqm:2.0.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	1	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512	FALSE
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	FALSE
mod-quick-marc	1	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	1	mod-feesfines:19.1.0	2	1024	896	128	768	88	128	FALSE
mod-invoice-storage	1	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512	FALSE
mod-service-interaction	1	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512	FALSE
mod-dcb	1	mod-dcb:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-patron	1	mod-patron:6.1.0	2	1024	896	128	768	88	128	FALSE
mod-data-export	1	mod-data-export:5.0.0	1	2048	1524	1024	0	0	0	FALSE
mod-oai-pmh	1	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512	FALSE
edge-connexion	1	edge-connexion:1.2.0	2	1024	896	128	768	88	128	FALSE
mod-notes	1	mod-notes:5.2.0	2	1024	896	128	952	384	512	FALSE
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	FALSE
mod-login	1	mod-login:7.11.0	2	1440	1298	1024	768	384	512	FALSE
mod-organizations-storage	1	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128	FALSE
mod-data-export-spring	1	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512	FALSE
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	1	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128	FALSE

Methodology/Approach

Run data preparation script before each CI/CO test
Update .jmx file script for Quesnelia release
Test performed from load generator in the same region US East (N. Virginia)us-east-1
Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
Test CI/CO with 30 users for 24 hours to detect any trends in memory (only 16 hours results available).

Folio Development Teams

[ECS] Check-in-check-out Test Report (Quesnelia)

Overview

Summary

Recommendations

Test Runs

Results

Errors:

Resources Utilization

CPU Utilization

Memory Consumption

RDS CPU Utilization

RDS Database Connections

CPU (User) usage by broker

Appendix

Infrastructure

Methodology/Approach