Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.

PERF-879 - Getting issue details... STATUS

Summary IN-PROGRESS

Check-In/Check-Out tests on Quesnelia release (Eureka) in QECP1 environment carried out with 8, 20, 30, 75 virtual users during 45 minutes and with 30 virtual users during longevity test (24 hours) to detect any trends in resource utilization

45 minute tests

Longevity test

Comparison with Poppy Results:

Resources

CPU utilization

Memory consumption

RDS CPU utilization average

CPU (User) usage by broker

Common notes

Recommendations

Test Runs

The following table contains tests configuration information

Test #	vUsers	Ramp-up, sec	Duration, sec
1	8	80	2700
2	20	200	2700
3	30	300	2700
4	75	750	2700
5	30	300	86400

Results

Errors:

Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. 422/Unprocessable Entity. Happen expectedly if instance was checked out already. Error rate - 0.06% which is acceptable.

Response time

The table contains results of Check-in, Check-out tests in Quesnelia release (Eureka).

45 minute tests

	8 vUsers			20 vUsers			30 vUsers			75 vUsers
Requests	Response Times (ms)			Response Times (ms)			Response Times (ms)			Response Times (ms)
Label	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct	#Samples	Average	95th pct
Check-Out Controller
Check-In Controller
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)
GET_circulation/loans (Submit_barcode_checkout)

Longevity test (with absent dcb-system-user)

30 vUsers Longevity
Requests	Samples, Response Times
Label	#Samples	95th pct	Average
Check-Out Controller
Check-In Controller
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)

Comparisons

This table has comparison between average values of response times of Quesnelia and Quesnelia (Eureka) releases

	8 vUsers				20 vUsers				30 vUsers				75 vUsers
Requests	Response Times				Response Times				Response Times				Response Times
	Poppy	Quesnelia			Poppy	Quesnelia			Poppy	Quesnelia			Poppy	Quesnelia
Label	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%
Check-Out Controller
Check-In Controller

Comparison of Longevity test (absent dcb-system-user in Quesnelia)

30 vUsers Longevity
Response Times
Poppy	Quesnelia
Average	Average	Delta,ms	Difference,%

API requests where response times >= 100 ms

API

30 vUsers Poppy

Average, ms

30 vUsers Quesnelia

Average, ms

POST checkout-by-barcode

POST checkin-by-barcode

GET circulation/loans

Resources Utilization

CPU Utilization

45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-users-b - 132%, mod-authtoken-b - 98%, nginx-okapi - 84%.
45 minute tests

CPU by modules

Longevity test (with absent dcb-system-user)

Memory Consumption

45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%

Memory usage by modules

45 minute tests

Longevity test

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test.

45 minute tests

Longevity test

RDS Database Connections

For both 45 minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.

45 minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 15% with equal distribution between brokers

45 minute tests

Longevity test

Database load

75 vUsers

Resources Utilization

CPU Utilization

45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-dcb-b - 407% with 128 CPU, mod-authtoken-b - 120%, nginx-okapi - 102%.
Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU.

45 minute tests

CPU utilization by modules

Service Name	CPU 8 vUsers	CPU 20 vUsers	CPU 30 vUsers	CPU 75 vUsers, 128	CPU 30 vUsers Longevity	CPU 75 vUsers, 256
mod-users-b	29.31	65.11	89.61	206.63	96.57	199.92
mod-dcb-b	21.53	46.25	69.61	407.79	155.82	201.69
okapi-b	16.52	38.85	55.66	90.6	56.77	93
mod-inventory-b	14.28	18.17	17.97	15.14	15.58	15.77
mod-authtoken-b	12.08	20.81	32.34	120.62	30.98	115.95
mod-configuration-b	10.73	21.98	33.95	71.33	30.02	70.5
mod-pubsub-b	9.17	11.67	13.61	18.09	12.39	26.5
mod-feesfines-b	8.26	17.1	23.88	54.79	28.87	54.37
mod-quick-marc-b	7.2	7.63	7.57	8.19	7.93	7.88
pub-okapi	3.91	9.45	13.99	33.44	12.47	27.68
mod-inventory-storage-b	3.44	7.65	11.99	13.24	10.67	12.16
nginx-okapi	2.56	6.35	9.35	102.62	7.69	69.58
mod-patron-b	2.31	2.52	2.52	2.3	2.28	2.44
mod-oa-b	2.28	2.17	2.67	2.75	3.17	3.52
mod-di-converter-storage-b	2.2	2.19	2.29	2.39	2.12	1.97
mod-source-record-manager-b	1.93	1.9	1.87	1.88	1.81	1.95
mod-data-import-b	1.7	1.02	1.02	0.96	1.57	1.59
mod-source-record-storage-b	1.68	1.72	1.73	1.77	1.76	1.83
mod-circulation-storage-b	1.46	2.59	3.64	7.43	3.53	7.06
mod-circulation-b	1.46	3.54	4.58	10.25	5.62	11.21
mod-search-b	1.21	2.66	2.34	3.28	2.94	3.94
edge-patron-b	0.92	0.96	1	1.06	0.95	0.97
mod-patron-blocks-b	0.75	1.38	1.8	2.74	2.38	2.97
mod-users-bl-b	0.51	0.54	0.53	0.5	0.58	0.66
mod-entities-links-b	0.35	0.36	0.44	0.42	0.4	0.39
pub-edge	0.14	0.06	0.1	0.08	0.07	0.11

45 minute test with 75 vUsers (revision with 256 CPU for mod-dcb-b)

Longevity test

Memory Consumption

45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.

Memory usage by modules

Service Name	Memory 75 vUsers	Memory 30 vUsers Longevity
mod-oa-b	81	79.24
mod-dcb-b	68.63	69.16
mod-users-b	63.65	62.98
mod-search-b	55.34	40.83
mod-pubsub-b	54.79	61.98
okapi-b	48.94	60.38
mod-inventory-b	47.49	50.79
mod-source-record-manager-b	45.06	45.76
mod-quick-marc-b	43.51	29.92
mod-di-converter-storage-b	35.02	29.58
mod-feesfines-b	34.18	34.12
mod-source-record-storage-b	33.76	31.37
mod-inventory-storage-b	33.12	40.06
mod-entities-links-b	32.3	25.08
mod-configuration-b	32.07	30.21
mod-data-import-b	31.31	39.73
mod-patron-blocks-b	31.13	30.92
mod-users-bl-b	28.95	25.82
mod-circulation-b	27.41	26.71
mod-patron-b	26.62	25.17
mod-circulation-storage-b	26.43	28.83
mod-authtoken-b	21.84	22.1
edge-patron-b	18.3	18.14
nginx-okapi	4.85	4.8
pub-okapi	4.69	4.69
pub-edge	4.58	4.63

45 minute tests

Longevity test

RDS CPU Utilization

45 minute tests

Longevity test

RDS Database Connections

45 minute tests

Longevity test

CPU (User) usage by broker

45 minute tests

Longevity test

Database load

Appendix

Infrastructure

PTF -environment qecp1
11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 1 instance of db.r6.xlarge database instance: Writer instance MSK tenant [ kafka configurations] 4 kafka.m5.2xlarge brokers in 2 zones Apache Kafka version 2.8.0 EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=3

DB table records size:

fs09000000
- instances - 24493326
- items - 25662862
- holding records - 24459683

Modules

Check-In / Check-Out related modules

All modules

Methodology/Approach

Run data preparation script before each CI/CO test
Update .jmx file script for Quesnelia release
Create artefact and upload to carrier-io
Use Jenkins job to change parameters and run tests
Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
Test CI/CO with 30 users for 24 hours to detect any trends in memory.

[Eureka] Check-in-check-out Test Report (Quesnelia)

Overview

Summary IN-PROGRESS

Recommendations

Test Runs

Results

Errors:

Comparisons

Resources Utilization

CPU Utilization

Memory Consumption

RDS CPU Utilization

RDS Database Connections

CPU (User) usage by broker

Database load

Appendix

Infrastructure

Methodology/Approach