Overview

The purpose of the tests is to measure API calls in Quesnelia release in scope of Performance testing of UXPROD-4070 - Reading Room Access. Ticket: PERF-908 - Getting issue details... STATUS
A new feature was deployed as a new App - Reading Room to qcon environment. It should support maximum 25 simultaneous requests to reading rooms API.

Summary

Tests finished successfully for all Deny and Allow access flows. No errors.
Average response time for POST requests were 30 ms and for the whole flow - 145 ms.
Resource utilization:
- Module CPU
  - mod-users-bl-b - 63%, mod-users-b - 36%, mod-authtoken-b - 27%, mod-reading-room-b - 25%
- Module Memory
  - mod-data-export-worker-b - 85%, mod-inventory-b - 80%, mod-consortia-b - 66%, mod-orders-b - 64%, mod-dcb-b - 60%, mod-reading-room-b - 52%, mod-search-b - 48%
- RDS CPU - 22%

Test Runs

This table contains response time

Test #	Test Conditions	Duration	Status
1.	25 vUsers simultaneously	20 mins	Success

Results

The table contains results for testing API requests. TC_RRO is a transaction controller which include all steps during the flow (entering Reading Room App, searching patron by patron id, deny or allow action with 50% probability).

Requests	Executions		Response Times (ms)
Label	#Samples	FAIL	Average	90th pct
TC_RRO: Deny access flow	3369	0	146	166
TC_RRO: Allow access flow	3382	0	147	166
POST Deny /reading-room/{readingRoomsId}/access-log	3370	0	30	34
POST Allow/reading-room/{readingRoomsId}/access-log	3382	0	30	34

GET /users/configurations/entry	6758	0	21	27
GET /users	6754	0	32	41
GET /reading-room-patron-permission/{patronId}	6754	0	34	37
GET /reading-room	6758	0	28	30

Memory Utilization

Memory consumption

Module	Memory
mod-data-export-worker-b	84.84
mod-inventory-b	80.47
mod-consortia-b	66.7
mod-orders-b	63.49
mod-dcb-b	60.42
mod-reading-room-b	52.41
mod-search-b	48.77
mod-agreements-b	48.26
mod-invoice-b	47.25
mod-notes-b	46.03
mod-copycat-b	45.8
mod-service-interaction-b	45.27
mod-calendar-b	44.72
mod-circulation-item-b	43.46
mod-data-import-b	43
mod-source-record-manager-b	42.77
mod-tags-b	42.69
mod-serials-management-b	42.21
mod-users-b	41.9
mod-licenses-b	40.57
mod-data-export-b	39.83
mod-erm-usage-harvester-b	38.79
edge-dcb-b	35.83
mod-permissions-b	35.58
edge-caiasoft-b	35.38
edge-dematic-b	34.93
edge-courses-b	34.77
mod-source-record-storage-b	34.61
mod-orders-storage-b	34.01
okapi-b	33.82

Memory graph do not show any growing trend during testing

CPU Utilization

Reading room related services worked without any spikes.

CPU utilization

Module	CPU
mod-permissions-b	202.19
mod-users-bl-b	62.73
mod-users-b	36.07
mod-authtoken-b	27.61
mod-reading-room-b	25.16
nginx-okapi	22.47
mod-consortia-b	20.09
okapi-b	14.72
mod-inventory-b	11.1
mod-configuration-b	8.88
mod-quick-marc-b	7.3
mod-pubsub-b	7.21
mod-dcb-b	7.05
mod-audit-b	6.32
pub-okapi	5.55
mod-erm-usage-harvester-b	4.34
edge-dematic-b	3.66
mod-licenses-b	3.07
mod-kb-ebsco-java-b	2.72
mod-tags-b	2.63
mod-service-interaction-b	2.58
mod-circulation-item-b	2.55
mod-sender-b	2.44
mod-notes-b	2.44
mod-erm-usage-b	2.41
mod-batch-print-b	2.39
mod-template-engine-b	2.39
mod-rtac-b	2.39
mod-feesfines-b	2.37
mod-lists-b	2.36

MSK cluster

CPU (User) usage by broker

Disk usage by broker

Open Search

Dedicated master nodes

Data nodes

RDS CPU Utilization

RDS CPU utilized 22% during testing

RDS Database Connections

Appendix

Infrastructure

PTF - environment qcon

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer

Name	Memory GIB	vCPUs	Engine version
db.r6g.xlarge	32 GB	4 vCPUs	16.1

MSK tenant
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
Open Search fse
- Data nodes Instance type: r6g.xlarge.search
- Number of nodes: 6, dedicated master nodes - 3
- Version: OpenSearch 2.13 (latest)
- EBS volume size (GiB): 500 GiB
Number of records in DB:
- cs00000int_0001
  - instances - 6828236
  - items - 7859770
  - holdings - 7328737
- cs00000int_0002
  - instances - 1163315
  - items - 1754121
  - holdings - 1332559
- cs00000int_0003
  - instances - 1135806
  - items - 1735291
  - holdings - 1309387

Modules

Modules QCON

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-finance-storage	1	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-ncip	1	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-agreements	1	mod-agreements:7.0.0	2	1592	1488	128	0	0	0
mod-ebsconet	1	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-consortia	1	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
mod-organizations	1	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-serials-management	1	mod-serials-management:1.0.0	2	2480	2312	128	1792	384	512
mod-settings	1	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-search	9	mod-search:3.3.0-SNAPSHOT.224	2	2592	2480	2048	1440	512	1024
edge-dematic	1	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-data-import	1	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
mod-tags	1	mod-tags:2.2.0	2	1024	896	128	768	88	128
mod-authtoken	5	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-inventory-update	1	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	1	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	1	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	1	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	1	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-password-validator	1	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-licenses	1	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	1	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-bulk-operations	1	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	1	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	1	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	1	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	1	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	1	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	1	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	1	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	1	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	1	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	4	mod-permissions:6.6.0-SNAPSHOT.379	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	1	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	1	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-invoice	1	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-inventory-storage	2	mod-inventory-storage:27.2.0-SNAPSHOT.738	2	4096	3690	2048	3076	384	512
mod-user-import	1	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	1	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-source-record-storage	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-calendar	1	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-event-config	1	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	1	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	3	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.2.2	2	1024	896	128	768	88	128
edge-orders	1	edge-orders:3.0.0	2	1024	896	128	768	88	128
edge-rtac	1	edge-rtac:2.7.1	2	1024	896	128	768	88	128
mod-users	2	mod-users:19.3.2-SNAPSHOT.691	2	1024	896	128	768	88	128
mod-template-engine	1	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	1	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	1	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	1	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-reading-room	7	mod-reading-room:1.0.0-SNAPSHOT.11	2	1024	896	128	768	88	128
mod-service-interaction	1	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-patron	1	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-data-export	1	mod-data-export:5.0.0	1	2048	1524	1024	0	0	0
mod-oai-pmh	1	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-notes	1	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-login	1	mod-login:7.11.0	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	1	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Quesnelia release

Methodology/Approach

Test flow:
1. Go to Reading Room App
2. Search patron by barcode from prepared file with patron ids
3. Deny or Allow access with 50% probability
Configuration of load:
1. 25 vUsers during 20 minutes
2. Ramp-up - 250 seconds (10 seconds for each vUser)
Preparing to testing:
1. Generate the csv file with patron barcodes
2. Reading Room should be public to make it possible to Deny or Allow access (by default - Allowed)

Performance testing of UXPROD-4070 - Reading Room Access