Overview

The purpose of the tests is to measure API calls in Poppy release in scope of PERF-804 - Getting issue details... STATUS ticket on PCP1 environment.

Summary

API with S3 bucket storage work faster with big files. More information in Test Results section.
API with S3 bucket storage handle concurrent requests better than API with DB storage. With DB response times increased twice or more, with S3 bucket increased 25% - 45%.
Some growing trend detected for CPU utilization in mod-users module.

Tests with 1 API request POST /users/profile-picture sequentially with 1 user and 2 API requests POST /users/profile-picture, GET /users/profile-picture/[profile] with 3 users concurrently
- Test set #1
  - 1000 requests x 100kb
  - 1000 requests x 3.6Mb
  - 1000 requests x 5Mb
- Test set #2
  - 100 requests x 100kb
  - 100 requests x 3.6Mb
  - 100 requests x 5Mb
  - 100 requests x 10Mb

Results

Test set #1
- DB storage
  - Test #2 1000 request
    - Total duration: 46 minutes
      - Duration 100kb - < 1 minute, Response times - 24 ms
      - Duration 3.6Mb - 19 minutes, Response times - 1115 ms
      - Duration 5Mb - 26 minutes, Response times - 1588 ms
- S3 bucket storage
  - Test #2 1000 requests
    - Total duration: 19 minutes
      - Duration 100kb - 1 minute 50 sec, Response times - 111 ms
      - Duration 3.6Mb - 6 minutes, Response times - 380 ms
      - Duration 5Mb - 11 minutes, Response times - 642 ms
Test set #2
- DB storage
  - 1 user
    - 100kb - Response times - 25 ms
    - 3.6Mb - Response times - 1074 ms
    - 5Mb - Response times - 1559 ms
    - 10Mb - Response times - 2889 ms
  - 3 users
    - 100kb - Response times - 29 ms
    - 3.6Mb - Response times - 2301 ms
    - 5Mb - Response times - 3523 ms
    - 10Mb - Response times - 5851 ms
- S3 bucket storage
  - 1 user
    - 100kb - Response times - 128 ms
    - 3.6Mb - Response times - 394 ms
    - 5Mb - Response times - 709 ms
    - 10Mb - Response times - 1216 ms
  - 3 users
    - 100kb - Response times - 150ms
    - 3.6Mb - Response times - 532 ms
    - 5Mb - Response times - 926 ms
    - 10Mb - Response times - 1858 ms

Resource utilization

Test set #1
- CPU utilization mod-users module
  - Test with DB storage - 63%
  - Test with S3 bucket - 170%
- Memory consumption mod-users module
  - Test with DB storage - 54%
  - Test with S3 bucket - 54%

Test set #2
- CPU utilization mod-users module
  - Test with DB storage - 1 user - 148%, 3 users - 250%
  - Test with S3 bucket - 1 user - 254%, 3 users - 547%
- Memory consumption mod-users module
  - Test with DB storage - 73%
  - Test with S3 bucket - 77%

Recommendations & Jiras

For S3 bucket storage tests a bucket was created with name fs09000000 (tenant Id) and AWS_URL created in task definition to point S3 service. It's recommended to add cluster name to the link as well to exclude possibility of cross requests from different clusters to the same bucket.
It's impossible to detect file name or file size or createdDate for the records in DB [tenant]_mod_users.profile_picture table. Consider adding this to get more info about uploaded profile pictures.
Allocate more CPU for mod-users as it grew up to 550% during 3 concurrent requests.
Before allocating more CPU resources investigate mod-users behaviour because during Test set#2 of tests it started with 144% with 1 request on DB storage. But previously we saw that it started with 63%.

Test Runs & Results

Test set #1 - 1000 requests (POST)

		DB Storage			S3 bucket Storage
	Requests	Executions		Response Times (ms)	Executions		Response Times (ms)
	Label	#Samples	Error %	Average	#Samples	Error %	Average
1 request tests	POST /users/profile-picture 3.6MB	1	0.00%	674	1	0.00%	870
	POST /users/profile-picture 5MB	1	0.00%	765	1	0.00%	1244
	POST /users/profile-picture 100kB	1	0.00%	251	1	0.00%	721

1000 request tests	POST /users/profile-picture 3.6MB	1000	0.00%	1115.44	1000	0.00%	380.64
	POST /users/profile-picture 5MB	1000	0.00%	1588.01	1000	0.00%	642.39
	POST /users/profile-picture 100kB	1000	0.00%	23.85	1000	0.00%	111.38

Test set #2 - 100 requests (POST, GET)

		DB Storage						S3 bucket Storage
		1 user			3 users			1 user			3 users
	Requests	Executions		Response Times (ms)	Executions		Response Times (ms)	Executions		Response Times (ms)	Executions		Response Times (ms)
	Label	#Samples	Error %	Average	#Samples	Error %	Average	#Samples	Error %	Average	#Samples	Error %	Average
100 /POST	POST /users/profile-picture 100kB	100	0.00%	25.08	300	0.00%	29.12	100	0.00%	128.9	300	0.00%	150.73
	POST /users/profile-picture 3.6MB	100	0.00%	1074.09	300	0.00%	2301.68	100	0.00%	394.64	300	0.00%	532.75
	POST /users/profile-picture 5MB	100	0.00%	1555.85	300	0.00%	3523.36	100	0.00%	709.25	300	0.00%	926.81
	POST /users/profile-picture 10MB	100	0.00%	2889.33	300	0.00%	5851.31	100	0.00%	1216.04	300	0.00%	1858.78

100 /GET	GET /users/profile-picture/[ppid] 100kB	100	0.00%	23.85	300	0.00%	29.47	100	0.00%	52.37	300	0.00%	85.17
	GET /users/profile-picture/[ppid] 3.6MB	100	0.00%	382.76	300	0.00%	391.92	100	0.00%	401.89	300	0.00%	449.69
	GET /users/profile-picture/[ppid] 5MB	100	0.00%	532.76	300	0.00%	568.9	100	0.00%	569.52	300	0.00%	773.59
	GET /users/profile-picture/[ppid] 10MB	100	0.00%	962.61	300	0.00%	1014.88	100	0.00%	1014.96	300	0.00%	1166.17

Test set #2

Service CPU Utilization

Service Memory Consumption

During 3 concurrent profile picture uploading to S3 bucket mod-users used 77%. Uploading to DB mod-users utilized 73%.

Test set #1 Test #2 with DB storage

Service CPU Utilization

mod-users module CPU Utilization was 63% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

RDS CPU Utilization

RDS Database Connections

RDS connections level before and during test - 385

Test set #1 Test #2 with S3 bucket storage source

Service CPU Utilization

mod-users module CPU Utilization was 170% during tests

Service Memory Consumption

mod-users module memory consumption was 54%

Response Times Over Time

Test set #1 Test #2 with DB storage 1 user 1000 requests

Test set #1 Test #2 with S3 bucket storage 1 user 1000 requests

Test set #2 Test #2 with DB storage 1 user 100 requests

Test set #2 Test #2 with S3 bucket storage 1 user 100 requests

Test set #2 Test #2 with DB storage 3 user 100 requests

Test set #2 Test #2 with S3 bucket storage 3 user 100 requests

Appendix

Methodology/Approach

To get config Id and change storage source use API
- GET /users/configurations/entry
- PUT /users/configurations/entry/{configId}
To upload picture to DB use API
- POST /users/profile-picture
To get or to check uploaded profile picture use API where profileId the record in DB [tenant]_mod_users.profile_picture
- GET /users/profile-picture/{profileId}
To connect AWS S3 bucket storage add to task definition of mod-users module and restart it with changes in revision
- ```
{
"name": "AWS_URL",
"value": "https://s3.amazonaws.com"
}
```

Infrastructure

Environment: PCP1
Release: Poppy (2023 R2)

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK tenant
- 4 brokers
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- og.retention.minutes=480
- default.replication.factor=3

Modules

All modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcp1-pvt
Thu Feb 29 11:11:48 UTC 2024
mod-remote-storage	11	mod-remote-storage:3.0.0	2	4920	4472	1024	3960	512	512	FALSE
mod-ncip	10	mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-finance-storage	10	mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128	FALSE
mod-agreements	10	mod-agreements:6.0.2	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	10	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256	FALSE
edge-sip2	8	edge-sip2:3.1.1	2	1024	896	128	768	88	128	FALSE
mod-organizations	10	mod-organizations:1.8.0	2	1024	896	128	700	88	128	FALSE
mod-settings	11	mod-settings:1.0.2	2	1024	896	200	768	88	128	FALSE
edge-dematic	10	edge-dematic:2.1.0	1	1024	896	128	768	88	128	FALSE
mod-data-import	20	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-search	20	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024	FALSE
mod-tags	10	mod-tags:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-authtoken	16	mod-authtoken:2.14.1	2	1440	1152	512	922	88	128	FALSE
edge-courses	2	edge-courses:1.3.0	2	1024	896	128	768	88	128	FALSE
mod-notify	10	mod-notify:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-inventory-update	10	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128	FALSE
mod-configuration	10	mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-orders-storage	10	mod-orders-storage:13.6.0	2	1024	896	512	700	88	128	FALSE
edge-caiasoft	10	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-login-saml	18	mod-login-saml:2.7.1	2	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	11	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128	FALSE
mod-password-validator	10	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512	FALSE
mod-licenses	10	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512	FALSE
mod-gobi	10	mod-gobi:2.7.1	2	1024	896	128	700	88	128	FALSE
mod-fqm-manager	7	mod-fqm-manager:1.0.3	2	1024	896	128	768	88	128	FALSE
mod-bulk-operations	9	mod-bulk-operations:1.1.7	2	3072	2600	1024	1536	384	512	FALSE
mod-graphql	16	mod-graphql:1.12.0	2	1024	896	128	768	88	128	FALSE
mod-finance	10	mod-finance:4.8.0	2	1024	896	128	700	88	128	FALSE
mod-erm-usage	13	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128	FALSE
mod-copycat	10	mod-copycat:1.5.0	2	1024	896	128	768	88	128	FALSE
mod-lists	6	mod-lists:1.0.5	2	3000	2600	128	2048	384	512	FALSE
mod-entities-links	15	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024	FALSE
mod-permissions	43	mod-permissions:6.3.2	2	1684	1544	512	1024	384	512	FALSE
pub-edge	9	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-orders	10	mod-orders:12.7.1	2	2048	1440	1024	1024	384	512	FALSE
edge-patron	10	edge-patron:5.0.0	2	1024	896	256	768	88	128	FALSE
edge-ncip	11	edge-ncip:1.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	10	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	15	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
mod-invoice	10	mod-invoice:5.7.2	2	1440	1152	512	922	88	128	FALSE
mod-user-import	10	mod-user-import:3.8.0	2	1024	896	128	768	88	128	FALSE
mod-sender	10	mod-sender:1.11.0	2	1024	896	128	768	88	128	FALSE
edge-oai-pmh	8	edge-oai-pmh:2.7.1	2	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	10	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512	FALSE
mod-rtac	10	mod-rtac:3.5.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	14	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	FALSE
mod-calendar	10	mod-calendar:2.5.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-storage	18	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-event-config	10	mod-event-config:2.6.0	2	1024	896	128	768	88	128	FALSE
mod-courses	10	mod-courses:1.4.8	2	1024	896	128	768	88	128	FALSE
mod-inventory	14	mod-inventory:20.1.7	2	2880	2592	1024	1814	384	512	FALSE
mod-email	10	mod-email:1.16.0	2	1024	896	128	768	88	128	FALSE
mod-di-converter-storage	18	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128	FALSE
mod-circulation	14	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	13	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512	FALSE
edge-orders	10	edge-orders:2.9.1	2	1024	896	128	768	88	128	FALSE
edge-rtac	7	edge-rtac:2.6.2	2	1024	896	128	768	88	128	FALSE
mod-template-engine	10	mod-template-engine:1.19.1	2	1024	896	128	768	88	128	FALSE
mod-users	25	mod-users:19.3.0-SNAPSHOT.669	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	10	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	FALSE
edge-fqm	21	edge-fqm:1.0.1	2	1024	896	128	768	88	128	FALSE
mod-audit	10	mod-audit:2.8.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	17	mod-source-record-manager:3.7.8	2	5600	5000	2048	3500	384	512	FALSE
nginx-edge	9	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	FALSE
mod-quick-marc	11	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	11	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	11	mod-feesfines:19.0.0	2	1024	896	128	768	88	128	FALSE
mod-invoice-storage	10	mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512	FALSE
mod-service-interaction	10	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512	FALSE
mod-data-export	12	mod-data-export:4.8.7	1	1024	896	1024	768	88	128	FALSE
mod-patron	10	mod-patron:6.0.0	2	1024	896	128	768	88	128	FALSE
mod-oai-pmh	5	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512	FALSE
edge-connexion	10	edge-connexion:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-kb-ebsco-java	10	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	FALSE
mod-notes	10	mod-notes:5.1.0	2	1024	896	128	952	384	512	FALSE
mod-organizations-storage	10	mod-organizations-storage:4.6.0	2	1024	896	128	700	88	128	FALSE
mod-data-export-spring	12	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512	FALSE
mod-login	10	mod-login:7.10.1	2	1440	1298	1024	768	384	512	FALSE
pub-okapi	9	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	13	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128	FALSE

Overview

Summary

Results

Resource utilization

Recommendations & Jiras

Test Runs & Results

Test set #1 - 1000 requests (POST)

Test set #2 - 100 requests (POST, GET)

Test set #2

Service CPU Utilization

Service Memory Consumption

Test set #1 Test #2 with DB storage

Service CPU Utilization

Service Memory Consumption

RDS CPU Utilization

RDS Database Connections

Test set #1 Test #2 with S3 bucket storage source

Service CPU Utilization

Service Memory Consumption

Response Times Over Time

Test set #1 Test #2 with DB storage 1 user 1000 requests

Test set #1 Test #2 with S3 bucket storage 1 user 1000 requests

Test set #2 Test #2 with DB storage 1 user 100 requests

Test set #2 Test #2 with S3 bucket storage 1 user 100 requests

Test set #2 Test #2 with DB storage 3 user 100 requests

Test set #2 Test #2 with S3 bucket storage 3 user 100 requests

Appendix

Methodology/Approach

GET /users/configurations/entry

PUT /users/configurations/entry/{configId}

POST /users/profile-picture

GET /users/profile-picture/{profileId}

Infrastructure

Browser not supported