1 Overview
2 Summary
- 2.1 Recommendations & Jiras
3 Test Runs
4 Results
- 4.1 Memory Utilization
- 4.2 CPU Utilization
- 4.3 RDS Metrics
- 4.4 Additional information from module and database logs
5 Appendix
- 5.1 Infrastructure
- 5.2 Methodology/Approach

Overview

Thees tests performed In order to define baseline response time and resource usages. Find possible issues/red flags in infrastructure and/or workflows behaviour.

Summary

Low memory on 4xLarge DB during test. this was first assumption that it’s causing slowness.
Memory on 4xLarge DB is low even when system in idle state. (±30 Gb from 120Gb available on instance type). May be due to high disk swap rate, lots of indexes and schemas
With 8xLarge DB - response times for most workflows increases from 300 ms up to few seconds. However error rate becomes lower.
with 8xLarge DB - problem with low RAM dissapered, however it’s still consuming a lot of men in idle state.
Multiple (optimistic locking) errors was found in DB logs

Recommendations & Jiras

Original ticket: PERF-807 Fixed load tests of MOBIUS workflows.

Test Runs

Test #

Test Conditions

Duration

Load generator size (recommended)

Load generator Memory(GiB) (recommended)

Notes

(Optional)

Mobius baseline test

30 mins

m5a.xlarge

9

4xLarge DB

2.

Mobius baseline test

30 mins

m5a.xlarge

9

8xLarge DB

Results

Flow	Test 1 4xLarge DB	error count	Test 2 8xLarge DB	error count

Flow	Test 1 4xLarge DB	error count	Test 2 8xLarge DB	error count
CI	6,42	1	6,707	0
CO	10,52	4	10,820	0
create invoice	34,73	0	35,422	0
approve invoice	33,13	0	33,008	0
pay invoice	31,119	0	31,354	0
sahre local instance	11,018	43	11,500	19
export custom workflow	353,296	0	327,27	0
export default	336,563	0	338,833	0
view account	22,45	21	22,642	0
create ILR	5,085	2	5,372	0
search	1,88	52	3,519	12
create order	99,937	0	97,97	0
add order line	185,765	415!!!!!	186,44	415!!!!
approve order	146,352	0	143,998	0
RTAC	33,15	0	32,68	0
singe record create	30,395	44	29,336	19
single record update	96,046	3	94,419	1
receiving order line	307,271	0	301,103	0
serial receiving	282,336	0	279,132	0
unrecieving a piece	106,148	0	103,033	0
renew	8,023	6	8,341	0

Note: Here with 8XLarge DB comparing to 4XLarge DB response times is higher, however errors count is smaller.

Note: add order line add order line workflow has high error rate because of one call that is failing all the time. [GET] erm/entitlements failing with 503.

Memory Utilization

Note: for most of a modules memory utilisation looks good and without growing trend. Except few modules on next chart.

Note: here - few modules behaviour is suspicious. Most disturbing ones is mod-inventory and mod-search

CPU Utilization

Note: most stressed modules: mod-authtoken -110%, okapi - 70% in both tests

RDS Metrics

Note: Here, predictably 8xLarge DB has lower CPU utilisation than 4XLarge DB.

For 4XLarge DB - CPU utilisation did reach ±90% only once at the4 beginning of a test. Average CPU utilisation for 4XLarge is 50% , for 8Xlarge 30%.

Note: RDS connections rate is more or less the same for 4xLarge and 8Xlarge and approximately is 3-3,5K concurrent connections.

Note: Here we can see that with 4XLarge DB we are running low on available memory. As you can see - before the test it has only 30Gb available (while DB instance itself has 120Gb), and during test it doped to >1G.

Additional information from module and database logs

Appendix

Infrastructure

PTF -environment mcpt

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r6.4xlarge (db.r6.8xlarge) database instance, one writer
ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2

Module mcpt-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize

Module mcpt-pvt	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
mod-remote-storage	10	mod-remote-storage:3.0.1	2	4920	4472	1024	3960	512	512
mod-ncip	6	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	6	mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128
mod-agreements	11	mod-agreements:6.0.2	0	1592	1488	1024	0	0	0
mod-ebsconet	6	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256
edge-sip2	6	edge-sip2:3.1.1	2	1024	896	128	768	88	128
mod-consortia	11	mod-consortia:1.0.3	2	2048	1802	128	768	88	128
mod-organizations	6	mod-organizations:1.8.0	2	1024	896	128	700	88	128
mod-settings	11	mod-settings:1.0.2	2	1024	896	200	768	88	128
edge-dematic	9	edge-dematic:2.1.1	1	1024	896	128	768	88	128
mod-data-import	7	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512
mod-search	12	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024
mod-tags	6	mod-tags:2.1.0	2	1024	896	128	768	88	128
mod-authtoken	7	mod-authtoken:2.14.1	2	1440	1152	1024	922	88	128
edge-courses	6	edge-courses:1.3.0	2	1024	896	128	768	88	128
mod-inventory-update	6	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128
mod-notify	6	mod-notify:3.1.0	2	1024	896	128	768	88	128
mod-configuration	6	mod-configuration:5.9.2	2	1024	896	128	768	88	128
mod-orders-storage	7	mod-orders-storage:13.6.0	2	1024	896	512	700	88	128
edge-caiasoft	6	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128
mod-login-saml	9	mod-login-saml:2.7.2	2	1024	896	128	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128
mod-password-validator	6	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512
mod-gobi	6	mod-gobi:2.7.1	2	1024	896	128	700	88	128
mod-licenses	9	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512
mod-fqm-manager	6	mod-fqm-manager:1.0.3	2	3000	2600	128	2048	384	512
mod-bulk-operations	12	mod-bulk-operations:1.1.9	2	3072	2600	1024	1536	384	512
mod-graphql	6	mod-graphql:1.12.0	0	1024	896	128	768	88	128
mod-finance	6	mod-finance:4.8.0	2	1024	896	128	700	88	128
mod-erm-usage	7	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128
mod-lists	9	mod-lists:1.0.5	2	3000	2600	128	2048	384	512
mod-copycat	6	mod-copycat:1.5.0	2	1024	896	128	768	88	128
mod-permissions	14	mod-permissions:6.4.0	2	1684	1544	512	1024	384	512
mod-entities-links	10	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	7	mod-orders:12.7.1	2	2048	1440	1024	1024	384	512
edge-patron	6	edge-patron:5.0.0	2	1024	896	256	768	88	128
edge-ncip	6	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	6	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128
edge-ea-data-export	1	edge-ea-data-export:4.0.1	2	1024	896	128	768	88	128
mod-invoice	7	mod-invoice:5.7.2	2	1440	1152	512	922	88	128
mod-inventory-storage	10	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512
mod-user-import	6	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	6	mod-sender:1.11.0	2	1024	896	128	768	88	128
edge-oai-pmh	9	edge-oai-pmh:2.7.2	2	1512	1360	1024	1440	384	512
mod-data-export-worker	9	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512
mod-rtac	6	mod-rtac:3.5.0	2	1024	896	128	768	88	128
mod-task-list	1	mod-task-list:1.9.2	2	1024	896	128	768	88	128
mod-circulation-storage	10	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512
mod-calendar	6	mod-calendar:2.5.0	2	1024	896	128	768	88	128
mod-source-record-storage	10	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512
mod-event-config	6	mod-event-config:2.6.0	2	1024	896	128	768	88	128
mod-courses	6	mod-courses:1.4.8	2	1024	896	128	768	88	128
mod-inventory	11	mod-inventory:20.1.8	2	2880	2592	1024	1814	384	512
mod-email	6	mod-email:1.16.0	2	1024	896	128	768	88	128
mod-circulation	12	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512
mod-pubsub	10	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512
mod-di-converter-storage	9	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128
edge-rtac	9	edge-rtac:2.6.2	2	1024	896	128	768	88	128
edge-orders	9	edge-orders:2.9.1	2	1024	896	128	768	88	128
mod-template-engine	6	mod-template-engine:1.19.1	2	1024	896	128	768	88	128
mod-users	11	mod-users:19.2.2	2	1024	896	128	768	88	128
mod-patron-blocks	6	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128
mod-audit	7	mod-audit:2.8.0	2	1024	896	128	768	88	128
edge-fqm	7	edge-fqm:1.0.1	2	1024	896	128	768	88	128
mod-source-record-manager	10	mod-source-record-manager:3.7.7	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	10	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	512	0	0	0
okapi-b	1	okapi:5.1.2	3	1684	1440	1024	922	384	512
mod-feesfines	6	mod-feesfines:19.0.0	2	1024	896	128	768	88	128
mod-invoice-storage	6	mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512
mod-service-interaction	9	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512
mod-data-export	10	mod-data-export:4.8.7	1	1024	896	1024	768	88	128
mod-patron	6	mod-patron:6.0.0	2	1024	896	128	768	88	128
mod-oai-pmh	9	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512
edge-connexion	6	edge-connexion:1.1.1	2	1024	896	128	768	88	128
mod-kb-ebsco-java	6	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	6	mod-notes:5.1.0	2	1024	896	128	952	384	512
mod-data-export-spring	9	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512
mod-login	6	mod-login:7.10.1	2	1440	1298	1024	768	384	512
mod-organizations-storage	7	mod-organizations-storage:4.6.0	2	1024	896	128	700	88	128
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
edge-erm	1	edge-erm:1.0.0	2	1024	896	128	768	88	128
mod-eusage-reports	7	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128

Front End: [ front end app versions (optional)]

Item Check-in (folio_checkin-7.2.0)
Item Check-out (folio_checkout-8.2.0)