Overview

This is initial report for EBSCONET orders renewal workflow testing PERF-285 - Getting issue details... STATUS . The purpose of this document is to highlight KPIs of EBSCONET workflow, find possible bottlenecks/ issues, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.

Summary

During testing we've find several issues:

NullPointerException
Internal server error

No memory leaks were found.

High level results:

Test #	VUsers	Duration	avg	75%	95%	avg	75%	95%	Error rate
Test #	VUsers	Duration	response time (get) ms.			response time (put) ms			Error rate
1	2	7 min 10 s	249	257	287	178	184	197	2.6% (52 calls)
2	2	7 min	247	254	288	178	184	197	3.3% (67 calls)
3	10	4 min 30 s	298	319	404	236	253	339	1.14% (57 calls)
4	10	4 min 10 s	308	332	417	242	263	465	1.56% (78 calls)

Recommendations & Jiras

MODEBSNET-56 - Getting issue details... STATUS Ticket to handle NullPointerException Which is reproducible even with 2 users.
MODEBSNET-57 - Getting issue details... STATUS Ticket to handle internal server error .
We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)

Test Runs

Test #	VUsers	Data set	Load generator size	Load generator Memory(GiB)
normal conditions	2 users	Number of order lines being renewed 1000 Number of orders being renewed 804 4 of these orders have 50 POLs each - 800 orders with 1 POL each	t3.medium	3
2. normal conditions	2 users	Number of order lines being renewed 1000 Number of orders being renewed 804 4 of these orders have 50 POLs each - 800 orders with 1 POL each	t3.medium	3
3. extreme conditions	10 users	Number of order lines being renewed 5000 Number of orders being renewed 4804 4 of these orders have 50 POLs each - 4800 orders with 1 POL each	t3.medium	3
4 extreme conditions	10 users	Number of order lines being renewed 5000 Number of orders being renewed 4804 4 of these orders have 50 POLs each - 4800 orders with 1 POL each	t3.medium	3

Results

Test #	VUsers	Duration	avg	75%	95%	avg	75%	95%	Error rate
Test #	VUsers	Duration	response time (get) ms.			response time (put) ms			Error rate
1	2	7 min 10 s	249	257	287	178	184	197	2.6% (52 calls)
2	2	7 min	247	254	288	178	184	197	3.3% (67 calls)
3	10	4 min 30 s	298	319	404	236	253	339	1.14% (57 calls)
4	10	4 min 10 s	308	332	417	242	263	465	1.56% (78 calls)

Memory Utilization

	Nolana Avg%
mod-organizations	25%
mod-organizations-storage	24%
mod-finance	29%
mod-finance-storage	28%
nginx-edge	2%
mod-ebsconet	37%
nginx-okapi	3%
mod-orders	44%
mod-orders-storage	34%
okapi	36%
mod-notes	43%
mod-configuration	25%
edge-orders	20%

CPU Utilization

Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage

RDS CPU Utilization

Appendix

Infrastructure

PTF -environment ncp3-pvt

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameter

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-organizations	1.6.0	2	2	128	1024	896	128	700
mod-organizations-storage	4.4.0	2	2	128	1024	896	128	700
mod-finance	4.6.2	2	2	128	1024	896	128	700
mod-finance-storage	8.3.1	2	2	128	1024	896	128	700
nginx-edge	nginx-edge:2022.03.02	1	2	128	1024	896	N/A	N/A
mod-ebsconet	1.4.0	2	2	128	1024	896	256	700
nginx-okapi	nginx-okapi:2022.03.02	1	2	128	1024	896	N/A	N/A
mod-orders	12.5.4	2	2	1024	2048	1440	256	896
mod-orders-storage	13.4.0	2	2	128	1024	896	128	700
okapi	4.14.7	1	3	1024	1684	1440	922	922
mod-mod-notes	4.0.0	2	2	128	1024	896	128	322
mod-configuration	5.9.0	3	2	128	1024	896	128	768
edge-orders	2.7.0	2	2	128	1024	896	128	700

Folio Development Teams

EBSCONet Orders Renewal (Nolana)