EBSCONet Orders Renewal (Nolana)

Overview

This is initial report for EBSCONET orders renewal workflow testing PERF-285 - Getting issue details... STATUS . The purpose of this document is to highlight KPIs of EBSCONET workflow, find possible bottlenecks/ issues, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.

Summary

During testing we've find several issues: 

  • NullPointerException
  • Internal server error

No memory leaks were found. 

High level results:

Test #VUsersDurationresponse time (get) ms.

response time (put) ms

Error rate
avg75%95%avg75%95%
127 min 10 s249 2572871781841972.6% (52 calls)
227 min2472542881781841973.3% (67 calls)
3104 min 30 s2983194042362533391.14% (57 calls)
4104 min 10 s3083324172422634651.56% (78 calls)


Recommendations & Jiras

  • MODEBSNET-56 - Getting issue details... STATUS  Ticket to handle NullPointerException Which is reproducible even with 2 users.
  • MODEBSNET-57 - Getting issue details... STATUS Ticket to handle internal server error .
  • We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)

Test Runs 

Test #

VUsers

Data setLoad generator size Load generator Memory(GiB) 
  1. normal conditions
2 users
  • Number of order lines being renewed 1000
  • Number of orders being renewed 804
  • 4 of these orders have 50 POLs each - 800 orders with 1 POL each


t3.medium3
2. normal conditions2 users
  • Number of order lines being renewed 1000
  • Number of orders being renewed 804
  • 4 of these orders have 50 POLs each - 800 orders with 1 POL each


t3.medium3

3. extreme conditions

10 users
  • Number of order lines being renewed 5000
  • Number of orders being renewed 4804
  • 4 of these orders have 50 POLs each - 4800 orders with 1 POL each
t3.medium3
4 extreme conditions10 users
  • Number of order lines being renewed 5000
  • Number of orders being renewed 4804
  • 4 of these orders have 50 POLs each - 4800 orders with 1 POL each
t3.medium3

Results


Test #VUsersDurationresponse time (get) ms.

response time (put) ms

Error rate
avg75%95%avg75%95%
127 min 10 s249 2572871781841972.6% (52 calls)
227 min2472542881781841973.3% (67 calls)
3104 min 30 s2983194042362533391.14% (57 calls)
4104 min 10 s3083324172422634651.56% (78 calls)


Memory Utilization



Nolana Avg%

mod-organizations

25%

mod-organizations-storage

24%

mod-finance

29%

mod-finance-storage

28%

nginx-edge

2%

mod-ebsconet

37%

nginx-okapi

3%

mod-orders

44%

mod-orders-storage

34%

okapi

36%

mod-notes

43%

mod-configuration 

25%

edge-orders

20%

CPU Utilization 

Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage



RDS CPU Utilization 

Appendix

Infrastructure

PTF -environment ncp3-pvt

  • m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3 
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameter

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-organizations

1.6.0221281024896128700

mod-organizations-storage

4.4.022128

1024

896

128700

mod-finance

4.6.2221281024896128700

mod-finance-storage

8.3.1221281024896128700

nginx-edge

nginx-edge:2022.03.02121281024896N/AN/A

mod-ebsconet

1.4.0221281024896256700

nginx-okapi

nginx-okapi:2022.03.02121281024896N/AN/A

mod-orders

12.5.422102420481440256896
mod-orders-storage13.4.0221281024896128700
okapi4.14.713102416841440922922

mod-mod-notes

4.0.0221281024896128322
mod-configuration5.9.0321281024896128768

edge-orders

2.7.0221281024896128700