| Table of Contents |
|---|
Overview
This is initial report for EBSCONET workflow testing. The purpose of this orders renewal workflow testing
. The purpose of this document is to highlight KPI KPIs of EBSCONET workflow, find possible bottlenecks/ issues. Define baselineJira Legacy server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-285
Summary
During testing we've find several issues:
- NullPointerException
- Internal server error
No memory leaks were found.
used modules:
...
, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.
Summary
During testing we've find several issues:
- NullPointerException
- Internal server error
No memory leaks were found.
High level results:
| Test # | VUsers | Duration | response time (get) ms. | response time (put) ms | Error rate | ||||
| avg | 75% | 95% | avg | 75% | 95% | ||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 7 min 10 s | 249 | 257 | 287 | 178 | 184 | 197 | 2.6% (52 calls) |
| 2 | 2 | 7 min | 247 | 254 | 288 | 178 | 184 | 197 | 3.3% (67 calls) |
| 3 | 10 | 4 min 30 s | 298 | 319 | 404 | 236 | 253 | 339 | 1.14% (57 calls) |
| 4 | 10 | 4 min 10 s | 308 | 332 | 417 | 242 | 263 | 465 | 1.56% (78 calls) |
Recommendations & Jiras
Jira Legacy server
...
System JIRA serverId
...
01505d01-
...
b853-
...
3c2e-
...
90f1-
...
Ticket to handle NullPointerException Which is reproducible even with 2 users.ee9b165564fc key MODEBSNET-56 Jira Legacy server
...
System JIRA serverId
...
01505d01-
...
b853-
...
3c2e-
...
90f1-
...
Ticket to handle internal server erroree9b165564fc key MODEBSNET-57
...
- .
- We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)
Test Runs
Test # | VUsers | Data set | Load generator size | Load generator Memory(GiB) |
| 2 users |
| t3.medium | 3 |
| 2. normal conditions | 2 users |
| t3.medium | 3 |
3. extreme conditions | 10 users |
| t3.medium | 3 |
| 4 extreme conditions | 10 users |
| t3.medium | 3 |
Results
| Test # | VUsers | Duration | response time (get) ms. | response time (put) ms | Error rate | ||||
| avg | 75% | 95% | avg | 75% | 95% | ||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 7 min 10 s | 249 | 257 | 287 | 178 | 184 | 197 | 2.6% (52 calls) |
| 2 | 2 | 7 min | 247 | 254 | 288 | 178 | 184 | 197 | 3.3% (67 calls) |
| 3 | 10 | 4 min 30 s | 298 | 319 | 404 | 236 | 253 | 339 | 1.14% (57 calls) |
| 4 | 10 | 4 min 10 s | 308 | 332 | 417 | 242 | 263 | 465 | 1.56% (78 calls) |
Memory Utilization
Nolana Avg% | |
|---|---|
mod-organizations | 25% |
mod-organizations-storage | 24% |
mod-finance | 29% |
mod-finance-storage | 28% |
nginx-edge | 2% |
mod-ebsconet | 37% |
nginx-okapi | 3% |
mod-orders | 44% |
mod-orders-storage | 34% |
okapi | 36% |
mod-notes | 43% |
mod-configuration | 25% |
edge-orders | 20% |
CPU Utilization
Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage
RDS CPU Utilization
Appendix
Infrastructure
PTF -environment ncp3-pvt
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-organizations | 1.6.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-organizations-storage | 4.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance | 4.6.2 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance-storage | 8.3.1 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
nginx-edge | nginx-edge:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-ebsconet | 1.4.0 | 2 | 2 | 128 | 1024 | 896 | 256 | 700 |
nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-orders | 12.5.4 | 2 | 2 | 1024 | 2048 | 1440 | 256 | 896 |
| mod-orders-storage | 13.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 922 | 922 |
mod-mod-notes | 4.0.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 322 |
| mod-configuration | 5.9.0 | 3 | 2 | 128 | 1024 | 896 | 128 | 768 |
edge-orders | 2.7.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
Methodology/Approach
According to EBSCONET order renewal integration testing we have designed test containing two calls [get] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY}
...
{
"currency": "USD",
"fundCode": "NEW2023",
"poLineNumber": "${polineNumber}",
"quantity": 1,
"unitPrice": 1.0,
"vendor": "ZHONEWAX$%",
"vendorAccountNumber": "libraryorders@library.tam",
"vendorReferenceNumbers": [],
"workflowStatus": "Open"
}
Test data creation
In order to create test data (orders with PO lines) we used SQL script. This script will create orders for particular organization ()
...