EBSCONet Orders Renewal (Nolana)
Overview
This is initial report for EBSCONET orders renewal workflow testing - PERF-285Getting issue details... STATUS . The purpose of this document is to highlight KPIs of EBSCONET workflow, find possible bottlenecks/ issues, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.
Summary
During testing we've find several issues:
- NullPointerException
- Internal server error
No memory leaks were found.
High level results:
Test # | VUsers | Duration | response time (get) ms. | response time (put) ms | Error rate | ||||
avg | 75% | 95% | avg | 75% | 95% | ||||
---|---|---|---|---|---|---|---|---|---|
1 | 2 | 7 min 10 s | 249 | 257 | 287 | 178 | 184 | 197 | 2.6% (52 calls) |
2 | 2 | 7 min | 247 | 254 | 288 | 178 | 184 | 197 | 3.3% (67 calls) |
3 | 10 | 4 min 30 s | 298 | 319 | 404 | 236 | 253 | 339 | 1.14% (57 calls) |
4 | 10 | 4 min 10 s | 308 | 332 | 417 | 242 | 263 | 465 | 1.56% (78 calls) |
Recommendations & Jiras
- - MODEBSNET-56Getting issue details... STATUS Ticket to handle NullPointerException Which is reproducible even with 2 users.
- - MODEBSNET-57Getting issue details... STATUS Ticket to handle internal server error .
- We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)
Test Runs
Test # | VUsers | Data set | Load generator size | Load generator Memory(GiB) |
| 2 users |
| t3.medium | 3 |
2. normal conditions | 2 users |
| t3.medium | 3 |
3. extreme conditions | 10 users |
| t3.medium | 3 |
4 extreme conditions | 10 users |
| t3.medium | 3 |
Results
Test # | VUsers | Duration | response time (get) ms. | response time (put) ms | Error rate | ||||
avg | 75% | 95% | avg | 75% | 95% | ||||
---|---|---|---|---|---|---|---|---|---|
1 | 2 | 7 min 10 s | 249 | 257 | 287 | 178 | 184 | 197 | 2.6% (52 calls) |
2 | 2 | 7 min | 247 | 254 | 288 | 178 | 184 | 197 | 3.3% (67 calls) |
3 | 10 | 4 min 30 s | 298 | 319 | 404 | 236 | 253 | 339 | 1.14% (57 calls) |
4 | 10 | 4 min 10 s | 308 | 332 | 417 | 242 | 263 | 465 | 1.56% (78 calls) |
Memory Utilization
Nolana Avg% | |
---|---|
mod-organizations | 25% |
mod-organizations-storage | 24% |
mod-finance | 29% |
mod-finance-storage | 28% |
nginx-edge | 2% |
mod-ebsconet | 37% |
nginx-okapi | 3% |
mod-orders | 44% |
mod-orders-storage | 34% |
okapi | 36% |
mod-notes | 43% |
mod-configuration | 25% |
edge-orders | 20% |
CPU Utilization
Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage
RDS CPU Utilization
Appendix
Infrastructure
PTF -environment ncp3-pvt
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameter
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
---|---|---|---|---|---|---|---|---|
mod-organizations | 1.6.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-organizations-storage | 4.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance | 4.6.2 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance-storage | 8.3.1 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
nginx-edge | nginx-edge:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-ebsconet | 1.4.0 | 2 | 2 | 128 | 1024 | 896 | 256 | 700 |
nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-orders | 12.5.4 | 2 | 2 | 1024 | 2048 | 1440 | 256 | 896 |
mod-orders-storage | 13.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 922 | 922 |
mod-mod-notes | 4.0.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 322 |
mod-configuration | 5.9.0 | 3 | 2 | 128 | 1024 | 896 | 128 | 768 |
edge-orders | 2.7.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |