| Table of Contents |
|---|
Overview
This is initial report for EBSCONET workflow testing. The purpose orders renewal workflow testing
. The purpose of this document is to highlight KPI KPIs of EBSCONET workflow, find possible bottlenecks/ issues. Define baseline, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.Jira Legacy server FOLIO Issue Tracker serverId 6ccf3fe4-3301-368a-983e-20c466b11a49 key PERF-285
Summary
During testing we've find several issues:
...
- mod-organizations
- mod-organizations-storage
- mod-finance
- mod-finance-storage
- nginx-edge
- mod-ebsconet
- nginx-okapi
- mod-orders
- mod-orders-storage
- okapi
- mod-mod-notes
- mod-configuration
- edge-orders
Recommendations & Jiras
Ticket to handle NullPointerException Which is reproducible even with 2 users.Jira Legacy server FOLIO Issue Tracker serverId 6ccf3fe4-3301-368a-983e-20c466b11a49 key MODEBSNET-56
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Test Runs
Test # | VUsers | Data set | Load generator size | Load generator Memory(GiB) |
| 2 users |
| t3.medium | 3 |
| 2. normal conditions | 2 users |
| t3.medium | 3 |
3. extreme conditions | 10 users |
| t3.medium | 3 |
| 4 extreme conditions | 10 users |
| t3.medium | 3 |
Results
| Test # | VUsers | Duration | Error rate |
|---|---|---|---|
| 1 | 2 | 7 min 10 s | 2.6% (52 calls) |
| 2 | 2 | 7 min | 3.3% (67 calls) |
| 3 | 10 | 4 min 30 s | 1.14% (57 calls) |
| 4 | 10 | 4 min 10 s | 1.56% (78 calls) |
Memory Utilization
Nolana Avg% | |
|---|---|
mod-organizations | 25% |
mod-organizations-storage | 24% |
mod-finance | 29% |
mod-finance-storage | 28% |
nginx-edge | 2% |
mod-ebsconet | 37% |
nginx-okapi | 3% |
mod-orders | 44% |
mod-orders-storage | 34% |
okapi | 36% |
mod-notes | 43% |
mod-configuration | 25% |
edge-orders | 20% |
CPU Utilization
Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage
RDS CPU Utilization
Appendix
Infrastructure
PTF -environment ncp3-pvt
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-organizations | 1.6.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-organizations-storage | 4.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance | 4.6.2 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-finance-storage | 8.3.1 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
nginx-edge | nginx-edge:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-ebsconet | 1.4.0 | 2 | 2 | 128 | 1024 | 896 | 256 | 700 |
nginx-okapi | nginx-okapi:2022.03.02 | 1 | 2 | 128 | 1024 | 896 | N/A | N/A |
mod-orders | 12.5.4 | 2 | 2 | 1024 | 2048 | 1440 | 256 | 896 |
| mod-orders-storage | 13.4.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
| okapi | 4.14.7 | 1 | 3 | 1024 | 1684 | 1440 | 922 | 922 |
mod-mod-notes | 4.0.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 322 |
| mod-configuration | 5.9.0 | 3 | 2 | 128 | 1024 | 896 | 128 | 768 |
edge-orders | 2.7.0 | 2 | 2 | 128 | 1024 | 896 | 128 | 700 |
Methodology/Approach
According to EBSCONET order renewal integration testing we have designed test containing two calls [get] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY}
...
{
"currency": "USD",
"fundCode": "NEW2023",
"poLineNumber": "${polineNumber}",
"quantity": 1,
"unitPrice": 1.0,
"vendor": "ZHONEWAX$%",
"vendorAccountNumber": "libraryorders@library.tam",
"vendorReferenceNumbers": [],
"workflowStatus": "Open"
}
Test data creation
In order to create test data (orders with PO lines) we used SQL script. This script will create orders for particular organization ()
...