Check-In / Check-Out - NLA report
IN PROGRESS
Overview
In the scope of PERF-601: NLA: Test CICO IndividuallyClosed
Perform a fixed load test on the CICO functionality with the same average and high loads that are used for NLA testing to get baseline numbers.
Summary
Response time for CICO separate workflow is higher than for high load and normal load.
Average CPU usage of mod-inventory - up to 11% for all other modules did not exceed 5 %. Instance CPU Utilization did not exceed 7,5%.
Service memory utilization was stable, and no memory leaks were suspected during tests.
DB CPU usage was at the same level as without load- about 7,2%. 342 connections count without load was increased to 360 connections during CICO.
Test Runs & Results
Test # | # configuration | Test duration | comments |
|---|---|---|---|
1,2,3 | Throughput for CI/CO operations is 1 per minute (1 CI and 1 CO) or about 60 per hour | 1 hour | NO errors |
Test results from 1st test run (1st, 2nd and 3rd test run results are similar):
Test # | Workflow name | Total time it takes to complete workflow | Avg Latency for baseline test | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Avg (sec) Baseline | 95th pct (sec) Baseline | Avg With DI (sec) normal load * | 95th pct with DI (sec) normal load | Avg (sec) high load with FYR | 95th pct (sec) high load with FYR | Avg (sec) high load | 95th pct (sec) high load ** | Avg (sec) normal load | 95th pct (sec) normal load | |
1 | Checkin | 1.233 | 1.570 | 1.238 | 1.506 | 1.218 | 1.488 | 1.093 | 1.362 | 1.054 | 1.591 | 1.155 |
2 | Checkout | 2.237 | 2.787 | 2.156 | 2.829 | 2.272 | 2.780 | 1.830 | 2.187 | 1.650 | 1.948 | 2.125 |
Throughput graphs
For Test#1, Test#2 and Test#3 graphs are almost the same.
Memory Utilization
This graph represents memory usage of test runs and shows that no memory leak is suspected for all of the modules.
Service CPU Utilization
Average CPU usage of mod-inventory - up to 11% for all other modules did not exceed 5 %.
Detailed CICO modules
Instance CPU Utilization
Instance CPU Utilization did not exceed 7,5%.
RDS CPU Utilization
DB CPU usage was at the same level as without load- about 7,2%.
RDS Database Connections
Test# 1 - Test# 3 - 342 connections count without load was increased to 360 connections during CICO.
Appendix
Infrastructure
Load generator
Instance Type: t3.2xlarge (Ram memory in GB available per load generator - 30 GB needed for the test with high load)
PTF -environment ncp3
9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
MSK ptf-kakfa-3
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Kafka topics partitioning: - 2 partitions for DI topics
Modules memory and CPU parameters
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
mod-inventory-storage | 26.0.0 | 1 | 2 | 1024 | 2208 | 1952 | 384 | 1440 |
mod-inventory | 20.0.4 | 1 | 2 | 1024 | 2880 | 2592 | 512 | 1814 |
mod-tags | 2.0.1 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-gobi | 2.6.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 700 |
mod-remote-storage | 2.0.2 | 1 | 2 | 1024 | 4920 | 4472 | 512 | 3960 |
mod-invoice-storage | 5.6.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 700 |
edge-sip2 | 3.0.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-users-bl | 7.5.0 | 1 | 2 | 512 | 1440 | 1152 | 128 | 922 |
edge-rtac | 2.6.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-feesfines | 18.2.1 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-rtac | 3.5.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-erm-usage-harvester | 4.3.0 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-search | 2.0.1 | 1 | 2 | 400 | 2592 | 2480 | 1024 | 1440 |
mod-service-interaction | 2.2.2 | 1 | 2 | 256 | 2048 | 1844 | 512 | 1290 |
edge-ncip | 1.8.1 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-authtoken | 2.13.0 | 1 | 2 | 512 | 1440 | 1152 | 128 | 922 |
mod-permissions | 6.3.1 | 2 | 2 | 512 | 1684 | 1544 | 512 | 1024 |
mod-circulation-storage | 16.0.0 | 1 | 2 | 1024 | 1536 | 1440 | 512 | 896 |
mod-ncip | 1.13.1 | 1 | 2 | 128 | 1024 | 896 | 128 | 768 |
mod-pubsub | 2.9.1 | 1 | 2 | 1024 | 1536 | 1440 | 512 | 922 |
edge-orders | 2.8.1 | 1 | 2 | 1024 | 1536 | 1440 | 512 | 922 |
mod-circulation | 23.5.4 | 1 | 2 | 1536 | 2880 | 2592 | 128 | 700 |
edge-caiasoft | 2.0.0 | 1 | 2 | 128 | 1024 | 896 | - | - |
mod-data-export | 4.7.1 | 1 |