[Orchid] Check-IN/Check-OUT Capacity test
Overview
The target of the test is to investigate the capacity of the Check-In/Check-Out activities.
- PERF-459Getting issue details... STATUS
Summary
- Test results (Test #1 & Test #2) show approx. the same saturation point - ~134 vUsers
- The start point of the Performance degradation zone (Test #1 & Test#2):
- check-in: ~110 vUsers & ~98 vUsers
- check-out: ~75 vUsers & ~77 vUsers
Recommendations
- Up to ~75 vUsers could be used for benchmark testing if it's necessary and based on business non-functional requirements
- Run fixed load tests for the confidence of the Check-In/Check-Out activities under the 'comfort' load
Test Runs
Test # | Test Conditions | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) |
1. | CI/CO >200 int. 1 min | 125 min | t3.medium | 1 |
2. | CI/CO >200 int. 1 min | 125 min | t3.medium | 1 |
Results
Response Times
Grafana:
~134 vUsers peak - Capacity point (~683 op), throughput growth has been decreased and response issues appeared (5** status code of 'okapi' due to unavailability of related modules) (see Grafana snapshot Test #1 - Throughput)
Based on Investigation of variability of test results:
- AVG/Median check-in: [400;500] ms
- AVG/Median check-out: [700;800] ms
~110 vUsers check-in - potential start point of performance degradation zone, the response time has grown up, a significant amount of requests has a response time of more than 500 ms (see Grafana snapshot Test #1 - Response time heatmap (option 'Check-in'))
~75 vUsers check-out - potential start point of performance degradation zone, the response time has grown up, a significant amount of requests has a response time of more than 800 ms (see Grafana snapshot Test #1 - Response time heatmap (option 'Check-out'))
~134 vUsers peak Capacity point (~680 op), throughput growth has been decreased and response issues appeared (5** status code of 'okapi' due to unavailability of related modules) (see Grafana snapshot Test #2 - Throughput)
Based on Investigation of variability of test results:
- AVG/Median check-in: [400;500] ms
- AVG/Median check-out: [700;800] ms
~98 vUsers check-in - potential start point of performance degradation zone, the response time has grown up, a significant amount of requests has a response time of more than 500 ms (see Grafana snapshot Test #2 - Response time heatmap (option 'Check-in'))
~62 vUsers check-out - potential start point of performance degradation zone, the response time has grown up, a significant amount of requests has a response time of more than 800 ms (see Grafana snapshot Test #2 - Response time heatmap (option 'Check-out'))
Memory Utilization
Test №1
mod-circulation was a little grown up 58.4% > 67.5%
Test №2
CPU Utilization
Test №1
nginx-okapi: 0.1% >1'012%
mod-users-b: 0.5% > 254.6%
Test №2
nginx-okapi: 0.1% >853.7%
mod-users-b: 0.6% > 308%
RDS CPU Utilization
Test №1
DB Writer: up to 81.6%
Test №2
DB Writer: up to 79.12%
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Module | SoftLimit | XMX | Revision | Version | desiredCount | CPUUnits | RWSplitEnabled | HardLimit | Metaspace | MaxMetaspaceSize |
---|---|---|---|---|---|---|---|---|---|---|
mod-inventory-storage-b | 1952 | 1440 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.1.0-SNAPSHOT.644 | 2 | 1024 | False | 2208 | 384 | 512 |
mod-inventory-b | 2592 | 1814 | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.0-SNAPSHOT.392 | 2 | 1024 | False | 2880 | 384 | 512 |
okapi-b | 1440 | 922 | 1 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.0-SNAPSHOT.1352 | 3 | 1024 | False | 1684 | 384 | 512 |
mod-source-record-storage-b | 1440 | 908 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.7.0-SNAPSHOT.170 | 2 | 1024 | False | 1536 | 384 | 512 |
mod-source-record-manager-b | 3688 | 2048 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.0-SNAPSHOT.197 | 2 | 1024 | False | 4096 | 384 | 512 |
mod-data-import-cs-b | 896 | 768 | 1 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import-converter-storage:1.16.0-SNAPSHOT.132 | 0 | 128 | False | 1024 | 88 | 128 |
mod-data-import-b | 1844 | 1292 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:2.7.0-SNAPSHOT.101 | 1 | 256 | False | 2048 | 384 | 512 |
mod-quick-marc-b | 2176 | 1664 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.0-SNAPSHOT.79 | 1 | 128 | False | 2288 | 384 | 512 |
mod-feesfines-b | 896 | 768 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:18.3.0-SNAPSHOT.141 | 2 | 128 | False | 1024 | 88 | 128 |
mod-patron-blocks-b | 896 | 768 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.9.0-SNAPSHOT.90 | 2 | 1024 | False | 1024 | 88 | 128 |
mod-pubsub-b | 1440 | 922 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.10.0-SNAPSHOT.124 | 2 | 1024 | False | 1536 | 384 | 512 |
mod-authtoken-b | 1152 | 922 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.0-SNAPSHOT.238 | 2 | 512 | False | 1440 | 88 | 128 |
mod-circulation-storage-b | 1440 | 896 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.1.0-SNAPSHOT.305 | 2 | 1024 | False | 1536 | 384 | 512 |
mod-circulation-b | 896 | 768 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:23.5.0-SNAPSHOT.556 | 2 | 1024 | False | 1024 | 88 | 128 |
mod-configuration-b | 896 | 768 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.2-SNAPSHOT.291 | 2 | 128 | False | 1024 | 88 | 128 |
mod-users-bl-b | 1152 | 922 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.6.0-SNAPSHOT.230 | 2 | 512 | False | 1440 | 88 | 128 |
mod-remote-storage-b | 5000 | 3960 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.0-SNAPSHOT.83 | 1 | 2024 | False | 5200 | 1024 | 1024 |
mod-users-b | 896 | 768 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.2.0-SNAPSHOT.584 | 2 | 128 | False | 1024 | 88 | 128 |
Methodology/Approach
- Executed check ins with up to 200 concurrent vUsers and the interval between vUsers is 1 min. (On carrier-io use the artifact "circulation_checkInCheckOut_orchid.zip")
- Scripts used:
- DB Refresh - checkin-checkout-db-restore.sql
- DB Update - circ-data-load_item-level-requests.sh
Additional Screenshots of graphs or charts
Test #1
Test #2