Check-in-check-out Test Report Sunflower [non-ECS]
- Roman_Fedynyshyn
Overview
Regression testing of Circulation Check-in Check-out (CICO) workflow in Sunflower release in scope of PERF-1109: [Sunflower] [non-ECS] CI/COClosed ;
Note: Starting from Sunflower release all environments are Eureka;
The purpose of CICO testing:
Define system behaviour (ECS resource usage, DB resource usage, queries) with different load;
Measure response times of Sunflower release and compare them with Ramsons (Eureka and Okapi) to define trends;
Find anomalies and performance issues if any. If possible recommend improvements;
Summary
Most Check-In Check-Out tests finished successfully with few failed requests per test due to data issues.
Tests for 8,20,30 and 75 users were performed on row, no issues defined “in between tests“. No signs of memory leaks on related modules.
Longevity test started to fail after 8 hours of test due to data issues so only 8 hours taken for analysis. No memory leaks or anomalies found.
Response times are 10-15% slower compared to Ramsons Eureka for Check-Out and the same or 10% faster for Check-in.
Response times are 20-30% slower compared to Ramsons Okapi .
Longevity test response times are the same if not better in some cases compared with Eureka and Okapi Ramsons.
Test Runs
Test # | vUsers | Ramp-up, sec | Duration, sec |
|---|---|---|---|
1 | 8 | 80 | 2700 |
2 | 20 | 200 | 2700 |
3 | 30 | 300 | 2700 |
4 | 75 | 750 | 2700 |
5 | 30 | 300 | 36000 |
Results
Test results
Label | Response times (ms) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||
#Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | #Samples | 95th pct | Avg. | |
Check-In Controller | 2009 | 601 | 450 | 4834 | 636 | 456 | 7040 | 744 | 464 | 16589 | 1395 | 782 |
Check-Out Controller | 2539 | 1320 | 995 | 6420 | 1465 | 1000 | 9590 | 1650 | 1040 | 21707 | 2711 | 1629 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 2010 | 302 | 213 | 4834 | 321 | 211 | 7040 | 401 | 219 | 16596 | 642 | 381 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 2539 | 499 | 367 | 6422 | 560 | 365 | 9590 | 694 | 388 | 21712 | 1317 | 650 |
GET_circulation/loans (Submit_barcode_checkout) | 2539 | 245 | 179 | 6420 | 295 | 179 | 9590 | 326 | 188 | 21709 | 538 | 305 |
Comparisons vs Ramsons Eureka RETEST
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | S | Ramsons |
|
| S | Ramsons |
|
| S | Ramsons |
|
| S | Ramsons |
|
|
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta,% | ||||
Check-Out Controller | 995 | 924 | +71 | +7.68% | 1000 | 862 | +138 | +16.01% | 1040 | 946 | +94 | +9.94% | 1629 | 1472 | +157 | +10,6% |
Check-In Controller | 450 | 496 | -46 | -9.27% | 456 | 467 | -11 | -2.35% | 464 | 495 | -31 | -6.26% | 782 | 933 | -151 | -16% |
Comparisons vs Ramsons Okapi
| Response Times, milliseconds | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
8 vUsers (test #1) | 20 vUsers (test #2) | 30 vUsers (test #3) | 75 vUsers (test #4) | |||||||||||||
Requests | S | Ramsons |
|
| S | Ramsons |
|
| S | Ramsons |
|
| S | Ramsons |
|
|
Label | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | Average | Delta,ms | Delta% | ||||
Check-Out Controller | 995 | 811 | +184 | +22% | 1000 | 835.63 | +164 | +19% | 1040 | 822.01 | +217 | +26% | 1629 | 1100.28 | +528 | +48% |
Check-In Controller | 450 | 470 | -20 | -4% | 456 | 482.65 | -26 | -5% | 464 | 455.46 | +8 | +1,8% | 782 | 606.04 | +175 | +29% |
Comparison of longevity test
30 vUsers Longevity (test #5) |
|
| ||||
|---|---|---|---|---|---|---|
Response Times, milliseconds avg. | Eureka comparison | Okapi comparison | ||||
Ramsons Eureka | Ramsons Okapi | Sunflower | Delta,ms | Delta,% | Delta,ms | Delta,% |
971 | 1130 | 1080 | 109 | +10.09% | -50 | -4,6% |
442 | 519 | 414 | -28 | -6,7% | -105 | -25,3% |
Service CPU utilization
most used modules:
module | max CPU |
|---|---|
mod-users-keycloak | 70% |
mod-remote-storage | 70% |
mod-roles-keycloak | 65% |
mod-users-keycloak and mod-roles-keycloak are highly loaded, maybe it makes sense to add more resources to this modules and retest.
Service memory usage
Memory usage showed stable trends with no spikes and growths.
Database metrics
Longevity test
Service CPU utilization
Service memory usage
DB CPU utilization
Appendix
Infrastructure
PTF -environment recp1 |
|---|
|