...
- RDS CPU utilization average
- 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 63% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test that can be explained by absent dcb-system-user in mod-dcb module.
- CPU (User) usage by broker
- Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
Common notes
- Longevity test revealed CPU Utilization growing trend for mod-dcb-b module.
...
| Expand |
|---|
| title | Summary for tests with absent dcb-system-user |
|---|
|
45 minute tests - Response times in tests with 8, 20, 30 virtual users didn't differ a lot but the lowest response times were with 8 virtual users. Average Check-In - 470 ms, average Check-Out - 838 ms
- The highest response times results were with 75 virtual users. Average Check-In - 720 ms, Check-Out - 1301 ms
- Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU. For this test average Check-In - 587 ms, Check-Out - 1055 ms. which is better than Poppy release - 5%.
Longevity test - Average Check-In - 454 ms, Check-Out - 1060 ms
- No memory leaks during longevity test
Comparison with Poppy Results: - CI/CO response times increased by 4-7% for 20 and 30 vUsers tests.
- For 75 vUsers test, the increase was 15%.
- For 75 vUsers test CO response time was 1116 ms in Poppy and 1301 ms in Quesnelia
CPU utilization - 45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-dcb-b - 407% with 128 CPU, mod-authtoken-b - 120%, nginx-okapi - 102%.
- Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU.
Memory consumption - 45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.
RDS CPU utilization - 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test.
CPU (User) usage by broker - Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
Common notes - Longevity test revealed CPU Utilization growing trend for mod-dcb-b module.
Recommendations - Investigate mod-dcb-b module behaviour. After 75 vUsers test modules require some time to stabilize CPU consumption.
|
...
Test # | vUsers | Ramp-up, sec | Duration, sec |
1 | 8 | 80 | 2700 |
| 2 | 20 | 200 | 2700 |
| 3 | 30 | 300 | 2700 |
| 4 | 75 | 750 | 2700 |
| 5 | 30 | 300 | 86400 |
...
Results
Errors:
- Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. 422/Unprocessable Entity. Happen expectedly if instance was checked out already. Error rate - 0.06% which is acceptable.
Response time
The table contains results of Check-in, Check-out tests in Quesnelia release.
...
This table has comparison between average values of response times of Quesnelia and Poppy releases
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
|---|
| Requests | Response Times | Response Times | Response Times | Response Times |
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
|
| Label | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% |
| Check-Out Controller | 832 | 741 | -91 | -10.94% | 805 | 729 | -76 | -9.44% | 807 | 767 | -40 | -4.96% | 1116 | 1039 | -77 | -6.90% |
| Check-In Controller | 479 | 408 | -71 | -14.82% | 450 | 404 | -46 | -10.22% | 449 | 427 | -22 | -4.90% | 622 | 580 | -42 | -6.75% |
...
| Expand |
|---|
| title | Results for tests with absent dcb-system-user |
|---|
|
Response time The table contains results of Check-in, Check-out tests in Quesnelia release. 45 minute tests
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
|---|
Requests | Samples, Response Times | Samples, Response Times | Samples, Response Times | Samples, Response Times | Label | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average | | Check-Out Controller | 1785 | 1007.7 | 837.94 | 4576 | 993 | 842.65 | 6496 | 1020 | 870.2 | 14379 | 1876 | 1301.92 | | Check-In Controller | 1437 | 554 | 470.2 | 3285 | 550.7 | 467.71 | 5045 | 562 | 480.62 | 10607 | 1047 | 720.2 | | POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 1785 | 378 | 290.32 | 4577 | 334 | 282.21 | 6499 | 342 | 288.07 | 14383 | 665 | 426.66 | | POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 1439 | 266 | 207.48 | 3293 | 237 | 195.47 | 5056 | 235 | 200.12 | 10629 | 464 | 295.4 |
Longevity test | 30 vUsers Longevity |
|
|
|
|---|
| Requests | Samples, Response Times |
|
| | Label | #Samples | 95th pct | Average | | Check-Out Controller | 135591 | 1247 | 1065 | | Check-In Controller | 101990 | 525 | 454 | | POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 135593 | 410 | 340 | | POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 101981 | 228 | 193 |
Comparisons This table has comparison between average values of response times of Quesnelia and Poppy releases
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
|---|
| Requests | Response Times |
|
| Response Times |
|
| Response Times |
|
| Response Times |
|
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| | Label | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | | Check-Out Controller | 832 | 837.94 | 5.94 | 0.71% | 805 | 842.65 | 37.65 | 4.68% | 807 | 870.2 | 63.2 | 7.83% | 1116 | 1301.92 | 185.92 | 16.66% | | Check-In Controller | 479 | 470.2 | -8.8 | -1.84% | 450 | 467.71 | 17.71 | 3.94% | 449 | 480.62 | 31.62 | 7.04% | 622 | 720.2 | 98.2 | 15.79% |
Comparison table for tests after mod-dcb-b CPU 128 as default raised to 256 (Task definition: revision: qcp1-mod-dcb:3). Additional test without mod-dcb-b (# of tasks = 0)
| 75 vUsers mod-dcb-b CPU 128 | 75 vUsers mod-dcb-b CPU 256 | 75 vUsers mod-dcb-b - Tasks = 0 |
|---|
| Requests | Response Times |
|
| Response Times |
|
| Response Times |
|
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| | Label | Average | Average | Delta,ms | Difference,% | Average | Average | Delta,ms | Difference,% | Average | Average | Delta,ms | Difference,% | | Check-Out Controller | 1116 | 1301.92 | 185.92 | 16.66% | 1116 | 1055 | -61 | -5.47% | 1116 | 1117 | 1 | 0.09% | | Check-In Controller | 622 | 720.2 | 98.2 | 15.79% | 622 | 587 | -35 | -5.63% | 622 | 629 | 7 | 1.13% |
Comparison of Longevity test | 30 vUsers Longevity |
|---|
| Response Times |
|
| | Poppy | Quesnelia |
|
| | Average | Average | Delta,ms | Difference,% | | 1025 | 1065 | 40 | 3.90% | | 471 | 454 | -17 | -3.61% |
API requests where response times >= 100 ms
|
...
45 minute tests
Longevity test
RDS Database Connections
For both 45 minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.
45 minute tests
Longevity test
Image Modified
CPU (User) usage by broker
Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
45 minute tests
Longevity test
Image Modified
Database load
...
| Expand |
|---|
| title | Resources Utilization |
|---|
|
CPU Utilization 45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-dcb-b - 407% with 128 CPU, mod-authtoken-b - 120%, nginx-okapi - 102%.
Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU. 45 minute tests | Expand |
|---|
| title | CPU utilization by modules |
|---|
|
| Service Name | CPU 8 vUsers | CPU 20 vUsers | CPU 30 vUsers | CPU 75 vUsers, 128 | CPU 30 vUsers Longevity | CPU 75 vUsers, 256 |
|---|
| mod-users-b | 29.31 | 65.11 | 89.61 | 206.63 | 96.57 | 199.92 | | mod-dcb-b | 21.53 | 46.25 | 69.61 | 407.79 | 155.82 | 201.69 | | okapi-b | 16.52 | 38.85 | 55.66 | 90.6 | 56.77 | 93 | | mod-inventory-b | 14.28 | 18.17 | 17.97 | 15.14 | 15.58 | 15.77 | | mod-authtoken-b | 12.08 | 20.81 | 32.34 | 120.62 | 30.98 | 115.95 | | mod-configuration-b | 10.73 | 21.98 | 33.95 | 71.33 | 30.02 | 70.5 | | mod-pubsub-b | 9.17 | 11.67 | 13.61 | 18.09 | 12.39 | 26.5 | | mod-feesfines-b | 8.26 | 17.1 | 23.88 | 54.79 | 28.87 | 54.37 | | mod-quick-marc-b | 7.2 | 7.63 | 7.57 | 8.19 | 7.93 | 7.88 | | pub-okapi | 3.91 | 9.45 | 13.99 | 33.44 | 12.47 | 27.68 | | mod-inventory-storage-b | 3.44 | 7.65 | 11.99 | 13.24 | 10.67 | 12.16 | | nginx-okapi | 2.56 | 6.35 | 9.35 | 102.62 | 7.69 | 69.58 | | mod-patron-b | 2.31 | 2.52 | 2.52 | 2.3 | 2.28 | 2.44 | | mod-oa-b | 2.28 | 2.17 | 2.67 | 2.75 | 3.17 | 3.52 | | mod-di-converter-storage-b | 2.2 | 2.19 | 2.29 | 2.39 | 2.12 | 1.97 | | mod-source-record-manager-b | 1.93 | 1.9 | 1.87 | 1.88 | 1.81 | 1.95 | | mod-data-import-b | 1.7 | 1.02 | 1.02 | 0.96 | 1.57 | 1.59 | | mod-source-record-storage-b | 1.68 | 1.72 | 1.73 | 1.77 | 1.76 | 1.83 | | mod-circulation-storage-b | 1.46 | 2.59 | 3.64 | 7.43 | 3.53 | 7.06 | | mod-circulation-b | 1.46 | 3.54 | 4.58 | 10.25 | 5.62 | 11.21 | | mod-search-b | 1.21 | 2.66 | 2.34 | 3.28 | 2.94 | 3.94 | | edge-patron-b | 0.92 | 0.96 | 1 | 1.06 | 0.95 | 0.97 | | mod-patron-blocks-b | 0.75 | 1.38 | 1.8 | 2.74 | 2.38 | 2.97 | | mod-users-bl-b | 0.51 | 0.54 | 0.53 | 0.5 | 0.58 | 0.66 | | mod-entities-links-b | 0.35 | 0.36 | 0.44 | 0.42 | 0.4 | 0.39 | | pub-edge | 0.14 | 0.06 | 0.1 | 0.08 | 0.07 | 0.11 |
|
45 minute test with 75 vUsers (revision with 256 CPU for mod-dcb-b) 
Longevity test 
Memory Consumption 45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. | Expand |
|---|
| title | Memory usage by modules |
|---|
|
| Service Name | Memory 75 vUsers | Memory 30 vUsers Longevity |
|---|
| mod-oa-b | 81 | 79.24 | | mod-dcb-b | 68.63 | 69.16 | | mod-users-b | 63.65 | 62.98 | | mod-search-b | 55.34 | 40.83 | | mod-pubsub-b | 54.79 | 61.98 | | okapi-b | 48.94 | 60.38 | | mod-inventory-b | 47.49 | 50.79 | | mod-source-record-manager-b | 45.06 | 45.76 | | mod-quick-marc-b | 43.51 | 29.92 | | mod-di-converter-storage-b | 35.02 | 29.58 | | mod-feesfines-b | 34.18 | 34.12 | | mod-source-record-storage-b | 33.76 | 31.37 | | mod-inventory-storage-b | 33.12 | 40.06 | | mod-entities-links-b | 32.3 | 25.08 | | mod-configuration-b | 32.07 | 30.21 | | mod-data-import-b | 31.31 | 39.73 | | mod-patron-blocks-b | 31.13 | 30.92 | | mod-users-bl-b | 28.95 | 25.82 | | mod-circulation-b | 27.41 | 26.71 | | mod-patron-b | 26.62 | 25.17 | | mod-circulation-storage-b | 26.43 | 28.83 | | mod-authtoken-b | 21.84 | 22.1 | | edge-patron-b | 18.3 | 18.14 | | nginx-okapi | 4.85 | 4.8 | | pub-okapi | 4.69 | 4.69 | | pub-edge | 4.58 | 4.63 |
|
45 minute tests 
Longevity test 
RDS CPU Utilization RDS CPU utilized: 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test. 45 minute tests 
Longevity test
RDS Database Connections For both 45 minute and longevity tests RDS used max 615 connections. Without test it was 550 connections. 45 minute tests  Image Modified
Longevity test Image Modified CPU (User) usage by broker Common CPU utilization by broker during all tests was 15% with equal distribution between brokers 45 minute tests  Image Modified
Longevity testImage Modified  Image Modified
 Image Modified
 Image Modified
|
Appendix
Infrastructure
...