Overview
The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.
PERF-835
-
Getting issue details...
STATUS
Summary
Check-In/Check-Out tests on Quesnelia carried out with 8, 20, 30, 75 virtual during 45 minutes and with to detect any trends in resource utilization
45 minute tests
- Response times in tests with 8, 20, 30 virtual users didn't differ a lot but the lowest response times were with 20 virtual users.
- The highest response times results were with 75 virtual users. Average Check-In - 580 ms, Check-Out - 1039 ms
Longevity test
- Average Check-In - 454 ms, Check-Out - 1060 ms (the results from the test with absent dcb-system-user)
- No memory leaks during longevity test
Comparison with Poppy Results:
Resources
- CPU utilization
- 45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-users-b - 132%, mod-authtoken-b - 98%, nginx-okapi - 84%.
- Memory consumption
- 45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.
- RDS CPU utilization average
- 8 vUsers - , 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 63% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test that can be explained by absent dcb-system-user in mod-dcb module.
- CPU (User) usage by broker
- Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
Common notes
- Longevity test revealed CPU Utilization growing trend for mod-dcb-b module.
Recommendations
- Control mod-dcb-b module related system user creation. . It may lead to a huge performance degradation.
- Add more CPU to mod-users to exclude module fluctuations
Summary for tests with absent dcb-system-user
45 minute tests
- Response times in tests with 8, 20, 30 virtual users didn't differ a lot but the lowest response times were with 8 virtual users. Average Check-In - 470 ms, average Check-Out - 838 ms
- The highest response times results were with 75 virtual users. Average Check-In - 720 ms, Check-Out - 1301 ms
- Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU. For this test average Check-In - 587 ms, Check-Out - 1055 ms. which is better than Poppy release - 5%.
Longevity test
- Average Check-In - 454 ms, Check-Out - 1060 ms
- No memory leaks during longevity test
Comparison with Poppy Results:
- CI/CO response times increased by 4-7% for 20 and 30 vUsers tests.
- For 75 vUsers test, the increase was 15%.
- For 75 vUsers test CO response time was 1116 ms in Poppy and 1301 ms in Quesnelia
CPU utilization
- 45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-dcb-b - 407% with 128 CPU, mod-authtoken-b - 120%, nginx-okapi - 102%.
- Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU.
Memory consumption
- 45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.
RDS CPU utilization
- 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test.
CPU (User) usage by broker
- Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
Common notes
- Longevity test revealed CPU Utilization growing trend for mod-dcb-b module.
Recommendations
- Investigate mod-dcb-b module behaviour. After 75 vUsers test modules require some time to stabilize CPU consumption.
Test Runs
The following table contains tests configuration information
Test # | vUsers | Ramp-up, sec | Duration, sec |
1 | 8 | 80 | 2700 |
2 | 20 | 200 | 2700 |
3 | 30 | 300 | 2700 |
4 | 75 | 750 | 2700 |
5 | 30 | 300 | 86400 |
Results
- Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. 422/Unprocessable Entity. Happen expectedly if instance was checked out already. Error rate - 0.06% which is acceptable.
Response time
The table contains results of Check-in, Check-out tests in Quesnelia release.
45 minute tests
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
---|
Requests | Response Times (ms) | Response Times (ms) | Response Times (ms) | Response Times (ms) |
Label | #Samples | Average | 95th pct | #Samples | Average | 95th pct | #Samples | Average | 95th pct | #Samples | Average | 95th pct |
Check-Out Controller | 1857 | | 904.1 | 4614 | 729.26 | 859 | 6637 | 767.83 | 935 | 14637 | 1039.21 | 1323 |
Check-In Controller | 1394 | 408.2 | 484 | 3378 | 404.24 | 475 | 5063 | 427.22 | 527 | 11136 | 580.9 | 752 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 1858 | 256.34 | 337 | 4615 | 247.44 | 297 | 6639 | 252.35 | 310 | 14642 | 343.5 | 457 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 1395 | 172.66 | 225.2 | 3382 | 170.31 | 205 | 5080 | 174.75 | 224.95 | 11159 | 242.61 | 323 |
GET_circulation/loans (Submit_barcode_checkout) | 1856 | 136.69 | 185.15 | 4613 | 133.16 | 161 | 6636 | 138.46 | 176 | 14628 | 186.43 | 259 |
Longevity test (with absent dcb-system-user)
30 vUsers Longevity |
|
|
|
---|
Requests | Samples, Response Times |
|
|
Label | #Samples | 95th pct | Average |
Check-Out Controller | 135591 | 1247 | 1065 |
Check-In Controller | 101990 | 525 | 454 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 135593 | 410 | 340 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 101981 | 228 | 193 |
Comparisons
This table has comparison between average values of response times of Quesnelia and Poppy releases
| | | | |
---|
Requests | Response Times | Response Times | Response Times | Response Times |
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
|
Label | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% |
Check-Out Controller | 832 | 741 | -91 | -10.94% | 805 | 729 | -76 | -9.44% | 807 | 767 | -40 | -4.96% | 1116 | 1039 | -77 | -6.90% |
Check-In Controller | 479 | 408 | -71 | -14.82% | 450 | 404 | -46 | -10.22% | 449 | 427 | -22 | -4.90% | 622 | 580 | -42 | -6.75% |
Comparison of Longevity test (absent dcb-system-user in Quesnelia)
30 vUsers Longevity |
---|
Response Times |
|
|
Poppy | Quesnelia |
|
|
Average | Average | Delta,ms | Difference,% |
1025 | 1065 | 40 | 3.90% |
471 | 454 | -17 | -3.61% |
API requests where response times >= 100 ms
Results for tests with absent dcb-system-user
Response time
The table contains results of Check-in, Check-out tests in Quesnelia release.
45 minute tests
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
---|
Requests | Samples, Response Times | Samples, Response Times | Samples, Response Times | Samples, Response Times |
Label | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average | #Samples | 95th pct | Average |
Check-Out Controller | 1785 | 1007.7 | 837.94 | 4576 | 993 | 842.65 | 6496 | 1020 | 870.2 | 14379 | 1876 | 1301.92 |
Check-In Controller | 1437 | 554 | 470.2 | 3285 | 550.7 | 467.71 | 5045 | 562 | 480.62 | 10607 | 1047 | 720.2 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 1785 | 378 | 290.32 | 4577 | 334 | 282.21 | 6499 | 342 | 288.07 | 14383 | 665 | 426.66 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 1439 | 266 | 207.48 | 3293 | 237 | 195.47 | 5056 | 235 | 200.12 | 10629 | 464 | 295.4 |
Longevity test
30 vUsers Longevity |
|
|
|
---|
Requests | Samples, Response Times |
|
|
Label | #Samples | 95th pct | Average |
Check-Out Controller | 135591 | 1247 | 1065 |
Check-In Controller | 101990 | 525 | 454 |
POST_circulation/check-out-by-barcode (Submit_barcode_checkout) | 135593 | 410 | 340 |
POST_circulation/check-in-by-barcode (Submit_barcode_checkin) | 101981 | 228 | 193 |
Comparisons
This table has comparison between average values of response times of Quesnelia and Poppy releases
| 8 vUsers | 20 vUsers | 30 vUsers | 75 vUsers |
---|
Requests | Response Times |
|
| Response Times |
|
| Response Times |
|
| Response Times |
|
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
|
Label | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% | Average | Delta,ms | Difference,% |
Check-Out Controller | 832 | 837.94 | 5.94 | 0.71% | 805 | 842.65 | 37.65 | 4.68% | 807 | 870.2 | 63.2 | 7.83% | 1116 | 1301.92 | 185.92 | 16.66% |
Check-In Controller | 479 | 470.2 | -8.8 | -1.84% | 450 | 467.71 | 17.71 | 3.94% | 449 | 480.62 | 31.62 | 7.04% | 622 | 720.2 | 98.2 | 15.79% |
Comparison table for tests after mod-dcb-b CPU 128 as default raised to 256 (Task definition: revision: qcp1-mod-dcb:3). Additional test without mod-dcb-b (# of tasks = 0)
| 75 vUsers mod-dcb-b CPU 128 | 75 vUsers mod-dcb-b CPU 256 | 75 vUsers mod-dcb-b - Tasks = 0 |
---|
Requests | Response Times |
|
| Response Times |
|
| Response Times |
|
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
| Poppy | Quesnelia |
|
|
Label | Average | Average | Delta,ms | Difference,% | Average | Average | Delta,ms | Difference,% | Average | Average | Delta,ms | Difference,% |
Check-Out Controller | 1116 | 1301.92 | 185.92 | 16.66% | 1116 | 1055 | -61 | -5.47% | 1116 | 1117 | 1 | 0.09% |
Check-In Controller | 622 | 720.2 | 98.2 | 15.79% | 622 | 587 | -35 | -5.63% | 622 | 629 | 7 | 1.13% |
Comparison of Longevity test
30 vUsers Longevity |
---|
Response Times |
|
|
Poppy | Quesnelia |
|
|
Average | Average | Delta,ms | Difference,% |
1025 | 1065 | 40 | 3.90% |
471 | 454 | -17 | -3.61% |
API requests where response times >= 100 ms
Resources Utilization
CPU Utilization
45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-users-b - 132%, mod-authtoken-b - 98%, nginx-okapi - 84%.
45 minute tests
CPU utilization by modules
Service Name | CPU 8 vUsers | CPU 20 vUsers | CPU 30 vUsers | CPU 75 vUsers |
---|
mod-users-b | 21.08 | 38.95 | 60.75 | 132.25 |
mod-inventory-b | 14.75 | 16.84 | 18.23 | 18.04 |
okapi-b | 14.54 | 33.37 | 48.86 | 82.97 |
mod-configuration-b | 11.74 | 24.4 | 36.58 | 79.42 |
mod-authtoken-b | 9.89 | 23.43 | 31.37 | 98.15 |
mod-pubsub-b | 7.88 | 9.08 | 11.25 | 15.37 |
mod-dcb-b | 7.76 | 11.25 | 10.33 | 16.89 |
mod-feesfines-b | 7.66 | 14.71 | 21.54 | 47.69 |
mod-quick-marc-b | 6.68 | 5.96 | 6.13 | 6.41 |
mod-data-import-b | 4.07 | 4.22 | 4.17 | 4.67 |
pub-okapi | 3.94 | 9.33 | 13.32 | 32.52 |
mod-inventory-storage-b | 3.64 | 7.59 | 11.93 | 14.2 |
mod-oa-b | 2.69 | 2.41 | 2.46 | 3.58 |
nginx-okapi | 2.28 | 5.03 | 7.99 | 84.13 |
mod-di-converter-storage-b | 2.23 | 2.37 | 2.42 | 2.51 |
mod-patron-b | 2.22 | 2.38 | 2.38 | 2.95 |
mod-source-record-manager-b | 1.92 | 1.89 | 1.91 | 1.98 |
mod-source-record-storage-b | 1.86 | 1.88 | 1.91 | 1.95 |
mod-circulation-storage-b | 1.49 | 2.56 | 3.52 | 7.36 |
mod-search-b | 1.48 | 1.17 | 3.64 | 1.25 |
mod-circulation-b | 1.33 | 3.01 | 4.02 | 9.26 |
edge-patron-b | 0.97 | 1 | 1.06 | 1.11 |
mod-patron-blocks-b | 0.84 | 1.41 | 1.98 | 3.47 |
mod-users-bl-b | 0.44 | 0.45 | 0.45 | 0.5 |
mod-entities-links-b | 0.39 | 0.32 | 0.35 | 0.37 |
pub-edge | 0.08 | 0.09 | 0.09 | 0.09 |
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_11-39-45.png?version=1&modificationDate=1712047186002&cacheVersion=1&api=v2&width=1200&height=516)
Longevity test (with absent dcb-system-user)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-5-57.png?version=1&modificationDate=1711703159405&cacheVersion=1&api=v2&width=1200&height=506)
Memory Consumption
45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%
Memory usage by modules
Service Name | Memory 75 vUsers |
---|
mod-search-b | 94.2 |
mod-oa-b | 78.38 |
mod-inventory-b | 72.3 |
mod-dcb-b | 70.98 |
mod-users-b | 63.74 |
okapi-b | 48.33 |
mod-pubsub-b | 47.6 |
mod-source-record-storage-b | 45.07 |
mod-data-import-b | 40.66 |
mod-di-converter-storage-b | 39.51 |
mod-source-record-manager-b | 35.98 |
mod-inventory-storage-b | 35.84 |
mod-feesfines-b | 34.69 |
mod-quick-marc-b | 31.16 |
mod-configuration-b | 30.76 |
mod-patron-blocks-b | 30.69 |
mod-users-bl-b | 30.24 |
mod-circulation-b | 29.17 |
mod-entities-links-b | 26.99 |
mod-patron-b | 24 |
mod-authtoken-b | 21.89 |
mod-circulation-storage-b | 21.31 |
edge-patron-b | 18.19 |
nginx-okapi | 5.3 |
pub-okapi | 4.63 |
pub-edge | 4.46 |
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_11-42-24.png?version=1&modificationDate=1712047345129&cacheVersion=1&api=v2&width=1200&height=390)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-32-41.png?version=1&modificationDate=1711704762702&cacheVersion=1&api=v2&width=1200&height=409)
RDS CPU Utilization
RDS CPU utilized:
8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test.
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-8-7.png?version=1&modificationDate=1712048888616&cacheVersion=1&api=v2&width=1200&height=384)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-39-44.png?version=1&modificationDate=1711705186128&cacheVersion=1&api=v2&width=1200&height=454)
RDS Database Connections
For both 45 minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-7-37.png?version=1&modificationDate=1712048858553&cacheVersion=1&api=v2&width=1200&height=403)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-44-36.png?version=1&modificationDate=1711705478468&cacheVersion=1&api=v2&width=1200&height=419)
CPU (User) usage by broker
Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_13-25-8.png?version=1&modificationDate=1712053509238&cacheVersion=1&api=v2&width=1200&height=391)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-53-32.png?version=1&modificationDate=1711706014117&cacheVersion=1&api=v2&width=1200&height=405)
Database load
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-11-39.png?version=1&modificationDate=1712049100646&cacheVersion=1&api=v2&width=1200&height=411)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-12-30.png?version=1&modificationDate=1712049151351&cacheVersion=1&api=v2&width=1200&height=396)
75 vUsers
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-13-48.png?version=1&modificationDate=1712049229310&cacheVersion=1&api=v2&width=1200&height=398)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-4-2_12-15-10.png?version=1&modificationDate=1712049310966&cacheVersion=1&api=v2&width=1200&height=330)
Resources Utilization
CPU Utilization
45 minute and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-dcb-b - 407% with 128 CPU, mod-authtoken-b - 120%, nginx-okapi - 102%.
Additional test was carried out with changed configuration for mod-dcb-b (raised from 128 to 256). It helped to decrease CPU utilization for mod-dcb-b twice from 407% to 202% and nginx-okapi from 102% to 70% which helped to decrease response times more than 20% if to compare results with 128 CPU and 256 CPU.
45 minute tests
CPU utilization by modules
Service Name | CPU 8 vUsers | CPU 20 vUsers | CPU 30 vUsers | CPU 75 vUsers, 128 | CPU 30 vUsers Longevity | CPU 75 vUsers, 256 |
---|
mod-users-b | 29.31 | 65.11 | 89.61 | 206.63 | 96.57 | 199.92 |
mod-dcb-b | 21.53 | 46.25 | 69.61 | 407.79 | 155.82 | 201.69 |
okapi-b | 16.52 | 38.85 | 55.66 | 90.6 | 56.77 | 93 |
mod-inventory-b | 14.28 | 18.17 | 17.97 | 15.14 | 15.58 | 15.77 |
mod-authtoken-b | 12.08 | 20.81 | 32.34 | 120.62 | 30.98 | 115.95 |
mod-configuration-b | 10.73 | 21.98 | 33.95 | 71.33 | 30.02 | 70.5 |
mod-pubsub-b | 9.17 | 11.67 | 13.61 | 18.09 | 12.39 | 26.5 |
mod-feesfines-b | 8.26 | 17.1 | 23.88 | 54.79 | 28.87 | 54.37 |
mod-quick-marc-b | 7.2 | 7.63 | 7.57 | 8.19 | 7.93 | 7.88 |
pub-okapi | 3.91 | 9.45 | 13.99 | 33.44 | 12.47 | 27.68 |
mod-inventory-storage-b | 3.44 | 7.65 | 11.99 | 13.24 | 10.67 | 12.16 |
nginx-okapi | 2.56 | 6.35 | 9.35 | 102.62 | 7.69 | 69.58 |
mod-patron-b | 2.31 | 2.52 | 2.52 | 2.3 | 2.28 | 2.44 |
mod-oa-b | 2.28 | 2.17 | 2.67 | 2.75 | 3.17 | 3.52 |
mod-di-converter-storage-b | 2.2 | 2.19 | 2.29 | 2.39 | 2.12 | 1.97 |
mod-source-record-manager-b | 1.93 | 1.9 | 1.87 | 1.88 | 1.81 | 1.95 |
mod-data-import-b | 1.7 | 1.02 | 1.02 | 0.96 | 1.57 | 1.59 |
mod-source-record-storage-b | 1.68 | 1.72 | 1.73 | 1.77 | 1.76 | 1.83 |
mod-circulation-storage-b | 1.46 | 2.59 | 3.64 | 7.43 | 3.53 | 7.06 |
mod-circulation-b | 1.46 | 3.54 | 4.58 | 10.25 | 5.62 | 11.21 |
mod-search-b | 1.21 | 2.66 | 2.34 | 3.28 | 2.94 | 3.94 |
edge-patron-b | 0.92 | 0.96 | 1 | 1.06 | 0.95 | 0.97 |
mod-patron-blocks-b | 0.75 | 1.38 | 1.8 | 2.74 | 2.38 | 2.97 |
mod-users-bl-b | 0.51 | 0.54 | 0.53 | 0.5 | 0.58 | 0.66 |
mod-entities-links-b | 0.35 | 0.36 | 0.44 | 0.42 | 0.4 | 0.39 |
pub-edge | 0.14 | 0.06 | 0.1 | 0.08 | 0.07 | 0.11 |
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_10-57-40.png?version=1&modificationDate=1711702662473&cacheVersion=1&api=v2&width=1200&height=509)
45 minute test with 75 vUsers (revision with 256 CPU for mod-dcb-b)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_15-45-22.png?version=1&modificationDate=1711719924817&cacheVersion=1&api=v2&width=1200&height=518)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-5-57.png?version=1&modificationDate=1711703159405&cacheVersion=1&api=v2&width=1200&height=506)
Memory Consumption
45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.
Memory usage by modules
Service Name | Memory 75 vUsers | Memory 30 vUsers Longevity |
---|
mod-oa-b | 81 | 79.24 |
mod-dcb-b | 68.63 | 69.16 |
mod-users-b | 63.65 | 62.98 |
mod-search-b | 55.34 | 40.83 |
mod-pubsub-b | 54.79 | 61.98 |
okapi-b | 48.94 | 60.38 |
mod-inventory-b | 47.49 | 50.79 |
mod-source-record-manager-b | 45.06 | 45.76 |
mod-quick-marc-b | 43.51 | 29.92 |
mod-di-converter-storage-b | 35.02 | 29.58 |
mod-feesfines-b | 34.18 | 34.12 |
mod-source-record-storage-b | 33.76 | 31.37 |
mod-inventory-storage-b | 33.12 | 40.06 |
mod-entities-links-b | 32.3 | 25.08 |
mod-configuration-b | 32.07 | 30.21 |
mod-data-import-b | 31.31 | 39.73 |
mod-patron-blocks-b | 31.13 | 30.92 |
mod-users-bl-b | 28.95 | 25.82 |
mod-circulation-b | 27.41 | 26.71 |
mod-patron-b | 26.62 | 25.17 |
mod-circulation-storage-b | 26.43 | 28.83 |
mod-authtoken-b | 21.84 | 22.1 |
edge-patron-b | 18.3 | 18.14 |
nginx-okapi | 4.85 | 4.8 |
pub-okapi | 4.69 | 4.69 |
pub-edge | 4.58 | 4.63 |
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-30-26.png?version=1&modificationDate=1711704628609&cacheVersion=1&api=v2&width=1200&height=415)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-32-41.png?version=1&modificationDate=1711704762702&cacheVersion=1&api=v2&width=1200&height=409)
RDS CPU Utilization
RDS CPU utilized:
8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 50% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test.
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-37-14.png?version=1&modificationDate=1711705035902&cacheVersion=1&api=v2&width=1200&height=463)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-39-44.png?version=1&modificationDate=1711705186128&cacheVersion=1&api=v2&width=1200&height=454)
RDS Database Connections
For both 45 minute and longevity tests RDS used max 615 connections. Without test it was 550 connections.
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-43-44.png?version=1&modificationDate=1711705426244&cacheVersion=1&api=v2&width=1200&height=460)
Longevity test
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-44-36.png?version=1&modificationDate=1711705478468&cacheVersion=1&api=v2&width=1200&height=419)
CPU (User) usage by broker
Common CPU utilization by broker during all tests was 15% with equal distribution between brokers
45 minute tests
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-53-8.png?version=1&modificationDate=1711705990775&cacheVersion=1&api=v2&width=1200&height=394)
Longevity test![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_11-53-32.png?version=1&modificationDate=1711706014117&cacheVersion=1&api=v2&width=1200&height=405)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_12-0-41.png?version=1&modificationDate=1711706443753&cacheVersion=1&api=v2&width=1200&height=408)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_12-1-37.png?version=1&modificationDate=1711706499576&cacheVersion=1&api=v2&width=1200&height=386)
![](https://folio-org.atlassian.net/wiki/download/thumbnails/115245057/image2024-3-29_12-2-20.png?version=1&modificationDate=1711706542399&cacheVersion=1&api=v2&width=1200&height=409)
Appendix
Infrastructure
PTF -environment qcp1 |
---|
- 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 1 instance of db.r6.xlarge database instance: Writer instance
- MSK[ kafka configurations]
|
DB table records size: - fs09000000
- instances - 24493326
- items - 25662862
- holding records - 24459683
|
---|
Modules
Check-In / Check-Out related modules
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
---|
qcp1-pvt |
|
|
|
|
|
|
|
|
|
Mon Mar 25 12:41:11 UTC 2024 |
|
|
|
|
|
|
|
|
|
mod-data-import | 4 | mod-data-import:3.1.0-SNAPSHOT.174 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 |
mod-search | 2 | mod-search:3.2.0-SNAPSHOT.199 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 |
mod-configuration | 2 | mod-configuration:5.11.0-SNAPSHOT.348 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-entities-links | 3 | mod-entities-links:3.0.0-SNAPSHOT.78 | 2 | 2592 | 2480 | 2096 | 1440 | 0 | 1024 |
pub-edge | 2 | pub-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
edge-patron | 2 | edge-patron:5.1.0-SNAPSHOT.153 | 2 | 1024 | 896 | 256 | 768 | 88 | 128 |
mod-users-bl | 2 | mod-users-bl:7.6.1-SNAPSHOT.282 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-inventory-storage | 2 | mod-inventory-storage:27.1.0-SNAPSHOT.719 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
mod-rtac | 2 | mod-rtac:3.6.0-SNAPSHOT.166 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-circulation-storage | 2 | mod-circulation-storage:17.2.0-SNAPSHOT.384 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
mod-circulation-item | 2 | mod-circulation-item:1.0.0-SNAPSHOT.16 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-pubsub | 2 | mod-pubsub:2.13.0-SNAPSHOT.196 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 |
mod-di-converter-storage | 2 | mod-di-converter-storage:2.2.0-SNAPSHOT.108 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-patron-blocks | 2 | mod-patron-blocks:1.10.0-SNAPSHOT.152 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 |
mod-source-record-manager | 2 | mod-source-record-manager:3.8.0-SNAPSHOT.306 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
mod-quick-marc | 2 | mod-quick-marc:5.1.0-SNAPSHOT.134 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 |
nginx-okapi | 2 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 1024 | 0 | 0 | 0 |
okapi-b | 2 | okapi:5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
mod-feesfines | 2 | mod-feesfines:19.1.0-SNAPSHOT.189 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-dcb | 2 | mod-dcb:1.1.0-SNAPSHOT.18 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-oai-pmh | 2 | mod-oai-pmh:3.13.0-SNAPSHOT.393 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
pub-okapi | 2 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
All modules
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
---|
qcp1-pvt |
|
|
|
|
|
|
|
|
|
Mon Mar 25 12:41:11 UTC 2024 |
|
|
|
|
|
|
|
|
|
mod-remote-storage | 2 | mod-remote-storage:3.2.0-SNAPSHOT.162 | 2 | 4920 | 4472 | 1024 | 3960 | 512 | 512 |
mod-ncip | 2 | mod-ncip:1.14.5-SNAPSHOT.224 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-finance-storage | 2 | mod-finance-storage:8.6.0-SNAPSHOT.159 | 2 | 1024 | 896 | 1024 | 700 | 88 | 128 |
mod-agreements | 2 | mod-agreements:6.1.0-SNAPSHOT.195 | 2 | 1592 | 1488 | 128 | 0 | 0 | 0 |
mod-ebsconet | 2 | mod-ebsconet:2.2.0-SNAPSHOT.74 | 2 | 1248 | 1024 | 128 | 700 | 128 | 256 |
mod-organizations | 2 | mod-organizations:1.9.0-SNAPSHOT.84 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-sip2 | 2 | edge-sip2:3.2.0-SNAPSHOT.209 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-serials-management | 2 | mod-serials-management:1.0.0-SNAPSHOT.25 | 2 | 2480 | 2312 | 128 | 1792 | 384 | 512 |
mod-settings | 2 | mod-settings:1.0.3-SNAPSHOT.63 | 2 | 1024 | 896 | 200 | 768 | 88 | 128 |
mod-data-import | 4 | mod-data-import:3.1.0-SNAPSHOT.174 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 |
edge-dematic | 2 | edge-dematic:2.1.2-SNAPSHOT.130 | 1 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-search | 2 | mod-search:3.2.0-SNAPSHOT.199 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 |
mod-inn-reach | 2 | mod-inn-reach:3.2.0-SNAPSHOT.86 | 2 | 3600 | 3240 | 1024 | 2880 | 512 | 1024 |
mod-tags | 2 | mod-tags:2.1.1-SNAPSHOT.132 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-courses | 2 | edge-courses:1.3.2-SNAPSHOT.1138 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-authtoken | 2 | mod-authtoken:2.15.0-SNAPSHOT.290 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-inventory-update | 2 | mod-inventory-update:3.2.2-SNAPSHOT.82 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-notify | 2 | mod-notify:3.1.1-SNAPSHOT.262 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-configuration | 2 | mod-configuration:5.11.0-SNAPSHOT.348 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-orders-storage | 2 | mod-orders-storage:13.7.0-SNAPSHOT.215 | 2 | 1024 | 896 | 512 | 700 | 88 | 128 |
edge-caiasoft | 2 | edge-caiasoft:2.1.1-SNAPSHOT.82 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-login-saml | 2 | mod-login-saml:2.8.0-SNAPSHOT.280 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-erm-usage-harvester | 2 | mod-erm-usage-harvester:4.5.0-SNAPSHOT.123 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-licenses | 2 | mod-licenses:5.1.0-SNAPSHOT.147 | 2 | 2480 | 2312 | 128 | 1792 | 384 | 512 |
mod-gobi | 2 | mod-gobi:2.8.0-SNAPSHOT.244 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-password-validator | 2 | mod-password-validator:3.1.1-SNAPSHOT.137 | 2 | 1440 | 1298 | 128 | 768 | 384 | 512 |
mod-bulk-operations | 2 | mod-bulk-operations:2.0.0-SNAPSHOT.81 | 2 | 3072 | 2600 | 1024 | 1536 | 384 | 512 |
mod-fqm-manager | 2 | mod-fqm-manager:1.1.0-SNAPSHOT.1083 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-dcb | 2 | edge-dcb:1.1.0-SNAPSHOT.16 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-graphql | 2 | mod-graphql:1.12.1.401 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-finance | 2 | mod-finance:4.9.0-SNAPSHOT.160 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-erm-usage | 2 | mod-erm-usage:4.7.0-SNAPSHOT.99 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-batch-print | 3 | mod-batch-print:1.2.0-SNAPSHOT.10 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-copycat | 2 | mod-copycat:1.6.0-SNAPSHOT.197 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-lists | 2 | mod-lists:1.1.0-SNAPSHOT.1266 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-entities-links | 3 | mod-entities-links:3.0.0-SNAPSHOT.78 | 2 | 2592 | 2480 | 2096 | 1440 | 0 | 1024 |
mod-permissions | 4 | mod-permissions:6.5.0-SNAPSHOT.374 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 |
pub-edge | 2 | pub-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
mod-orders | 2 | mod-orders:12.8.0-SNAPSHOT.332 | 2 | 2048 | 1440 | 1024 | 1024 | 384 | 512 |
edge-patron | 2 | edge-patron:5.1.0-SNAPSHOT.153 | 2 | 1024 | 896 | 256 | 768 | 88 | 128 |
edge-ncip | 2 | edge-ncip:1.9.3-SNAPSHOT.103 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-inn-reach | 2 | edge-inn-reach:3.1.1-SNAPSHOT.45 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-users-bl | 2 | mod-users-bl:7.6.1-SNAPSHOT.282 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-oa | 2 | mod-oa:2.1.0-SNAPSHOT.62 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-inventory-storage | 2 | mod-inventory-storage:27.1.0-SNAPSHOT.719 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
mod-invoice | 3 | mod-invoice:5.8.0-SNAPSHOT.187 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
mod-user-import | 2 | mod-user-import:3.9.0-SNAPSHOT.192 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-sender | 3 | mod-sender:1.11.1-SNAPSHOT.94 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-oai-pmh | 2 | edge-oai-pmh:2.8.0-SNAPSHOT.155 | 2 | 1512 | 1360 | 1024 | 1440 | 384 | 512 |
mod-data-export-worker | 2 | mod-data-export-worker:3.2.0-SNAPSHOT.195 | 2 | 3072 | 2800 | 1024 | 2048 | 384 | 512 |
mod-rtac | 2 | mod-rtac:3.6.0-SNAPSHOT.166 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-circulation-storage | 2 | mod-circulation-storage:17.2.0-SNAPSHOT.384 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
mod-calendar | 2 | mod-calendar:2.6.0-SNAPSHOT.218 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-source-record-storage | 2 | mod-source-record-storage:5.8.0-SNAPSHOT.353 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
mod-event-config | 2 | mod-event-config:2.6.1-SNAPSHOT.100 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-courses | 2 | mod-courses:1.4.9-SNAPSHOT.143 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-circulation-item | 2 | mod-circulation-item:1.0.0-SNAPSHOT.16 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-inventory | 2 | mod-inventory:20.2.0-SNAPSHOT.501 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
mod-email | 2 | mod-email:1.16.1-SNAPSHOT.122 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-pubsub | 2 | mod-pubsub:2.13.0-SNAPSHOT.196 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 |
mod-circulation | 2 | mod-circulation:24.2.0-SNAPSHOT.649 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
mod-di-converter-storage | 2 | mod-di-converter-storage:2.2.0-SNAPSHOT.108 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-rtac | 2 | edge-rtac:2.7.0-SNAPSHOT.212 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-orders | 2 | edge-orders:2.10.0-SNAPSHOT.123 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-users | 2 | mod-users:19.3.0-SNAPSHOT.679 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-template-engine | 2 | mod-template-engine:1.19.2-SNAPSHOT.111 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-patron-blocks | 2 | mod-patron-blocks:1.10.0-SNAPSHOT.152 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 |
mod-audit | 2 | mod-audit:2.9.0-SNAPSHOT.207 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
edge-fqm | 2 | edge-fqm:1.1.0-SNAPSHOT.84 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-source-record-manager | 2 | mod-source-record-manager:3.8.0-SNAPSHOT.306 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
nginx-edge | 2 | nginx-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 |
mod-quick-marc | 2 | mod-quick-marc:5.1.0-SNAPSHOT.134 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 |
nginx-okapi | 2 | nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 1024 | 0 | 0 | 0 |
okapi-b | 2 | okapi:5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
mod-feesfines | 2 | mod-feesfines:19.1.0-SNAPSHOT.189 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-invoice-storage | 2 | mod-invoice-storage:5.8.0-SNAPSHOT.123 | 2 | 1872 | 1536 | 1024 | 1024 | 384 | 512 |
mod-dcb | 2 | mod-dcb:1.1.0-SNAPSHOT.18 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-service-interaction | 2 | mod-service-interaction:3.1.0-SNAPSHOT.117 | 2 | 2048 | 1844 | 256 | 1290 | 384 | 512 |
mod-patron | 2 | mod-patron:6.1.0-SNAPSHOT.147 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-oai-pmh | 2 | mod-oai-pmh:3.13.0-SNAPSHOT.393 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
edge-connexion | 2 | edge-connexion:1.2.0-SNAPSHOT.72 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-kb-ebsco-java | 2 | mod-kb-ebsco-java:4.1.0-SNAPSHOT.162 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-notes | 2 | mod-notes:5.2.0-SNAPSHOT.306 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 |
mod-data-export-spring | 2 | mod-data-export-spring:3.2.0-SNAPSHOT.149 | 1 | 2048 | 1844 | 256 | 1536 | 384 | 512 |
mod-organizations-storage | 2 | mod-organizations-storage:4.7.0-SNAPSHOT.125 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
mod-login | 2 | mod-login:7.11.0-SNAPSHOT.314 | 2 | 1440 | 1298 | 1024 | 768 | 384 | 512 |
pub-okapi | 2 | pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
mod-eusage-reports | 2 | mod-eusage-reports:2.1.0-SNAPSHOT.65 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
Methodology/Approach
- Run data preparation script before each CI/CO test
- Update .jmx file script for Quesnelia release
- Create artefact and upload to carrier-io
- Use Jenkins job to change parameters and run tests
- Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.
- Test CI/CO with 30 users for 24 hours to detect any trends in memory.