Overview

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests. PERF-879 - Getting issue details... STATUS

Summary

Check-In/Check-Out tests on Quesnelia release (Eureka) in QECP1 environment carried out with 8, 20, 30, 75 virtual users during 45 minutes and with 30 virtual users during longevity test (24 hours) to detect any trends in resource utilization

45 minute tests

• Response times in tests with 8, 20, 30 virtual users delta was 200 ms. The lowest average response times were with 20 virtual users test: Check-In - 625 ms, average Check-Out - 1356 ms
• The highest average response times showed 75 vUsers test: Check-In - 4 sec 610 ms, Check-Out - 12 sec 589 ms (whole test), Check-In - 622 ms, Check-Out - 1540 ms (before saturation point with 33 vUsers)

Longevity test

• Longevity test was running 10 hours and began failing because of token expiration. Possible reason here is SSO Session Max limitation 10 hours (fs09000000/realm-settings/sessions).
  • Average response times in 10 hours test for Check-In - 1003 ms, Check-Out - 2870 ms
  • First 5 hours 30 minutes average response times Check-In - 770 ms, Check-Out - 1773 ms, the next 4 hours 30 minutes - Check-In - 1295 ms, Check-Out - 4280 ms
• Longevity during 24 hours (after session expiration issue was resolved by increasing SSO session Max parameter from 10 to 24 hours)
  • Average response times for all 24 test Check-In - 1311 ms, Check-Out - 4541 ms
  • The same degradation observed as in 10 hours test. Response time increased significantly in 6 hours after test start - Check-In from 1045 ms to 1526 ms, Check-out from 2523 ms to 5262 ms. See Graphs for response times.

Comparison with previous Quesnelia release (qcp1 environment) results:

• 8, 20, 30 vUsers tests were stable. And 75 vUsers test revealed saturation point during ramp-up period with 33 vUsers. After this point response times increased significantly with the same throughput.
• There's a significant performance degradation compared to previous results from tests in quesnelia release on qcp1 environment (non-eureka):
  • 8 vUsers - Check-out - 96%, Check-in - 48%
  • 20 vUsers - Check-out - 61%, Check-in - 33%
  • 30 vUsers - Check-out - 89%, Check-in - 55%
  • 75 vUsers - Check-out - 1111%, Check-in - 695% (during whole test)
  • 75 vUsers - Check-out - 48 %, Check-in - 7 % (before saturation point was found)
  • 30 vUsers (Longevity) - Check-out - 326%, Check-in - 188% (24 hours test)

Resources

• CPU utilization
  • 45 minute and longevity tests used CPU in correspondence with number of vUsers but there's module that used about 16% - mod-inventory-storage-b. We do not observe expected growth for module CPU utilization in 75 vUsers because of saturation point.

• Memory consumption
  • 45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%

• RDS CPU utilization average
  • For both 45 minute and longevity tests RDS used max 867 connections. Without test it was 812 connections.

• CPU (User) usage by broker
  • Common CPU utilization by broker during longevity was 18, 18, 16, 14% accordingly to broker distribution. Resource utilization decreased significantly after 6 hours 20 minutes (17:40 UTC) to 14, 12, 12, 11%

Notes:

• The reason of degradation, which occurs after 6 hours of testing, has not yet been found.

Recommendations

• Investigate degradation after 6 hours of longevity test.

Test Runs 

The following table contains  tests configuration information

Results

Response time

The table contains results of Check-in, Check-out tests in Quesnelia release (Eureka). All requests that are >= 100 ms (in average response times)

45 minute tests

Longevity test

Comparisons

This table has comparison between average values of response times of Quesnelia (Check-in-check-out Test Report (Quesnelia)) and Quesnelia (Eureka) releases

Comparison of Longevity test

Resources Utilization

CPU Utilization

45 minute and longevity tests used CPU in correspondence with number of vUsers but there's module that used about 16% - mod-inventory-storage-b. We do not observe expected growth for module CPU utilization in 75 vUsers.

45 minute tests

Longevity test 10 hours

Longevity test 24 hours

Memory Consumption

45 minute and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%

45 minute tests

Longevity test 10 hours

Longevity test 24 hours

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 14%, 20 vUsers - 26%, 30 vUsers - 35%, 75 vUsers - 36% During longevity test CPU stayed on the same level - 38% 

45 minute tests

Longevity test 10 hours 

Longevity test 24 hours

During the test RDS grew from 32% to 40% in average. Spikes observed after 10 hours

RDS Database Connections

For both 45 minute and longevity tests RDS used max 867 connections. Without test it was 812 connections.

45 minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 18, 15, 15, 13 % accordingly to broker distribution in 45 min tests

Common CPU utilization by broker during longevity was 18, 18, 16, 14% accordingly to broker distribution. Resource utilization decreased significantly after 6 hours 20 minutes (17:40 UTC) to 14, 12, 12, 11%

45 minute tests

Longevity test

Longevity

Target Group 5XX Count

Longevity test 24 hours

Appendix

Infrastructure

Modules

Methodology/Approach

• Steps to avoid possible session or token expiration:
  • modify tenant/realm-settings/tokens in keycloaks UI to 24 hours (1 day)
  • modify tenant/realm-settings/sessions in keycloaks UI to 24 hours (1 day) Default - 10 hours

• Run data preparation script before each CI/CO test
• Update .jmx file script for Quesnelia release
• Create  artefact and upload to carrier-io
• Use Jenkins job to change parameters and run tests
• Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each. 
• Test CI/CO with 30 users for 24 hours to detect any trends in memory.