Table of Contents

...

The test's target is to investigate the behaviour of the Check-In/Check-Out activities during fixed load tests.

Summary

...

Check-In/Check-Out tests on Quesnelia release (Eureka) in QECP1 environment carried out with 8, 20, 30, 75 virtual users during 45 minutes and with 30 virtual users during longevity test (24 hours) to detect any trends in resource utilization

...

• Longevity test was running 10 hours and began failing because of token expiration. Possible reason here is SSO Session Max limitation 10 hours (fs09000000/realm-settings/sessions).
  • Average response times in 10 hours test for Check-In - 1003 ms, Check-Out - 2870 ms
  • First 5 hours 30 minutes average response times Check-In - 770 ms, Check-Out - 1773 ms, the next 4 hours 30 minutes - Check-In - 1295 ms, Check-Out - 4280 ms
• Longevity during 24 hours (after session expiration issue was resolved by increasing SSO session Max parameter from 10 to 24 hours)
  • Average response times for all 24 test Check-In - 1311 ms, Check-Out - 4541 ms
  • The same degradation observed as in 10 hours test. Response time increased significantly in 6 hours after test start - Check-In from 1045 ms to 1526 ms, Check-out from 2523 ms to 5262 ms. See Graphs for response times.

Comparison with previous results in Quesnelia release (qcp1 environment) results:

• 8, 20, 30 vUsers tests were stable. And 75 vUsers test revealed saturation point during ramp-up period with 33 vUsers. After this point response times increased significantly with the same throughput.
• There's a significant performance degradation compared to previous results from tests in quesnelia release on qcp1 environment (non-eureka):
  • 8 vUsers - Check-out - 96%, Check-in - 48%
  • 20 vUsers - Check-out - 61%, Check-in - 33%
  • 30 vUsers - Check-out - 89%, Check-in - 55%
  • 75 vUsers - Check-out - 1111%, Check-in - 695% (during whole test)
  • 75 vUsers - Check-out - 48 %, Check-in - 7 % (before saturation point was found)
  • 30 vUsers (Longevity) - Check-out - 326%, Check-in - 188% (24 hours test)

...

• RDS CPU utilization average
  • For both 45 minute and longevity tests RDS used max 867 connections. Without test it was 812 connections.

• CPU (User) usage by broker
  • Common CPU utilization by broker during longevity was 18, 18, 16, 14% accordingly to broker distribution. Resource utilization decreased significantly after 6 hours 20 minutes (17:40 UTC) to 14, 12, 12, 11%

Notes:

• The reason of degradation, which occurs after 6 hours of testing, has not yet been found.

Recommendations

• Investigate degradation after 6 hours of longevity test.

Test Runs 

The following table contains  tests configuration information

Results

Errors:

Response time

The table contains results of Check-in, Check-out tests in Quesnelia release (Eureka). All requests that are >= 100 ms (in average response times)

...

• empty cells - response times less than 100 ms

Longevity test

...

During the test RDS grew from 32% to 40% in average. Spikes observed after 10 hours

RDS Database Connections

For both 45 minute and longevity tests RDS used max 867 connections. Without test it was 812 connections.

45 minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 18, 15, 15, 13 % accordingly to broker distribution in 45 min tests

Common CPU utilization by broker during longevity was 18, 18, 16, 14% accordingly to broker distribution. Resource utilization decreased significantly after 6 hours 20 minutes (17:40 UTC) to 14, 12, 12, 11%

45 minute tests

Longevity test

Longevity

Target Group 5XX Count

Longevity test 24 hours

Appendix

Infrastructure

...

Methodology/Approach

• Steps to avoid possible session or token expiration:
  
  • modify tenant/realm-settings/sessions in keycloaks UI to 24 hours (1 day) Default - 10 hours

• Run data preparation script before each CI/CO test
• Update .jmx file script for Quesnelia release
• Create  artefact and upload to carrier-io
• Use Jenkins job to change parameters and run tests
• Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each. 
• Test CI/CO with 30 users for 24 hours to detect any trends in memory. 

...