• 1 Overview

• 2 Summary

• 3 Test Runs 

• 4 Results

  • 4.1 Comparisons

  • 4.2 Cluster Resources Utilization

    • 4.2.1 CPU Utilization

    • 4.2.2 Memory Consumption

    • 4.2.3 RDS CPU Utilization

    • 4.2.4 RDS Database Connections

    • 4.2.5 Database load

  • 4.3 MSK resources utilization

    • 4.3.1 CPU (User) usage by broker

  • 5 Appendix

  • 5.1 Infrastructure

  • 5.2 Methodology and Approach

Overview

Regression testing of Check-In/Check-Out (CI/CO) fixed load tests on okapi based environment in Ramsons ECS release.

• Test on central tenant won’t happen as on ptf environment we have restored DB from Bugfest and central consortia tenant doesn’t have items. We’ll proceed testing on member tenants

• The purposes of CI/CO testing:

  • To test how different load from CI/CO flow affect the system

  • To define response times of Check-In and Check-Out

  • To define the response times for requests that take longer than 100 milliseconds

  • To find any trends for resource utilization and recommend improvements

  • To check how system behaves over extended period during longevity test

  • Compare results (current and previous)

  • The current ticket: PERF-1125 - [Sunflower] [ECS] CI/CO

  • The previous results report: [ECS] Check-in-check-out Test Report (Ramsons)

Summary

• Common results:

  • Check-in and check-out flows degraded as the number of users increased. However, check-in in Sunflower performed slightly better than Ramsons in general, while check-out consistently showed worse results. Compared to Ramsons, resource utilization is higher in Sunflower.

• Tests #1, #2, #3, #4

  • Response times in average: 

    • Check-in - minimum 445 ms with 20 vUsers and maximum 847 ms with 75 vUsers.

    • Check-out - minimum 1122 ms with 8 vUsers and maximum 2825 ms with 75 vUsers. 

• Test #5

  • Response times in average: 

    • Check-in - 517 ms with 30 vUsers 

    • Check-out - 1699 ms with 30 vUsers.

Resources

• CPU utilization

  • CPU utilization depends on number of concurrent virtual users and increasing accordingly. Top used modules: mod-users-keycloak, mod-pubsub, mod-inn-reach-b, mod-inventory-storage, mod-circulation, mod-users,

  • Nearly during all tests mod-users-keycloak consumed most CPU.

• Memory consumption

  • Memory consumption of mod-pubsub module grow when test users increased.

  • Mod-dcb and mgr-applications were highest at 80%.

  • Top 5 modules, consumption at % in 75 vUsers test: mgr-application - 82%, mod-dcb-b - 74%, mod-keycloak-b - 69%, mod-scheduler-b - 69%, mod-finance-b - 68%

• RDS CPU utilization average

  • 8 vUsers - 25%, 20 vUsers - 54%, 30 vUsers - 73%, 75 vUsers - 99% During longevity test CPU was 80%.

• CPU (User) usage by broker

  • 8 vUsers - 40%, 20 vUsers - 42%, 30 vUsers - 43%, 75 vUsers - 28%. During Longevity test - 3 brokers - above 30%, other broker - 26%. 

Test Runs 

The following table contains  tests configuration information

Results

Response time

The table contains results of Check-in, Check-out tests in Sunflower release. 

Test #1, #2, #3, #4

Graph (Response times) Test #1

Open

Graph (Response times) Test #2

Open

 

Graph (Response times) Test #3

Open

 

Graph (Response times) Test #4

Open

 

Test #5

Graph (Response times) Test #5

Open

 

Comparisons

This table has comparison between average values of response times of Sunflower and Ramsons releases

Comparison of longevity test

API requests where response times >= 100 milliseconds

Cluster Resources Utilization

CPU Utilization

CPU utilization depends on number of concurrent virtual users and increasing accordingly.