Test status: PASSED

Overview

• Regression testing of Check-In/Check-Out (CI/CO) fixed load tests on okapi based environment in Ramsons ECS release.

• Test on central tenant won’t happen as on ptf environment we have restored DB from Bugfest and central consortia tenant doesn’t have items. We’ll proceed testing on member tenants

• The purposes of CI/CO testing:

  • To test how different load from CI/CO flow affect the system

  • To define response times of Check-In and Check-Out

  • To define the response times for requests that take longer than 100 milliseconds

  • To find any trends for resource utilization and recommend improvements

  • To check how system behaves over extended period during longevity test

  • Compare results (current and previous)

  • The current ticket: 

    PERF-983 - [Ramsons] [ECS] CI/CO

  • The previous results report: [ECS] Check-in-check-out Test Report (Quesnelia)

Summary

• Common results:

  • Check-in/Check-out flows degraded 25% in average for all tests except longevity test.

• Tests #1, #2, #3, #4

  • Response times in average: 

    • Check-in - minimum 556 ms with 30 vUsers and maximum 659 ms with 75 vUsers.

    • Check-out - minimum 1120 ms with 30 vUsers and maximum 1483 ms with 75 vUsers. 

• Test #5

  • Response times in average: 

    • Check-in - 493 ms with 30 vUsers 

    • Check-out - 1095 ms with 30 vUsers. 

• Comparison with Quesnelia results:

  • CI/CO response times degradation (Tests #1, #2, #3, #4): 

  • CI/CO response times (test #5 - longevity test):

    • 30 vUsers - 13% improved in CO  and 8% degraded in CI flow.

Resources

• CPU utilization

  • CPU utilization depends on number of concurrent virtual users and increasing accordingly. Top used modules: okapi, mod-pubsub, mod-inventory, mod-inventory-storage, mod-circulation, mod-users, mod-authtoken, nginx-okapi.

  • During 75 vUsers test nginx-okapi module spiked from 1% to 12% during ramp-up period after 40 vUsers point.

• Memory consumption

  • Memory consumption of mod-pubsub module grow during test #4 75 vUsers test. It can be connected to saturation point after increased load. The highest usage was at 65%.

  • Mod-pubsub grew during longevity test. The highest usage was at 81%.

  • Top 5 modules, consumption at % in 75 vUsers test: mod-oa-b - 75%, mod-pubsub-b - 69%, mod-dcb-b - 62%, mod-inventory-b - 61%, mod-data-import-b - 54%

• RDS CPU utilization average

  • 8 vUsers - 15%, 20 vUsers - 26%, 30 vUsers - 32%, 75 vUsers - 54% During longevity test CPU was 30% in average.

• CPU (User) usage by broker

  • 8 vUsers - 12%, 20 vUsers - 13%, 30 vUsers - 14%, 75 vUsers - 17%. During Longevity test - 2 brokers - 17%, 2 other brokers - 28%.

Recommendations & Jiras

• mod-serials-management-b affect DB connection growth 200 connection in average. Disabling this module do not affect response times or error rate but significantly decrease DB connection number.

• Investigate mod-pubsub module memory growing trend. Check errors during successful 30 vUsers test in Error block. MODPUBSUB-311: Investigate mod-pubsub module memory growing during Check-in/Check-outClosedPreview

Test Runs 

The following table contains  tests configuration information

Results

Response time

The table contains results of Check-in, Check-out tests in Ramsons release. 

Test #1, #2, #3, #4

Graph (Response times) Test #1

Open

 

Graph (Response times) Test #2

Open

 

Graph (Response times) Test #3

Open

 

Errors here connected to a short time connectivity problem during 30 seconds. Additional run for 30 vUsers didn't reveal this connectivity problem.

Graph (Response times) Test #4

Open

Open

Open

Test #5

Graph (Response times) Test #5

Open

 

Comparisons

This table has comparison between average values of response times of Ramsons and Quesnelia releases

Comparison of longevity test