Overview

• Regression testing of Check-In/Check-Out (CI/CO) tests in Trillium ECS release.

• Test on central tenant won’t happen as on ptf environment we have restored DB from Bugfest and central consortia tenant doesn’t have items. We’ll proceed testing on member tenants

• The purposes of CI/CO testing:

  • To test how different load from CI/CO flow affect the system

  • To define response times of Check-In and Check-Out

  • To define the response times for requests that take longer than 100 milliseconds

  • To find any trends for resource utilization and recommend improvements

  • To check how system behaves over extended period during longevity test.

  • Compare results (current and previous)

  • The current ticket: https://folio-org.atlassian.net/browse/PERF-1372 , https://folio-org.atlassian.net/browse/PERF-1430

  • The previous results report: Check-in-check-out Test Report (Sunflower) [ECS] CSP1&CSP2

Summary

• Common results:

  • Check-in and check-out flows in Trillium showed improvement compared to Sunflower. Check-out improved by 15–31% for lower user loads and ~22% at 75 vUsers. Check-in improved by 12–34% for 8–30 vUsers but degraded by ~14.5% at 75 vUsers.

• Tests #1, #2, #3, #4

  • Response times in average:

    • Check-in — minimum 314 ms with 20 vUsers and maximum 970 ms with 75 vUsers.

    • Check-out — minimum 894 ms with 20 vUsers and maximum 2211 ms with 75 vUsers.

• Longevity test

  • Showed good performance, comparable with the rest of a tests CI 280 ms, CO 906 ms.

  • Got significantly smaller DB CPU usage comparing with Sunflower

  • mod-circulation start crashing after end of a test with OOM https://folio-org.atlassian.net/browse/CIRC-2630

  • mod-inventory storage got growing memory trend that will lead to OOM as well. https://folio-org.atlassian.net/browse/MODINVSTOR-1583

Resources

• CPU utilization

  • CPU utilization depends on number of concurrent virtual users and increases accordingly.

  • Top used modules: mod-circulation-b, mod-circulation-storage-b, mod-users-b, kong-b, mod-rtac-cache-b, mod-remote-storage-b, mod-feesfines-b, mod-inventory-storage-b.

  • CPU utilization for 30 vUsers: mod-circulation-188, mod-circulation-storage-107, mod-rtac-cache-96, mod-users-85, kong-82, mod-remote-storage-49, mod-feesfines-42, mod-inventory-storage-41

• Memory consumption

  • Memory consumption was stable during tests. The highest usage for top consumer Mgr-applications-b was at 63%.

• RDS CPU utilization average

  • 8 vUsers — 12%, 20 vUsers — 18%, 30 vUsers — 27%, 75 vUsers — 57%. Significantly lower than Sunflower (which reached 99% at 75 vUsers).

• CPU (User) usage by broker

  • 8 vUsers - 11%, 20 vUsers - 12%, 30 vUsers - 13%, 75 vUsers - 16%.

Test Runs 

The following table contains tests configuration information

Results

Response time

The table contains results of Check-in, Check-out tests in Trillium release. 

Test #1, #2, #3, #4

Graph (Response times) Test #1

Open

 

Graph (Response times) Test #2

Open

 

Graph (Response times) Test #3

Open

 

 

Graph (Response times) Test #4

Open

 

Longevity test:

Comparisons

This table has comparison between average values of response times of Trillium and Sunflower releases

API requests where response times >= 100 milliseconds

Longevity test comparison: