[ECS] Check-in-check-out Test Report (Quesnelia)

Overview

The test's target is to investigate the behavior of the Check-In/Check-Out activities during fixed load tests.

PERF-857 - Getting issue details... STATUS

Summary

Check-In/Check-Out tests on Quesnelia were carried out with 8, 20, 30, and 75 virtual users for 45 minutes and 30 virtual users for 24 hours to detect any trends in resource utilization (only 16 hours of results available).

45-minute tests

  1. Response times in tests with 8, 20, and 30 virtual users didn't differ a lot but the lowest response times were with 20 virtual users. Average Check-In - 441 ms, average Check-Out - 893 ms
  2. The highest response time results were with 75 virtual users. Average Check-In - 546 ms, Check-Out - 1176 ms
  3. Reports: reports_test_results_8.zip, reports_test_results_20.zipreports_test_results_30.zip, reports_test_results_75.zip

Longevity test

  • Average Check-In - 458 ms, Check-Out - 1258 ms (the results from the test 16 hours)
  • Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.
  • RDS CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.
  • Grafana results

Resources

  • CPU utilization
    • 45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.
  • Memory consumption
    • Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.
  • Average RDS CPU utilization
    • 8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.
  • CPU (User) usage by broker
    • Common CPU utilization by the broker during all tests was up to 12% with equal distribution between brokers

Common notes

  • Longevity test revealed Database CPU Utilization growing trend and CI and CO response time increasing.

Recommendations

  • Add more CPU to nginx-okapi to exclude module fluctuations

Test Runs 

The following table contains test configuration information

Test #

Test artefact

vUsersRamp-up, sec

Duration, sec 

1

circulation_checkInCheckOut_qcon8802700
2
202002700
3
303002700
4
757502700
5
3030086400

Results

Errors:

  • Error messages: Response 401 due to changed login process and token expiration time 10 min (0.04%).

Response time

The table contains results of Check-in, Check-out tests in the Quesnelia release. 

45-minute tests


8 vUsers  20 vUsers  30 vUsers  75 vUsers  
Requests

Response Times (ms)

 Response Times (ms) Response Times (ms) Response Times (ms)
Label

#Samples

Average

95th pct

#Samples

Average

95th pct

#Samples

Average

95th pct

#Samples

Average

95th pct

Check-Out Controller1815910.511102.404324893.261054.006626899.131058.00109861176.401569.00
Check-In Controller1391476.67598.803527441.99551.604911435.40544.0014564

546.74

760.00
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)1798275.86340.004275268.25315.006556268.31316.0014434331.68491.00
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)1388243.06313.553518214.64275.004906205.53267.0011159233.01348.00
GET_circulation/loans (Submit_patron_barcode)1805143.41173.004292141.66160.006586140.52161.0014493177.03273.00
GET_circulation/loans (Submit_barcode_checkout)1795140.59163.204275139.57158.006550140.00159.0014418174.65265.00

Longevity test

 30 vUsers Longevity 


RequestsSamples, Response Times 

Label#Samples95th pctAverage
Check-Out Controller1441291.9981.258
Check-In Controller1084470.6450.458
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)1424880.6130.383
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)1079070.3010.200

Resources Utilization

CPU Utilization

45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.

45 minute tests

Longevity test

Memory Consumption

45-minute tests didn't reveal some problems with memory usage by modules. Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

45-minute tests

Longevity test

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

45-minute tests

Longevity test

RDS Database Connections

For both 45-minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.

45-minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 12% with equal distribution between brokers

45 minute tests

Longevity test

Appendix

Infrastructure

PTF -environment qcon
  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 instance of db.r6.xlarge database instance: Writer instance
  • MSK - tenant
    • 4 kafka.m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

DB table records size:

  • fs09000000
    • instances -            24493326
    • items -                  25662862
    • holding records - 24459683

Modules

 Check-In / Check-Out related modules
Module qcon-pvtTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-agreements1mod-agreements:7.0.0215921488128000FALSE
mod-authtoken1mod-authtoken:2.15.121440115251292288128FALSE
mod-bulk-operations1mod-bulk-operations:2.0.023072260010241536384512FALSE
mod-calendar1mod-calendar:3.1.02102489612876888128FALSE
mod-circulation2mod-circulation:24.2.122880259215361814384512FALSE
mod-circulation-item1mod-circulation-item:1.0.021024896128000FALSE
mod-circulation-storage1mod-circulation-storage:17.2.022880259215361814384512FALSE
mod-configuration1mod-configuration:5.10.02102489612876888128FALSE
mod-dcb1mod-dcb:1.1.02102489612876888128FALSE
mod-feesfines1mod-feesfines:19.1.02102489612876888128FALSE
mod-inventory1mod-inventory:20.2.022880259210241814384512FALSE
mod-inventory-storage1mod-inventory-storage:27.1.024096369020483076384512FALSE
mod-login1mod-login:7.11.02144012981024768384512FALSE
mod-notes1mod-notes:5.2.021024896128952384512FALSE
mod-patron-blocks1mod-patron-blocks:1.10.021024896102476888128FALSE
mod-pubsub1mod-pubsub:2.13.02153614401024922384512FALSE
mod-remote-storage1mod-remote-storage:3.2.024920447210243960512512FALSE
mod-search1mod-search:3.2.0225922480204814405121024FALSE
mod-users1mod-users:19.3.12102489612876888128FALSE
nginx-okapi1nginx-okapi:2023.06.1421024896128000FALSE
okapi-b1okapi:5.3.03168414401024922384512FALSE
pub-edge1pub-edge:2023.06.142102489612876800FALSE
pub-okapi1pub-okapi:2023.06.142102489612876800FALSE
 All modules
ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
qcon-pvt
Fri Apr 05 15:19:05 UTC 2024
mod-remote-storage1mod-remote-storage:3.2.024920447210243960512512FALSE
mod-finance-storage1mod-finance-storage:8.6.021024896102470088128FALSE
mod-ncip1mod-ncip:1.14.42102489612876888128FALSE
mod-agreements1mod-agreements:7.0.0215921488128000FALSE
mod-ebsconet1mod-ebsconet:2.2.0212481024128700128256FALSE
mod-consortia1mod-consortia:1.1.023072204812820485121024FALSE
mod-organizations1mod-organizations:1.9.02102489612870088128FALSE
mod-serials-management1mod-serials-management:1.0.02248023121281792384512FALSE
mod-settings1mod-settings:1.0.32102489620076888128FALSE
mod-search1mod-search:3.2.0225922480204814405121024FALSE
edge-dematic1edge-dematic:2.2.01102489612876888128FALSE
mod-data-import1mod-data-import:3.1.01204818442561292384512FALSE
mod-tags1mod-tags:2.2.02102489612876888128FALSE
mod-authtoken1mod-authtoken:2.15.121440115251292288128FALSE
edge-courses1edge-courses:1.4.02102489612876888128FALSE
mod-inventory-update1mod-inventory-update:3.3.02102489612876888128FALSE
mod-notify1mod-notify:3.2.02102489612876888128FALSE
mod-configuration1mod-configuration:5.10.02102489612876888128FALSE
mod-orders-storage1mod-orders-storage:13.7.02102489651270088128FALSE
edge-caiasoft1edge-caiasoft:2.2.02102489612876888128FALSE
mod-login-saml1mod-login-saml:2.8.02102489612876888128FALSE
mod-erm-usage-harvester1mod-erm-usage-harvester:4.5.02102489612876888128FALSE
mod-password-validator1mod-password-validator:3.2.0214401298128768384512FALSE
mod-licenses1mod-licenses:6.0.02248023121281792384512FALSE
mod-gobi1mod-gobi:2.8.02102489612870088128FALSE
mod-bulk-operations1mod-bulk-operations:2.0.023072260010241536384512FALSE
mod-fqm-manager1mod-fqm-manager:2.0.12300026001282048384512TRUE
edge-dcb1edge-dcb:1.1.02102489612876888128FALSE
mod-graphql1mod-graphql:1.12.12102489612876888128FALSE
mod-finance1mod-finance:4.9.02102489612870088128FALSE
mod-erm-usage1mod-erm-usage:4.7.02102489612876888128FALSE
mod-batch-print1mod-batch-print:1.1.02102489612876888128FALSE
mod-copycat1mod-copycat:1.6.02102451212876888128FALSE
mod-lists1mod-lists:2.0.02300026001282048384512FALSE
mod-entities-links1mod-entities-links:3.0.0225922480400144001024FALSE
mod-permissions2mod-permissions:6.5.02168415445121024384512FALSE
pub-edge1pub-edge:2023.06.142102489612876800FALSE
mod-orders1mod-orders:12.8.022048144010241024384512FALSE
edge-patron1edge-patron:5.1.02102489625676888128FALSE
edge-ncip1edge-ncip:1.9.221024896