/
[ECS] Check-in-check-out Test Report (Quesnelia)

[ECS] Check-in-check-out Test Report (Quesnelia)

Apr 15, 2024

Overview

The test's target is to investigate the behavior of the Check-In/Check-Out activities during fixed load tests.

https://folio-org.atlassian.net/browse/PERF-857

Summary

Check-In/Check-Out tests on Quesnelia were carried out with 8, 20, 30, and 75 virtual users for 45 minutes and 30 virtual users for 24 hours to detect any trends in resource utilization (only 16 hours of results available).

45-minute tests

  1. Response times in tests with 8, 20, and 30 virtual users didn't differ a lot but the lowest response times were with 20 virtual users. Average Check-In - 441 ms, average Check-Out - 893 ms

  2. The highest response time results were with 75 virtual users. Average Check-In - 546 ms, Check-Out - 1176 ms

  3. Reports: , , ,

Longevity test

  • Average Check-In - 458 ms, Check-Out - 1258 ms (the results from the test 16 hours)

  • Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

  • RDS CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

  • Grafana results

Resources

  • CPU utilization

    • 45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.

  • Memory consumption

    • Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

  • Average RDS CPU utilization

    • 8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

  • CPU (User) usage by broker

    • Common CPU utilization by the broker during all tests was up to 12% with equal distribution between brokers

Common notes

  • Longevity test revealed Database CPU Utilization growing trend and CI and CO response time increasing.

Recommendations

  • Add more CPU to nginx-okapi to exclude module fluctuations

Test Runs 

The following table contains test configuration information

Test #

Test artefact

vUsers

Ramp-up, sec

Duration, sec 

1

circulation_checkInCheckOut_qcon

8

80

2700

2

 

20

200

2700

3

 

30

300

2700

4

 

75

750

2700

5

 

30

300

86400

Results

Errors:

  • Error messages: Response 401 due to changed login process and token expiration time 10 min (0.04%).

Response time

The table contains results of Check-in, Check-out tests in the Quesnelia release. 

45-minute tests

 

8 vUsers  

20 vUsers  

30 vUsers  

75 vUsers  

 

8 vUsers  

20 vUsers  

30 vUsers  

75 vUsers  

Requests

Response Times (ms)

 Response Times (ms)

 Response Times (ms)

 Response Times (ms)

Label

#Samples

Average

95th pct

#Samples

Average

95th pct

#Samples

Average

95th pct

#Samples

Average

95th pct

Check-Out Controller

1815

910.51

1102.40

4324

893.26

1054.00

6626

899.13

1058.00

10986

1176.40

1569.00

Check-In Controller

1391

476.67

598.80

3527

441.99

551.60

4911

435.40

544.00

14564

546.74

760.00

POST_circulation/check-out-by-barcode (Submit_barcode_checkout)

1798

275.86

340.00

4275

268.25

315.00

6556

268.31

316.00

14434

331.68

491.00

POST_circulation/check-in-by-barcode (Submit_barcode_checkin)

1388

243.06

313.55

3518

214.64

275.00

4906

205.53

267.00

11159

233.01

348.00

GET_circulation/loans (Submit_patron_barcode)

1805

143.41

173.00

4292

141.66

160.00

6586

140.52

161.00

14493

177.03

273.00

GET_circulation/loans (Submit_barcode_checkout)

1795

140.59

163.20

4275

139.57

158.00

6550

140.00

159.00

14418

174.65

265.00

Longevity test

 30 vUsers Longevity 

 

 

 

Requests

Samples, Response Times 

 

 

Label

#Samples

95th pct

Average

Check-Out Controller

144129

1.998

1.258

Check-In Controller

108447

0.645

0.458

POST_circulation/check-out-by-barcode (Submit_barcode_checkout)

142488

0.613

0.383

POST_circulation/check-in-by-barcode (Submit_barcode_checkin)

107907

0.301

0.200

Resources Utilization

CPU Utilization

45-minute and longevity tests consumed CPU in correspondence to the number of vUsers but some modules spiked during the 75 vUsers test - nginx-okapi - 423%, mod-users-b - 105%.

45 minute tests

Longevity test

Memory Consumption

45-minute tests didn't reveal some problems with memory usage by modules. Longevity tests show the growth of memory usage of mod-agreements from 73 to 80%. No memory leaks were suspected during the longevity test for all other modules.

45-minute tests

Longevity test

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 13%, 20 vUsers - 18%, 30 vUsers - 23%, 75 vUsers - 42% During longevity test CPU grew from 30% to 55% on average. So it has a growing trend during longevity test and spikes up to 96% every 30 min corresponding to the circulation job.

45-minute tests

Longevity test

RDS Database Connections

For both 45-minute and longevity tests RDS used max 615-630 connections. Without test it was 550 connections.

45-minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 12% with equal distribution between brokers

45 minute tests

Longevity test

Appendix

Infrastructure

PTF -environment qcon

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • MSK - tenant

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

DB table records size:

  • fs09000000

    • instances -            24493326

    • items -                  25662862

    • holding records - 24459683

Modules

Module qcon-pvt Task Def. Revision Module Version Task Count Mem Hard Limit Mem Soft limit CPU units Xmx MetaspaceSize MaxMetaspaceSize R/W split enabled mod-agreements 1 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 FALSE mod-authtoken 1 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 FALSE mod-bulk-operations 1 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 FALSE mod-calendar 1 mod-calendar:3.1.0 2 1024 896 128 768 88 128 FALSE mod-circulation 2 mod-circulation:24.2.1 2 2880 2592 1536 1814 384 512 FALSE mod-circulation-item 1 mod-circulation-item:1.0.0 2 1024 896 128 0 0 0 FALSE mod-circulation-storage 1 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 FALSE mod-configuration 1 mod-configuration:5.10.0 2 1024 896 128 768 88 128 FALSE mod-dcb 1 mod-dcb:1.1.0 2 1024 896 128 768 88 128 FALSE mod-feesfines 1 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 FALSE mod-inventory 1 mod-inventory:20.2.0 2 2880 2592 1024 1814 384 512 FALSE mod-inventory-storage 1 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 FALSE mod-login 1 mod-login:7.11.0 2 1440 1298 1024 768 384 512 FALSE mod-notes 1 mod-notes:5.2.0 2 1024 896 128 952 384 512 FALSE mod-patron-blocks 1 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 FALSE mod-pubsub 1 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 FALSE mod-remote-storage 1 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 FALSE mod-search 1 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 FALSE mod-users 1 mod-users:19.3.1 2 1024 896 128 768 88 128 FALSE nginx-okapi 1 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 FALSE okapi-b 1 okapi:5.3.0 3 1684 1440 1024 922 384 512 FALSE pub-edge 1 pub-edge:2023.06.14 2 1024 896 128 768 0 0 FALSE pub-okapi 1 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 FALSE
Module Task Def. Revision Module Version Task Count Mem Hard Limit Mem Soft limit CPU units Xmx MetaspaceSize MaxMetaspaceSize R/W split enabled qcon-pvt Fri Apr 05 15:19:05 UTC 2024 mod-remote-storage 1 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 FALSE mod-finance-storage 1 mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 FALSE mod-ncip 1 mod-ncip:1.14.4 2 1024 896 128 768 88 128 FALSE mod-agreements 1 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 FALSE mod-ebsconet 1 mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 FALSE mod-consortia 1 mod-consortia:1.1.0 2 3072 2048 128 2048 512 1024 FALSE mod-organizations 1 mod-organizations:1.9.0 2 1024 896 128 700 88 128 FALSE mod-serials-management 1 mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 FALSE mod-settings 1 mod-settings:1.0.3 2 1024 896 200 768 88 128 FALSE mod-search 1 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 FALSE edge-dematic 1 edge-dematic:2.2.0 1 1024 896 128 768 88 128 FALSE mod-data-import 1 mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 FALSE mod-tags 1 mod-tags:2.2.0 2 1024 896 128 768 88 128 FALSE mod-authtoken 1 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 FALSE edge-courses 1 edge-courses:1.4.0 2 1024 896 128 768 88 128 FALSE mod-inventory-update 1 mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 FALSE mod-notify 1 mod-notify:3.2.0 2 1024 896 128 768 88 128 FALSE mod-configuration 1 mod-configuration:5.10.0 2 1024 896 128 768 88 128 FALSE mod-orders-storage 1 mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 FALSE edge-caiasoft 1 edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 FALSE mod-login-saml 1 mod-login-saml:2.8.0 2 1024 896 128 768 88 128 FALSE mod-erm-usage-harvester 1 mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 FALSE mod-password-validator 1 mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 FALSE mod-licenses 1 mod-licenses:6.0.0 2 2480 2312 128 1792 384 512 FALSE mod-gobi 1 mod-gobi:2.8.0 2 1024 896 128 700 88 128 FALSE mod-bulk-operations 1 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 FALSE mod-fqm-manager 1 mod-fqm-manager:2.0.1 2 3000 2600 128 2048 384 512 TRUE edge-dcb 1 edge-dcb:1.1.0 2 1024 896 128 768 88 128 FALSE mod-graphql 1 mod-graphql:1.12.1 2 1024 896 128 768 88 128 FALSE mod-finance 1 mod-finance:4.9.0 2 1024 896 128 700 88 128 FALSE mod-erm-usage 1 mod-erm-usage:4.7.0 2 1024 896 128 768 88 128 FALSE mod-batch-print 1 mod-batch-print:1.1.0 2 1024 896 128 768 88 128 FALSE mod-copycat 1 mod-copycat:1.6.0 2 1024 512 128 768 88 128 FALSE mod-lists 1 mod-lists:2.0.0 2 3000 2600 128 2048 384 512 FALSE mod-entities-links 1 mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 FALSE mod-permissions 2 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 FALSE pub-edge 1 pub-edge:2023.06.14 2 1024 896 128 768 0 0 FALSE mod-orders 1 mod-orders:12.8.0 2 2048 1440 1024 1024 384 512 FALSE edge-patron 1 edge-patron:5.1.0 2 1024 896 256 768 88 128 FALSE edge-ncip 1 edge-ncip:1.9.2 2 1024 896 128 768 88 128 FALSE mod-users-bl 1 mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 FALSE mod-invoice 1 mod-invoice:5.8.0 2 1440 1152 512 922 88 128 FALSE mod-inventory-storage 1 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 FALSE mod-user-import 1 mod-user-import:3.8.0 2 1024 896 128 768 88 128 FALSE mod-sender 1 mod-sender:1.12.0 2 1024 896 128 768 88 128 FALSE edge-oai-pmh 1 edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 FALSE mod-data-export-worker 1 mod-data-export-worker:3.2.1 2 3072 2048 1024 2048 384 512 FALSE mod-rtac 1 mod-rtac:3.6.0 2 1024 896 128 768 88 128 FALSE mod-circulation-storage 1 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 FALSE mod-source-record-storage 1 mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 FALSE mod-calendar 1 mod-calendar:3.1.0 2 1024 896 128 768 88 128 FALSE mod-event-config 1 mod-event-config:2.7.0 2 1024 896 128 768 88 128 FALSE mod-courses 1 mod-courses:1.4.10 2 1024 896 128 768 88 128 FALSE mod-circulation-item 1 mod-circulation-item:1.0.0 2 1024 896 128 0 0 0 FALSE mod-inventory 1 mod-inventory:20.2.0 2 2880 2592 1024 1814 384 512 FALSE mod-email 1 mod-email:1.17.0 2 1024 896 128 768 88 128 FALSE mod-circulation 2 mod-circulation:24.2.1 2 2880 2592 1536 1814 384 512 FALSE mod-pubsub 1 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 FALSE mod-di-converter-storage 1 mod-di-converter-storage:2.2.0 2 1024 896 128 768 88 128 FALSE edge-orders 1 edge-orders:3.0.0 2 1024 896 128 768 88 128 FALSE edge-rtac 1 edge-rtac:2.7.1 2 1024 896 128 768 88 128 FALSE mod-users 1 mod-users:19.3.1 2 1024 896 128 768 88 128 FALSE mod-template-engine 1 mod-template-engine:1.20.0 2 1024 896 128 768 88 128 FALSE mod-patron-blocks 1 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 FALSE mod-audit 1 mod-audit:2.9.0 2 1024 896 128 768 88 128 FALSE edge-fqm 1 edge-fqm:2.0.0 2 1024 896 128 768 88 128 FALSE mod-source-record-manager 1 mod-source-record-manager:3.8.0 2 5600 5000 2048 3500 384 512 FALSE nginx-edge 1 nginx-edge:2023.06.14 2 1024 896 128 0 0 0 FALSE mod-quick-marc 1 mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 FALSE nginx-okapi 1 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 FALSE okapi-b 1 okapi:5.3.0 3 1684 1440 1024 922 384 512 FALSE mod-feesfines 1 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 FALSE mod-invoice-storage 1 mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 FALSE mod-service-interaction 1 mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 FALSE mod-dcb 1 mod-dcb:1.1.0 2 1024 896 128 768 88 128 FALSE mod-patron 1 mod-patron:6.1.0 2 1024 896 128 768 88 128 FALSE mod-data-export 1 mod-data-export:5.0.0 1 2048 1524 1024 0 0 0 FALSE mod-oai-pmh 1 mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 FALSE edge-connexion 1 edge-connexion:1.2.0 2 1024 896 128 768 88 128 FALSE mod-notes 1 mod-notes:5.2.0 2 1024 896 128 952 384 512 FALSE mod-kb-ebsco-java 1 mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 FALSE mod-login 1 mod-login:7.11.0 2 1440 1298 1024 768 384 512 FALSE mod-organizations-storage 1 mod-organizations-storage:4.7.0 2 1024 896 128 700 88 128 FALSE mod-data-export-spring 1 mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 FALSE pub-okapi 1 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 FALSE mod-eusage-reports 1 mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128 FALSE

 

Methodology/Approach

  • Run data preparation script before each CI/CO test

  • Update .jmx file script for Quesnelia release

  • Test performed from load generator in the same region US East (N. Virginia)us-east-1 

  • Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each. 

  • Test CI/CO with 30 users for 24 hours to detect any trends in memory (only 16 hours results available).