[Nolana] Check-IN + title-level requests

Owned by Hanna Tuhuzbayeva
Last updated: Feb 28, 2023

Overview

Test goal is to assess performance of circulation check-in functionality for items with 10 TLR (title-level requests) each.

Ticket: PERF-450 - Getting issue details... STATUS

Summary

Recommendations & Jiras

Ticket: Investigate performance degradation of Check-in with TLRs

PERF-465 - Getting issue details... STATUS

Test Runs 

Test #

Test Conditions

Duration 

Load generator size (recommended)

Load generator Memory(GiB) (recommended)

Notes


1.

Baseline, Checkin with 1, 8, 25 users

30 mint3.medium3

Without TLR

2.Verification, Checkin with 1, 8, 25 usersWIth 10 TLR per item

Results

Response Times

Baseline (items without TLR)

Verification (items with 10 TLRs)

Response time comparison


Users quantity

Response time 95prc, sec


Degradation, sec


Degradation, %

Baseline

(items without TLR)

Verification

(items with 10 TLR each)

1 user0.6070.9530.34657
8 users0.4840.7820.29862
25 users0.4412.0011.56354

Service CPU Utilization

Baseline (items without TLR)

Verification (items with 10 TLRs)

Service CPU Utilization 25 users comparison

Baseline (items without TLR)

Verification (items with 10 TLRs)



Modules

Average load
BaselineVerification
mod-users22.151.6
mod-circulation-storage3.37.2
mod-inventory-storage920.6
mod-circulation1.52.7
mod-okapi19.920.8

mod-feesfines

7.18.7
mod-patron-blocks0.61.1
mod-pubsub5.85.9
mod-authtoken5.46.3
mod-configuration16.513.2
mod-remote-storage16.419.3
mod-inventory9.611.8
mod-inventory-storage920.6

Instance CPU Utilization

Baseline (items without TLR)

Verification (items with 10 TLRs)

Memory Utilization

Baseline (items without TLR)

Verification (items with 10 TLRs)

RDS CPU Utilization

Baseline (items without TLR)

Verification (items with 10 TLRs)

RDS CPU Utilization 25 users comparison

Baseline (items without TLR)

Verification (items with 10 TLRs)

RDS DB connections

Baseline (items without TLR)

Verification (items with 10 TLRs)

Database Load

Baseline (items without TLR)

Verification (items with 10 TLRs)

Database Load 25 users comparison

Baseline (items without TLR)

Verification (items with 10 TLRs)

Top SQL 25 users comparison

Baseline (items without TLR)

Verification (items with 10 TLRs)

Appendix

Infrastructure

PTF -environment ncp3

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 2 instances of db.r6.xlarge database instances: Writer & reader instances
  • MSK ptf-kakfa-3 [ kafka configurations]
    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameters:

Modules

Version

Task Definition

Running Tasks 

CPU

Memory (Soft/Hard limits)

MaxMetaspaceSize

Xmx

okapi4.14.7131024

1440/1684

512922
mod-feesfines18.1.132128896/1024128768
mod-patron-blocks1.7.1421024896/1024128768
mod-pubsub2.7.0421024

1440/1536

512922
mod-authtoken2.12.032

512

1152/1440

128

922

mod-circulation-storage15.0.2321024

1440/1536

512896
mod-circulation23.3.2321024896/1024128768
mod-configuration5.9.032128896/1024128768
mod-users19.0.042128896/1024128768
mod-remote-storage1.7.1321281692/18725121178

Methodology/Approach

  1. Run necessary commands to return the database to the initial state. Do this before each test run. Wait several minutes before the test start.
  2. Conduct check-out for the items with JMeter script Create_TLR.jmx (disable "Create_TLR" step).
  3. Conduct baseline - run check-in load tests with different number of users.
  4. Conduct verification - repeat tests with the same approach but before each test also generate 10 TLR for each item by running JMeter script (Create_TLR.jmx) - enable both Check-in and Create_TLR steps.
  5. Compare test results.

Note - make sure to use the same list of items for Create_TLR.jmx script and Check-in script. Also, items should be selected for those instances which have 1 item per instance.

Grafana dashboard

Baseline (items without TLR): http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676273687683&to=1676283165261&var-percentile=95&var-test_type=baseline&var-test=circulation_checkIn_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Verification (items with 10 TLRs): 

1 user, 8 users

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676357540020&to=1676363791300&var-percentile=95&var-test_type=baseline&var-test=circulation_checkIn_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

25 ysers

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1676410942613&to=1676412905924&var-percentile=95&var-test_type=baseline&var-test=circulation_checkIn_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Please note that dashboards will expire in 6 weeks since test run.