Data Import (MARC Holdings) with CI/CO [Nolana]

Owned by Artem Vronik
Last updated: Aug 17, 2023 by Martin Tran

It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.

Overview

The target of the test is to investigate the impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations and to compare tested results with direct non-intersecting activities' baselines.

PERF-291 - Getting issue details... STATUS

Summary

  • Test results don't show a huge negative impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations:
    • Response time of CI/CO activities during Data Import - MARC Holdings was increased (degradation) by more than 25%.
    • Duration of Data Import - MARC Holdings with CI/CO activities was decreased by 66%. 
  • High Memory usage was detected for mod-inventory-storage (memory usage is 106%, it's >100%) but stable during the test 
  • Memory usage of mod-source-record-storage was increased from 55% to 58%
  • All modules show expected CPU behaviour and typical workload, except 'mod-source-record-manager' and 'mod-authtoken'
  • During the test, there were detected several requests with more than a 1-second response time. (see Additional Screenshots of graphs or charts)

Recommendations & Jiras

  • During future tests, pay attention to:
    • Memory usage of mod-inventory-storage, mod-source-record-storage
    • CPU usage of mod-source-record-manager, mod-authtoken
  • To discuss Data Import - MARC Holdings' behaviour during Check-In/Check-Out activities Martin Tran Artem Vronik Roman_Fedynyshyn 

Test Runs 

Test #

Test Conditions

Duration 

Load generator size (recommended)Load generator Memory(GiB) (recommended)

1.

8 users CI/CO + DI (1k, 5k, 10k, 80k) MARC Holding60 mint3.medium3

Results

Response Times 

Grafana: http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&from=1672996256462&to=1673000063590


Check-in/Check-outData Import

Average, msMARC Holding

Check-in Controller

Check-out Controller

1k Records5k Records10k Records80k records
Test 1573.79924.19"00:00:35.777""00:01:28.613""00:02:42.095""00:20:12.114"

Comparisons

Test #Report linkType of resultsCheck-In, msCheck-Out, msDI MARC Holding
1k5k10k80k
Test #1 - CI/CO 8 users onlyCheck-in-check-out Test Report (Nolana)[Baseline]456.00698.00xxxx
Test #2 - DI MARC Holding onlyData Import Create MARC holdings records [Nolana][Baseline]xx00:00:32.70000:04:20.80000:03:24.90000:21:22.600

Test #3 - CI/CO + DI MARC Holding

*CI/CO stable

Data Import (MARC Holdings) with CI/CO [Nolana][Tested]

573.79

461.17

924.19

711.94

00:00:35.77700:01:28.61300:02:42.09500:20:12.114
Diff:

+25.83%

*+1.13%

+32.41%

*+1.99%

+9.41%-66.02%-20.89%-5.50%

*Ci/CO stable:

Memory Utilization

mod-inventory-storage memory usage is 106%, it's >100% but stable during test 

mod-source-record-storage memory usage was increased from 55% to 58%

CPU Utilization 

All modules show expected and some workload, except 'mod-source-record-manager' and 'mod-authtoken':

  • mod-source-record-manager: MIN - 2.74%, MAX - 50.87%, AVG - 21.674%
  • mod-authtoken: MIN - 0.33%, MAX - 18.59%, AVG - 5.96%

TOP CPU usage modules:

  • nginx-okapi: MIN - 0.13%, MAX - 72.06%, AVG - 33.54%
  • mod-source-record-storage: MIN - 1.95%, MAX - 67.80%, AVG - 29.83% 
  • mod-inventory: MIN - 6.07%, MAX - 59.05%, AVG - 27.44%

RDS CPU Utilization 

MAX RDS CPU utilization is 49% under 80k DI Marc holding activity. 

Appendix

Infrastructure

PTF -environment ncp3 [ environment name] 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
  • 2 instances of db.r6.xlarge database instances: Writer & reader instances
  • MSK ptf-kakfa-3 [ kafka configurations]
    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameters:

Modules

Version

Task Definition

Running Tasks 

CPU

Memory (Soft/Hard Limits)

MaxMetaspaceSize

Xmx

(tick) mod-inventory19.0.27210242592/28805121814

(tick) mod-inventory-storage

25.0.3321024

1952/2208

5121440
(tick) okapi4.14.7131024

1440/1684

512922
(tick) mod-source-record-storage5.5.24210241440/1536512908

(tick) mod-source-record-manager

3.5.64210243688/40965122048
(tick) mod-data-import2.6.2412561844/20485121292
(tick) mod-data-import-cs1.15.112128896/1024128768
(tick) mod-quick-marc2.5.0311282176/22885121664
(tick) mod-feesfines18.1.132128896/1024128768
(tick) mod-patron-blocks1.7.1421024896/1024128768
(tick) mod-pubsub2.7.0421024

1440/1536

512922
(tick) mod-authtoken2.12.032

512

1152/1440

128

922

(tick) mod-circulation-storage15.0.2321024

1440/1536

512896
(tick) mod-circulation23.3.2321024896/1024128768
(tick) mod-configuration5.9.032128896/1024128768
(tick) mod-users19.0.042128896/1024128768
mod-remote-storage1.7.1321281692/18725121178

Methodology/Approach

  1. DB Refresh
  2. Run a CI/CO circulation test by Jenkins job
  3. Data Import MARC Holding job running with 5 minutes interval

Additional Screenshots of graphs or charts

During the test, there were detected several requests with more than a 1-second response time.