/
Data Import (MARC Holdings) with CI/CO [Nolana]

Data Import (MARC Holdings) with CI/CO [Nolana]

Aug 17, 2023

It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.

Overview

The target of the test is to investigate the impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations and to compare tested results with direct non-intersecting activities' baselines.

PERF-291: Test Data Import (MARC Holdings) with CICOClosed

Summary

  • Test results don't show a huge negative impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations:

    • Response time of CI/CO activities during Data Import - MARC Holdings was increased (degradation) by more than 25%.

    • Duration of Data Import - MARC Holdings with CI/CO activities was decreased by 66%. 

  • High Memory usage was detected for mod-inventory-storage (memory usage is 106%, it's >100%) but stable during the test 

  • Memory usage of mod-source-record-storage was increased from 55% to 58%

  • All modules show expected CPU behaviour and typical workload, except 'mod-source-record-manager' and 'mod-authtoken'

  • During the test, there were detected several requests with more than a 1-second response time. (see Additional Screenshots of graphs or charts)

Recommendations & Jiras

  • During future tests, pay attention to:

    • Memory usage of mod-inventory-storage, mod-source-record-storage

    • CPU usage of mod-source-record-manager, mod-authtoken

  • To discuss Data Import - MARC Holdings' behaviour during Check-In/Check-Out activities @Martin Tran @Artem Vronik @Roman_Fedynyshyn 

Test Runs 

Test #

Test Conditions

Duration 

Load generator size (recommended)

Load generator Memory(GiB) (recommended)

1.

8 users CI/CO + DI (1k, 5k, 10k, 80k) MARC Holding

60 min

t3.medium

3

Results

Response Times 

Grafana: http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&from=1672996256462&to=1673000063590

 

Check-in/Check-out

Data Import

 

Check-in/Check-out

Data Import

 

Average, ms

MARC Holding

 

Check-in Controller

Check-out Controller

1k Records

5k Records

10k Records

80k records

Test 1

573.79

924.19

"00:00:35.777"

"00:01:28.613"

"00:02:42.095"

"00:20:12.114"

Comparisons

Test #

Report link

Type of results

Check-In, ms

Check-Out, ms

DI MARC Holding

1k

5k

10k

80k

Test #1 - CI/CO 8 users only

Check-in-check-out Test Report (Nolana)

[Baseline]

456.00

698.00

x

x

x

x

Test #2 - DI MARC Holding only

Data Import Create MARC holdings records [Nolana]

[Baseline]

x

x

00:00:32.700

00:04:20.800

00:03:24.900

00:21:22.600

Test #3 - CI/CO + DI MARC Holding

*CI/CO stable

Data Import (MARC Holdings) with CI/CO [Nolana]

[Tested]

573.79

461.17

924.19

711.94

00:00:35.777

00:01:28.613

00:02:42.095

00:20:12.114

Diff:

+25.83%

*+1.13%

+32.41%

*+1.99%

+9.41%

-66.02%

-20.89%

-5.50%

*Ci/CO stable:

Memory Utilization

mod-inventory-storage memory usage is 106%, it's >100% but stable during test 

mod-source-record-storage memory usage was increased from 55% to 58%

CPU Utilization 

All modules show expected and some workload, except 'mod-source-record-manager' and 'mod-authtoken':

  • mod-source-record-manager: MIN - 2.74%, MAX - 50.87%, AVG - 21.674%

  • mod-authtoken: MIN - 0.33%, MAX - 18.59%, AVG - 5.96%

TOP CPU usage modules:

  • nginx-okapi: MIN - 0.13%, MAX - 72.06%, AVG - 33.54%

  • mod-source-record-storage: MIN - 1.95%, MAX - 67.80%, AVG - 29.83% 

  • mod-inventory: MIN - 6.07%, MAX - 59.05%, AVG - 27.44%

RDS CPU Utilization 

MAX RDS CPU utilization is 49% under 80k DI Marc holding activity. 

Appendix

Infrastructure

PTF -environment ncp3 [ environment name] 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 2 instances of db.r6.xlarge database instances: Writer & reader instances

  • MSK ptf-kakfa-3 [ kafka configurations]

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Modules memory and CPU parameters:

Modules

Version

Task Definition

Running Tasks 

CPU

Memory (Soft/Hard Limits)

MaxMetaspaceSize

Xmx

Modules

Version

Task Definition

Running Tasks 

CPU

Memory (Soft/Hard Limits)

MaxMetaspaceSize

Xmx

mod-inventory

19.0.2

7

2

1024

2592/2880

512

1814

mod-inventory-storage

25.0.3

3

2

1024

1952/2208

512

1440

okapi

4.14.7

1

3

1024

1440/1684

512

922

mod-source-record-storage

5.5.2

4

2

1024

1440/1536

512

908

mod-source-record-manager

3.5.6

4

2

1024

3688/4096

512

2048

mod-data-import

2.6.2

4

1

256

1844/2048

512

1292

mod-data-import-cs

1.15.1

1

2

128

896/1024

128

768

mod-quick-marc

2.5.0

3

1

128

2176/2288

512

1664

mod-feesfines

18.1.1

3

2

128

896/1024

128

768

mod-patron-blocks

1.7.1

4

2

1024

896/1024

128

768

mod-pubsub

2.7.0

4

2

1024

1440/1536

512

922

mod-authtoken

2.12.0

3

2

512

1152/1440

128

922

mod-circulation-storage

15.0.2

3

2

1024

1440/1536

512

896

mod-circulation

23.3.2

3

2

1024

896/1024

128

768

mod-configuration

5.9.0

3

2

128

896/1024

128

768

mod-users

19.0.0

4

2

128

896/1024

128

768

mod-remote-storage

1.7.1

3

2

128

1692/1872

512

1178

Methodology/Approach

  1. DB Refresh

  2. Run a CI/CO circulation test by Jenkins job

  3. Data Import MARC Holding job running with 5 minutes interval

Additional Screenshots of graphs or charts

During the test, there were detected several requests with more than a 1-second response time.