Data Import MARC BIB + Check-in-check-out with DB R/W Split enabled and disabled (Nolana) 2022-12-15

Owned by Olga Kondratenko
Last updated: Dec 29, 2022
6 min read
20-users testsAvgMax

mod-users21%21%

mod-pubsub5%5%

okapi14%14%

mod-circulation4%4%

mod-circulation-storage5%5%

mod-inventory7%7%

mod-inventory-storage7%7%

mod-patron-blocks1%1%

mod-feesfines17%17%

mod-authtoken31%20-users testsAvgMax


mod-users21%21%


mod-pubsub5%5%


okapi14%14%


mod-circulation4%4%


mod-circulation-storage5%5%


mod-inventory7%7%


mod-inventory-storage7%7%


mod-patron-blocks1%1%


mod-feesfines17%17%


mod-authtoken31%78%















Overview

This is a report for a series of Check-in-check-out tests (CI/CO) + Data Import (DI) tests run against the Nolana release and with DB R/W Split enabled. 

Infrastructure

PTF -environment ncp3

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-inventory19.0.112102428802592512m1814m

mod-inventory-storage

25.0.11-221024

2208

(1872 in MG)

1952

(1684 in MG)

512m1440m
okapi4.14.71-231024

1684

(1512 in MG)

1440

(1360 in MG)

512m922m
mod-feesfines18.1.01-221281024896128768
mod-patron-blocks1.7.11-2210241024896128768
mod-pubsub2.7.01-221024

1536

(1440 in MG)

1440

(1296 in MG)

512922
mod-authtoken2.12.01-22

512

(128 in MG)

1440

(1024 in MG)

1152

(896 in MG)

128

922

(768 in MG)

mod-circulation-storage15.0.01-221024

1536

(1152 in MG)

1440

(1024 in MG)

512896
mod-circulation23.3.01210241024896128768
mod-configuration5.9.01-221281024896128m768m
mod-users19.0.01-222581024896128m768m
mod-remote-storage1.7.01-2212818721684512m1178m
mod-data-import-cs1.15.11-222581024896128m768m
mod-quick-marc2.5.01-21128

2288

(2098 in MG)

2176

(1920 in MG)

512m1664m
mod-source-record-storage5.5.21-221024

1536

(1440 in MG)

1440

(1296 in MG)

512m908m
mod-source-record-manager3.5.42-321024

4096

(2048 in MG)

3688

(1844 in MG)

512m

(800M in MG)

2048m

(1024m in MG)

mod-data-import2.6.11-2125620481844512m1292m

MG- Morning Glory release

Front End:

  • Item Check-in (folio_checkin-7.2.0)
  • Item Check-out (folio_checkout-8.2.0)


Modules that had R/W split enabled

1

mod-inventory-storage

2mod-feesfines
3mod-patron-blocks
4mod-pubsub
5mod-authtoken
6mod-circulation-storage
7mod-configuration
8mod-users
9mod-remote-storage
10mod-data-import-cs
11mod-quick-marc
12mod-source-record-storage
13mod-source-record-manager
14mod-data-import

High-Level Summary

  • Response times are better for DB R/W Split enabled compared to disabled, especially for 20 users of the CI/CO with the DI MARC BIB Create test, for both CI/CO and DI as well:
    1. For 20 users CI/CO with DI MARC BIB Create with R/W Split enabled response time is 50% better compared to R/W Split disabled.
    2. For 8 users CI/CO with DI MARC BIB Update with R/W Split enabled response time is 20% better compared to R/W Split disabled.
    3. For DI MARC BIB Create with CI/CO 20 users (19 min 50 sec) for R/W Split enabled response time is 12,7% better compared to R/W Split disabled (22 min 40 sec).
    4. For DI MARC BIB Update with CI/CO 8 users  (22min) for R/W Split enabled response time is 12% better compared to R/W Split disabled (25 min).
    5. For DI MARC BIB Create with CI/CO 20 users (19 min 50 sec) for R/W Split enabled response time is about 16.6% better compared to baseline DI MARC BIB Create (without CI/CO )(23 min 43 s).
    6. For DI MARC BIB Update with CI/CO 8 users (22min) for R/W Split enabled response time is about 8.7% better compared to baseline DI MARC BIB Update (without CI/CO )(24 min 5 s).
  • When DB R/W split is enabled:
    1. requests to the database are separated between 2 instances of DB
    2. the database uses more connections in total
    3. writer instance uses less CPU (load distribution between reader and writer and reader)
  • Services' memory utilization increases during the test run. Probably it is a result of the modules' restart right before the tests.
  • Modules and instances of CPU usage are similar for tests with DB R/W Split enabled and disabled.

Test Runs

Test #

CI/CO

Virtual Users

CI/CO

Duration

Data ImportData Import profileR/W Split

1.

20 users30 mins50K MARC BIB CreatePTF - Create 2disabled

2.

20 users30 mins50K MARC BIB CreatePTF - Create 2enabled
3.8 users30 mins50K MARC BIB UpdatePTF - Updates Success - 1disabled
4.8 users30 mins50K MARC BIB UpdatePTF - Updates Success - 1enabled

Results

Response Times (Average of CI/CO time with MARC BIB) 




Average (seconds)50th %tile (seconds)75th %tile (seconds)95th %tile  (seconds)
Test #Short test descriptionDI timeCheck-inCheck-outCheck-inCheck-outCheck-inCheck-outCheck-inCheck-out

1.

MARC BIB Create (R/W disabled) + CI/CO

22 min 40 sec1.0151.7460.9561.6671.0961.8161.5692.448
2.MARC BIB Create (R/W enabled) + CI/CO r/w enabled19 min 50 sec0.4840.8770.4530.8220.4910.8690.5640.966
3.MARC BIB Update (R/W disabled)+ CI/CO25 min0.5781.0700.5230.9850.5771.0650.6891.250
4.MARC BIB Update (R/W enabled)+ CI/CO r/w enabled22min0.4630.8340.4310.7990.4660.8450.5370.994

Response times are better for DB R/W Split enabled compared to disabled, especially for 20 users of the CI/CO with the DI MARC BIB Create test.

Comparisons

The following tables compare Nolana's test results of CI/CO + DI MARC BIB for tests with DB R/W Split enabled and disabled. For 20 users CI/CO with DI MARC BIB Create with R/W Split enabled response time is 50% better compared to R/W Split disabled. For 8 users CI/CO with DI MARC BIB Update with R/W Split enabled response time is 20% better compared to R/W Split disabled. For DI MARC BIB Update (with CI/CO 8 and 20 users) with R/W Split enabled response time is 12% better compared to R/W Split disabled





Average (seconds)50th %tile (seconds)75th %tile (seconds)95th %tile  (seconds)
Test #Short test descriptionDI timeCompared to MARC BIB baseline (without CI/CO)Check-inCompared to CI/CO baseline (without DI)Check-outCompared to CI/CO baseline  (without DI)Check-inCompared to CI/CO baseline  (without DI)Check-outCompared to CI/CO baseline (without DI)Check-inCompared to CI/CO baseline (without DI)Check-outCompared to CI/CO baseline (without DI)Check-inCompared to CI/CO baseline (without DI)Check-outCompared to CI/CO baseline (without DI)

1.

MARC BIB Create (R/W disabled) + CI/CO

22 min 40 sec4.4%1.015-138%1.746-165%0.956-130%1.667-161%1.096-154%1.816-174%1.569-220%2.448-222%
2.MARC BIB Create (R/W enabled) + CI/CO r/w enabled19 min 50 sec16.6%0.484-13%0.877-33%0.453-9%0.822-28%0.491-14%0.869-31%0.564-15%0.966-27%
Comparison R/W enabled and disabled12.7%
52.3%
49.7%
52.6%
50.6%
55.2%

52.1%


64%
60.5%
3.MARC BIB Update (R/W disabled)+ CI/CO25 min-3.73%0.578-26%1.070-53%0.523-18%0.985-46%0.577-22%1.065-50%0.689-29%1.250-48%
4.MARC BIB Update (R/W enabled)+ CI/CO r/w enabled22min8.7%0.463-1.5%0.834-19%0.4312.2%0.799-18%0.4661%0.845-19%0.537-0.5%0.994-18%
Comparison R/W enabled and disabled12%
19.9%
22%
17.5%
18.9%
19.2%
20.6%
22%
20.4%
5.CI/CO baseline 8 users (without DI)--0.456
0.698
0.441
0.672
0.471
0.709
0.534
0.839
6.CI/CO baseline 20 users (without DI)--0.425
0.658
0.414
0.638
0.430
0.661
0.489
0.758
7.MARC BIB Create baseline (without CI/CO)23 min 43 s--
-
-
-
-
-
-
-
8.MARC BIB Update baseline (without CI/CO)24 min 5 s--
-
-
-
-
-
-
-

Response Time Comparison Graph

MARC BIB Create

MARC BIB Update

Memory Utilization

Services' memory utilization increases during the test run. Probably it is a result of the modules' restart.

MARC BIB Create


Modules CPUs Utilization

MARC BIB Create

MARC BIB Update

Database and network

MARC BIB Create




MARC BIB Update

Database connections 

MARC BIB Create



MARC BIB Update

SELECT queries DB writer instance while R/W split enabled:

  • SELECT save_event_and_decrease_flow_control_counter($1, $2)
  • select max("records_lb"."generation") as "generation" from "records_lb" join "snapshots_lb" on "records_lb"."snapshot_id" = "snapshots_lb"."id" where ("records_lb"."matched_id" = cast($1 as uuid) and "snapshots_lb"."status" in ($2::"job_execution_status", $3::"job_execution_status", $4::"job_execution_status") and "snapshots_lb"."updated_date" < (select "snapshots_lb"."processing_started_date" from "snapshots_lb" where "snapshots_lb"."id" = cast($5 as uuid)))
  • SELECT item.jsonb::text, holdings_record.jsonb::text FROM fs09000000_mod_inventory_storage.item LEFT JOIN fs09000000_mod_inventory_storage.holdings_record  ON holdings_record.id = $2 WHERE item.id = $1
  • select "records_lb"."id", "records_lb"."snapshot_id", "records_lb"."matched_id", "records_lb"."generation", "records_lb"."record_type", "records_lb"."external_id", "records_lb"."state", "records_lb"."leader_record_status", "records_lb"."order", "records_lb"."suppress_discovery", "records_lb"."created_by_user_id", "records_lb"."created_date", "records_lb"."updated_by_user_id", "records_lb"."updated_date", "records_lb"."external_hrid" from "records_lb" where ("records_lb"."matched_id" = cast($1 as
  • select "id", "content" from "marc_records_lb" where "id" = cast($1 as uuid)
  • select "snapshots_lb"."id", "snapshots_lb"."status", "snapshots_lb"."processing_started_date", "snapshots_lb"."created_by_user_id", "snapshots_lb"."created_date", "snapshots_lb"."updated_by_user_id", "snapshots_lb"."updated_date" from "snapshots_lb" where "snapshots_lb"."id" = cast($1 as uuid)
  • select "raw_records_lb"."id", "raw_records_lb"."content" from "raw_records_lb" where "raw_records_lb"."id" = cast($1 as uuid)
  • SELECT jsonb FROM fs09000000_mod_inventory_storage.item WHERE holdingsRecordId = ?

Miscellaneous