Data Import with Check-ins Check-outs (Orchid)

Owned by Hanna Tuhuzbayeva
Last updated: Jun 21, 2023
5 min read

Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Orchid release to detect performance trends.

Ticket: PERF-472 - Getting issue details... STATUS

Summary

  • There is a significant performance degradation of data import for Orchid in comparison to Nolana results. Response time are about 2 times higher. It might be due to fixing differences in the database schemas. More details...
  • Data import response times are up to 22% higher with parallel Check-in/Check-out than pure Data import results. More details...
  • Check-in/Check-out response times are up to 168% higher with parallel Data import than pure Check-in/Check-out results. More details...
  • There is memory utilization increase observed which is caused by previous modules restarting (everyday cluster shut down process). More details...
  • Average CPU usage did not exceed 130 % for all the modules. Spikes can be observed in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 170%. CPU usage is about 2 times lower than in the same test for Nolana. More details...
  • Average DB CPU usage during data import is about 95%. More details...

Test Runs 

Test #

Scenario

Load levelComment
1DI MARC Bib Create5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)
CICO 8 users
2DI MARC Bib Update5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)100k file was completed with errors (1 item discarded)
CICO 8 users

Test Results

Data import

Profile
MARC File
DI Duration		Check In, Check Out Response time (8 users)
CI Average secCO Average sec
DI MARC Bib Create (PTF - Create 2)5K.mrc00:05:01.7330.9611.442
10K.mrc00:09:06.7521.0581.624
25K.mrc00:24:28.1671.0561.621
50K.mrc00:43:03.7850.9361.519
100K.mrc01:35:50.7490.8681.468
DI MARC Bib Update (PTF - Updates Success - 1)5K.mrc00:04:52.4960.8551.339
10K.mrc00:09:22.7650.9161.398
25K.mrc00:24:02.2380.9221.425
50K.mrc00:47:13.8760.9041.456
100K.mrc01:40:25.5330.8381.415

Check-in/Check-out

ScenarioLoad levelRequestResponse time, sec

95 percaverage
Circulation Check-in/Check-out (without Data import)8 usersCheck-in0.4890.394
Check-out0.7930.724

Comparison

The following table compares test results of current release (Orchid) to the previous release numbers (Nolana) and to the baselines Orchid results (CICO without DI and DI without CICO).

File size


Data import duration

Deviation (compared DI Orchid without CICO and with CICO)


Check-in/Check-out 8 users response time  with Data Import (avg, sec)

Deviation

(compared CICO Orchid without DI and with DI)

without CICOwith CICO (8 users)Check-in Check-out 

Check-in 

Check-out 

Nolana*

Orchid**

Nolana*

Orchid 

Nolana*

Orchid  

Nolana*

Orchid 

5K MARC BIB Create

2 min 51 sec4 min 30 sec1 min 56 sec5 min 01 sec+11%

0.817

0.961

1.417

1.442+143%+96%

5K MARC BIB Update

2 min 27 sec4 min 2 sec2 min 51 sec4 min 52 sec+20%

0.747

0.855

1.094

1.339+117%+84%

10K MARC BIB Create

4 min 55 sec9 min 25 sec3 min 57 sec9 min 06 sec-3%

0.842

1.058

1.574

1.624+168%+124%
10K MARC BIB Update4 min 50 sec8 min 10 sec4 min 57 sec9 min 22 sec+14%

0.541

0.916

1.026

1.398+132%+93%
25K MARC BIB Create11 min 56 sec22 min 16 sec9 min 24 sec24 min 28 sec+9%

0.882

1.056

1.641

1.621+168%+123%
25K MARC BIB Update12 min 20 sec19 min 39 sec13 min 12 sec24 min 2 sec+22%

0.700

0.922

1.248

1.425+134%+96%
50K MARC BIB Create23 min 43 sec39 min 27 sec19 min 28 sec43 min 3 sec+9%

0.926

0.936

1.666

1.519+137%+109%
50K MARC BIB Update24 min 5 sec38 min 30 sec Completed with errors (1 item discarded)27 min 39 sec47 min 13 sec+22%

0.700

0.904

1.199

1.456+129%+101%
100K MARC BIB Create49 min 40 sec1 hour 38 min38 min 44 sec1 hour 35 min-3%

1.021

0.868

1.862

1.468+120%+102%
100K MARC BIB Update51 min 15 sec1 hour 33 min48 min 45 sec1 hour  40 min Completed with errors (1 item discarded)+7%

0.556

0.838

1.046

1.415+112%+95%

* Nolana DI and CICO results are taken from Data Import with Check-ins Check-outs Nolana.

** Orchid DI results are taken from Data Import test report (Orchid).

Detailed CICO response time comparison

Request*Response time (avg, sec)
Pure CICOCICO + 100K MARC BIB CreateCICO + 100K MARC BIB Update
Check-Out Controller0.7241.4701.415
Check-In Controller0.3940.8700.838
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)0.2510.5320.507
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)0.1730.4020.390
GET_circulation/loans (Submit_barcode_checkout)0.1270.2780.276
GET_users (Get_check_in_page)0.0220.0920.093
GET_inventory/items (Submit_barcode_checkin)0.0450.1130.113
GET_inventory/items (Submit_barcode_checkout)0.0480.1120.114
GET_automated-patron-blocks (Submit_patron_barcode)0.0210.0490.047
GET_configurations/entries (Get_check_in_page)0.0130.0440.040
GET_users (Submit_patron_barcode)0.0150.0400.039
GET_circulation/requests_status_openAwaitingPickup (Submit_patron_barcode)0.0180.0350.033

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI Bib Update + CICO

Service CPU Utilization

Average CPU usage did not exceed 130 % for all the modules. Spikes can be observed in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 170%.

CPU usage is about 2 times lower than in the same test for Nolana - Data Import with Check-ins Check-outs Nolana.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

DI MARC BIB Create + CICO

MARC BIB Update + CICO


DB CPU Utilization

Average DB CPU usage during data import is about 95%

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB Connections

Average connection count during data import is about 270 connections.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB load

DI MARC BIB Create + CICO

Top SQL-queries:

update "marc_records_lb" set "content" = cast($1 as jsonb) where "id" = cast($2 as uuid)
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log file:

MARC BIB Update + CICO

Top SQL-queries:

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log file:


Appendix

Infrastructure

Records count:

  • mod_source_record_storage.marc_records_lb = 22618121
  • mod_source_record_storage.raw_records_lb = 22650140
  • mod_source_record_storage.records_lb = 22650140
  • mod_source_record_storage.marc_indexers =  98256911(all records)
  • mod_source_record_storage.marc_indexers with field_no 010 = 139135
  • mod_source_record_storage.marc_indexers with field_no 035 = 4272473
  • mod_inventory_storage.authority = 7402975
  • mod_inventory_storage.holdings_record = 22027125
  • mod_inventory_storage.instance = 20986866
  • mod_inventory_storage.item = 22130108

PTF -environment ncp3 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, one reader, and one writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
  • Kafka topics partitioning: - 2 partitions for DI topics


Modules memory and CPU parameters

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-inventory-storage26.0.0102102422081952
512
1440
mod-inventory20.0.4821024288025925121814
mod-source-record-storage5.6.52422048409636885123076
mod-quick-marc3.0.051128228821765121664
mod-source-record-manager3.6.2162
1024
4096
3688
5123076
mod-di-converter-storage2.0.2521281024896128768
mod-data-import2.7.181256204818445121292
okapi5.0.163102416841440512922
nginx-okapi2022.03.02621281024896--
pub-okapi2022.03.02621281024896-768
mod-feesfines

18.2.1

121281024896128768
mod-patron-blocks1.8.01210241024896128768
mod-pubsub2.7.012102415361440512922
mod-authtoken2.12.01251214401152128922
mod-circulation-storage

15.0.2

12102415361440512896
mod-circulation23.3.21210241024896128768
mod-configuration5.9.0121281024896128768
mod-users19.0.012

128

1024896128768
mod-remote-storage

1.7.1

12

128

1872

1692

512

1178m

Methodology/Approach

To test Baseline for DI JMeter scripts were used with 5 min pauses between the tests.

Additional links

Grafana dashboard:

MARC Bib Create + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1686817177738&to=1686830183685&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

MARC Bib Update + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1686839353607&to=1686853077913&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All