Versions Compared

Version Old Version 20 New Version Current
Changes made by

Hanna Tuhuzbayeva

Hanna Tuhuzbayeva

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Overview

...

Ticket:

Jira Legacy
serverSystem JiraJIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-472

...

  • There is a significant performance degradation of data import for Orchid in comparison to Nolana results. Response time are about 2 times higher. Probabbly it's It might be due to fixing differences in the database schemas. More details...
  • Data import response times are up to 22% higher with parallel Check-in/Check-out than pure Data import results. More details...
  • Check-in/Check-out response times are up to 168% higher with parallel Data import than pure Check-in/Check-out results. More details...
  • There is memory utilization increase observed which is caused by previous modules restarting (everyday cluster shut down process). More details...
  • Average CPU usage did not exceed 130 % for all the modules. Spikes can be observed in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 170%. CPU usage is about 2 times lower than in the same test for Nolana. More details...
  • Average DB CPU usage during data import is about 95%. More details...

Test Runs 

Test #

Scenario

Load levelComment
1DI MARC Bib Create5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)
CICO 8 users
2DI MARC Bib Update5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)100k file was completed with errors (1 item discarded)
CICO 8 users

...

The following table compares test results of current release (Orchid) to the previous release numbers (Nolana) and to the baselines Orchid results (CICO withoud without DI and DI without CICO).

File size


Data import duration

Deviation (compared DI Orchid without CICO and with CICO)


Check-in/Check-out 8 users response time  with Data Import (avg, sec)

Deviation

(compared CICO Orchid without DI and with DI)

without CICOwith CICO (8 users)Check-in Check-out 

Check-in 

Check-out 

Nolana*

Orchid**

Nolana*

Orchid 

Nolana*

Orchid  

Nolana*

Orchid 

5K MARC BIB Create

2 min 51 sec4 min 30 sec1 min 56 sec5 min 01 sec+11%

0.817

0.961

1.417

1.442+143%+96%

5K MARC BIB Update

2 min 27 sec4 min 2 sec2 min 51 sec4 min 52 sec+20%

0.747

0.855

1.094

1.339+117%+84%

10K MARC BIB Create

4 min 55 sec9 min 25 sec3 min 57 sec9 min 06 sec-3%

0.842

1.058

1.574

1.624+168%+124%
10K MARC BIB Update4 min 50 sec8 min 10 sec4 min 57 sec9 min 22 sec+14%

0.541

0.916

1.026

1.398+132%+93%
25K MARC BIB Create11 min 56 sec22 min 16 sec9 min 24 sec24 min 28 sec+9%

0.882

1.056

1.641

1.621+168%+123%
25K MARC BIB Update12 min 20 sec19 min 39 sec13 min 12 sec24 min 2 sec+22%

0.700

0.922

1.248

1.425+134%+96%
50K MARC BIB Create23 min 43 sec39 min 27 sec19 min 28 sec43 min 3 sec+9%

0.926

0.936

1.666

1.519+137%+109%
50K MARC BIB Update24 min 5 sec38 min 30 sec Completed with errors (1 item discarded)27 min 39 sec47 min 13 sec+22%

0.700

0.904

1.199

1.456+129%+101%
100K MARC BIB Create49 min 40 sec1 hour 38 min38 min 44 sec1 hour 35 min-3%

1.021

0.868

1.862

1.468+120%+102%
100K MARC BIB Update51 min 15 sec1 hour 33 min48 min 45 sec1 hour  40 min Completed with errors (1 item discarded)+7%

0.556

0.838

1.046

1.415+112%+95%

...

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log dilefile:

View file
nameCreate_100k_logs-insights-results.csv
height250

...

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log dilefile:

View file
nameUpdate_100k_logs-insights-results.csv
height250

...

  • mod_source_record_storage.marc_records_lb = 22618121
  • mod_source_record_storage.raw_records_lb = 22650140
  • mod_source_record_storage.records_lb = 22650140
  • mod_source_record_storage.marc_indexers =  98256911(all records)
  • mod_source_record_storage.marc_indexers with field_no 010 = 139135
  • mod_source_record_storage.marc_indexers with field_no 035 = 4272473
  • mod_inventory_storage.authority = 7402975
  • mod_inventory_storage.holdings_record = 22027125
  • mod_inventory_storage.instance = 20986866
  • mod_inventory_storage.item = 22130108

PTF -environment ncp3 

  • 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, one reader, and one writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731


  • MSK ptf-kakfa-3
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3
  • Kafka topics partitioning: - 2 partitions for DI topics

...

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-inventory-storage26.0.0102102422081952
512
1440
mod-inventory20.0.4821024288025925121814
mod-source-record-storage5.6.52422048409636885123076
mod-quick-marc3.0.051128228821765121664
mod-source-record-manager3.6.2162
1024
4096
3688
5123076
mod-di-converter-storage2.0.2521281024896128768
mod-data-import2.7.181256204818445121292
okapi5.0.163102416841440512922
nginx-okapi2022.03.02621281024896--
pub-okapi2022.03.02621281024896-768
mod-feesfines

18.2.1

121281024896128768
mod-patron-blocks1.8.01210241024896128768
mod-pubsub2.7.012102415361440512922
mod-authtoken2.12.01251214401152128922
mod-circulation-storage

15.0.2

12102415361440512896
mod-circulation23.3.21210241024896128768
mod-configuration5.9.0121281024896128768
mod-users19.0.012

128

1024896128768
mod-remote-storage

1.7.1

12

128

1872

1692

512

1178m

...