Overview

This document contains the results of testing Data Import for MARC Bibliographic records in the Orchid release to detect performance trends. PERF-562 - Getting issue details... STATUS

Summary

• Duration for Orchid is increased almost twice due to fixing differences in the database schemas, mostly adding triggers functions. For example, 50K MARC Create took 39 min 27 sec for Orchid compared to 21 min 11 s, and 21 min 37s for Nolana and Morning Glory but for Lotus, it was 32 min 28 s. So we can assume that the trigger was missing for the previous 2 releases in our database.
• Memory utilization increasing due to previous modules restarting (everyday cluster shot down process) no memory leak is suspected for all of the modules due to investigation in the scope of the ticket PERF-541 - Getting issue details... STATUS .
• Average CPU usage did not exceed 130 % for all modules. We can observe spikes in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 270%.
• Approximately DB CPU usage is up to 97%.
•  We can not compare R/W split-enabled DI test to the first test from 13/06/2023 because of the different versions of modules tested. However, there are no significant improvements in DI duration for R/W split enabled if compared between new module versions. Details are in the table below and in the Data Import series of tests to understand a range of duration of each DI test (Orchid). However, R/W splitting could decrease the database load for the writer instance as a result other processes with the data import could be performed faster.

Recommendations and Jiras

Implement database schemas comparison process to each cluster deployment to avoid database misconfiguration.

Results

Orchid with R/W split enabled (07/09/2023) enabled for:

• mod-data-import
• mod-source-record-storage
• mod-source-record-manager
• mod-di-converter-storage

Orchid with R/W split enabled (08/09/2023) enabled for:

• mod-data-import
• mod-source-record-storage
• mod-source-record-manager
• mod-di-converter-storage
• mod-inventory-storage

There 2 modules that affect R/W split the most due to the number of read queries to the database:

• mod-inventory-storage (mostly)
• mod-source-record-manager

All other modules almost or at all do not have read queries to the database.

 * - io.vertx.core.impl.NoStackTraceThrowable: Current retry number 1 exceeded or equal given number 1 for the Item update for jobExecutionId '8fde78a8-2450-44c7-83ac-c98376a90491'

From ncp5/mod-inventory/

11:34:53 [] [] [] [] WARN  dateItemEventHandler OL error updating Item - ERROR: Cannot update record 955f8ba1-76ac-4931-8236-1c2fb4775379 because it has been changed (optimistic locking): Stored _version is 5, _version of request is 3 (23F09), status code 409. Retry UpdateItemEventHandler handler...

Memory Utilization

This has memory utilization increasing due to previous modules restarting (everyday cluster shot down process) no memory leak is suspected for all of the modules due to investigation in the scope of the ticket PERF-541 - Getting issue details... STATUS .

MARC BIB CREATE

MARC BIB UPDATE

Service CPU Utilization 

MARC BIB CREATE

Average CPU usage did not exceed 130 % for all modules. We can observe spikes in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 270%.

MARC BIB UPDATE

RDS CPU Utilization 

MARC BIB CREATE

Approximately DB CPU usage is up to 97%

MARC BIB UPDATE

CPU Utilization for BIB Update 100k records R/W split enabled

RDS Database Connections

MARC BIB CREATE
 For DI  job Create- Maximum 560 connections count.

MARC BIB UPDATE

For DI  job Update-- Maximum 500 connections count.

Average connection usage

Maximum connection usage

BIB Update 100k records R/W split enabled

BIB Update 100k records R/W split enabled

From reader DB instance

From writer DB instance

Appendix

Infrastructure

Records count :

• mod_source_record_storage.marc_records_lb = 22618121
• mod_source_record_storage.raw_records_lb = 22650140
• mod_source_record_storage.records_lb = 22650140
• mod_source_record_storage.marc_indexers =  98256911(all records)
• mod_source_record_storage.marc_indexers with field_no 010 = 139135
• mod_source_record_storage.marc_indexers with field_no 035 = 4272473
• mod_inventory_storage.authority = 7402975
• mod_inventory_storage.holdings_record = 22027125
• mod_inventory_storage.instance = 20986866
• mod_inventory_storage.item = 22130108

PTF -environment ncp5 

• 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters

512

1024

4096

3688

Methodology/Approach

To test Baseline for DI JMeter scripts were used.

• 5 min pauses between the tests