Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Orchid release to detect performance trends.

Ticket: PERF-472 - Getting issue details... STATUS

Summary

• There is a significant performance degradation of data import for Orchid in comparison to Nolana results. Response time are about 2 times higher. It might be due to fixing differences in the database schemas. More details...
• Data import response times are up to 22% higher with parallel Check-in/Check-out than pure Data import results. More details...
• Check-in/Check-out response times are up to 168% higher with parallel Data import than pure Check-in/Check-out results. More details...
• There is memory utilization increase observed which is caused by previous modules restarting (everyday cluster shut down process). More details...
• Average CPU usage did not exceed 130 % for all the modules. Spikes can be observed in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 170%. CPU usage is about 2 times lower than in the same test for Nolana. More details...
• Average DB CPU usage during data import is about 95%. More details...

Test Runs 

Test Results

Data import

Check-in/Check-out

Comparison

The following table compares test results of current release (Orchid) to the previous release numbers (Nolana) and to the baselines Orchid results (CICO without DI and DI without CICO).

* Nolana DI and CICO results are taken from Data Import with Check-ins Check-outs Nolana.

** Orchid DI results are taken from Data Import test report (Orchid).

Detailed CICO response time comparison

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI Bib Update + CICO

Service CPU Utilization

Average CPU usage did not exceed 130 % for all the modules. Spikes can be observed in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 170%.

CPU usage is about 2 times lower than in the same test for Nolana - Data Import with Check-ins Check-outs Nolana.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB CPU Utilization

Average DB CPU usage during data import is about 95%

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB Connections

Average connection count during data import is about 270 connections.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB load

DI MARC BIB Create + CICO

Top SQL-queries:

update "marc_records_lb" set "content" = cast($1 as jsonb) where "id" = cast($2 as uuid)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log file:

MARC BIB Update + CICO

Top SQL-queries:

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

RDS log file:


Appendix

Infrastructure

Records count:

• mod_source_record_storage.marc_records_lb = 22618121
• mod_source_record_storage.raw_records_lb = 22650140
• mod_source_record_storage.records_lb = 22650140
• mod_source_record_storage.marc_indexers =  98256911(all records)
• mod_source_record_storage.marc_indexers with field_no 010 = 139135
• mod_source_record_storage.marc_indexers with field_no 035 = 4272473
• mod_inventory_storage.authority = 7402975
• mod_inventory_storage.holdings_record = 22027125
• mod_inventory_storage.instance = 20986866
• mod_inventory_storage.item = 22130108

PTF -environment ncp3 

• 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters

512

1024

4096

3688

Methodology/Approach

To test Baseline for DI JMeter scripts were used with 5 min pauses between the tests.

Additional links

Grafana dashboard:

MARC Bib Create + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1686817177738&to=1686830183685&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

MARC Bib Update + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1686839353607&to=1686853077913&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_orchid&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All