Overview

This document contains the results of testing Data Import MARC Authorities in Orchid release to detect performance trends.

Ticket: https://folio-org.atlassian.net/browse/PERF-444

Previous test report: Data Import MARC Authorities (Nolana)

Summary

• All the tests were done successfully without errors or issues. 

• There is no specific trend for DI performance and resources utilization for Orchid release compared to Nonala. There is degradation for smaller file jobs, while for bigger file jobs processing time improved (up to 19%). See the details...

• Max CPU utilization was observed for mod-inventory module, it reached 43% during 50K DI import. See the details...

• Max DB CPU reached 86% during 50K DI import. No memory leaks found. See the details...

It should be noted that trigger process_marc_records_lb_insert_update_trigger was enabled for mod_source_record_storage.marc_records_lb table to make the environment production-like. It could influence the results in current release.

Test Results

Profile used for testing - "Default - Create SRS MARC Authority"

Instance CPU Utilization

Open

Service CPU Utilization

Max CPU utilization was observed for mod-inventory module, it reached 43% during 50K DI import.

Open

Max CPU utilization for non-DI modules was observed for mod-quick-marc module, it reached 53% during 50K DI import.

Open

Memory Utilization

Open

DB CPU Utilization

Max DB CPU reached 86% during 50K DI import.

Open

DB Connections

Open

Infrastructure

Records count :

• mod_source_record_storage.marc_records_lb = 25015778

• mod_source_record_storage.raw_records_lb = 25048647

• mod_source_record_storage.records_lb = 25048897

• mod_source_record_storage.marc_indexers =  283585109 (all records)

• mod_source_record_storage.marc_indexers with field_no 010 = 219260

• mod_source_record_storage.marc_indexers with field_no 035 = 10165466

• mod_inventory_storage.authority = 7493975

• mod_inventory_storage.holdings_record = 23900853

• mod_inventory_storage.instance = 22617024

• mod_inventory_storage.item = 23801604

PTF -environment ncp5 

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1  

• 2 database  instances, one reader, and one writer

• Number of connections for mod-source-record-manager and mod-source-record-storage: 30 connections.

• Profile used: Default - Create SRS MARC Authority

• MSK ptf-kakfa-3

  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

• Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters