Table of Contents

...

This document contains the results of testing Data Export (MARC BIB) on Orchid release with baseline Data Export tests.

Ticket:

Summary

• Data export jobs duration has no degradation for all DE files (1k, 100k, 500k). 
• Maximum CPU utilization was observed for mod-data-export module during DE job with 500k of instances - 10%.
  • average utilization for mod-data-export:
    • Service CPU Utilization: 10%
    • Memory Utilization: 72%
    • DB CPU Utilization: 18%
    • DB Connections: 165

Recommendations & Jiras

• Investigate negative values in 'Failed' column in UI

  Jira Legacy
  server System JIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-676

Test Results

Profile used for testing - "srs - holdings and items"

...

Instance CPU Utilization

Service CPU Utilization

Average mod-data-export: 10%

Memory Utilization

Average mod-data-export: 72%

DB CPU Utilization

Average mod-data-export: 18%

DB Connections

Average: 165

DB Load

SQL queries

Top-SQL statement: 

...

• 3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;
• All tests were carried out sequentially

Infrastructure

PTF -environment ncp5 

• 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1  

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• number of connections for mod-source-record-manager and mod-source-record-storage - 30 connections.
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

...