Table of Contents

Overview

...

Ticket:

Previous test report: Data Import Create MARC holdings records [Nolana]

Summary

• Data import jobs duration significantly increased compared to the Nolana results, see TestResults. 
• Maximum CPU utilization was observed for mod-inventory-storage module during 80k DI job - 88%, see ServiceCPUUtilization.
• There is a significant memory utilization increase for some modules during 10k and 80k DI jobs, see MemoryUtilization. 
• Maximum service memory utilization reached 97% for mod-inventory-storage during 80k DI job, see MemoryUtilization.
• During 80k DI job there is a switch of instrances for reader and writer role. This might happen because of overload and restart of the writer instance, see DBCPUUtilization.

It should be noted that trigger process_marc_records_lb_insert_update_trigger was enabled for mod_source_record_storage.marc_records_lb table to make the environment production-like. It could influence the results in current release. At the same time tests showed that DI jobs duration with and without the trigger stays the same.

Recommendations & Jiras

Issue with failing 80K DI jobs was reproduced in 100% (tested several times). This issues requires further investigation.

Ticket:  

Test Results

Profile used for testing - Default - Create Holdings and SRS MARC Holdings

...

Infrastructure

PTF -environment ncp5 

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1  

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• number of connections for mod-source-record-manager and mod-source-record-storage - 30 connections.
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

...

512

...