Overview

The purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in Poppy in scope of ticket  PERF-700 - Getting issue details... STATUS

Compared with results in previous test report: Data Import Create MARC holdings records [Orchid]

Summary

• Data import jobs duration decreased significantly compared to the Orchid release results for all tests
• Top CPU utilization: nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%,  mod-source-record-storage-b - 25%
• Top Memory consumption: mod-users-b - 60%, mod-data-import-b - 60%, okapi-b - 48%. There're no issues with memory.
• DI jobs duration is directly proportional to the size of the files being processed.

Recommendations & Jiras

• Autovacuum as background process increase DI job duration for 10%-25% so it's necessary to check DB activity before DI jobs start
• SQL queries are the possible place where performance may be improved:
  • SQL DB Load with WAITS - Lock:tuple, Lock:transactionid take a major time in queries: 

    INSERT INTO [tenant]_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

    UPDATE [tenant]_mod_inventory_storage.holdings_record SET jsonb = $1::jsonb WHERE id = 'UUID'

Test Results

Profile used for testing - Default - Create Holdings and SRS MARC Holdings

Comparison

Table contains comparison between Poppy and Orchid

Service CPU Utilization

nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%,  mod-source-record-storage-b - 25%

Memory Utilization

DB CPU Utilization

DB CPU was about 80% for all tests except of 1k - 23%.

DB Connections

DB connections number- 505

DB Load

SQL queries

Infrastructure

PTF - environment pcp1

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 1 database  instances, writer

• MSK ptf-mobius-testing2
  • 2 m5.2xlarge brokers in 2 zones
  • Apache Kafka version 2.8.0
  • EBS storage volume per broker 300 GiB
  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2