Overview IN PROGRESS

The purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in Quesnelia in scope of ticket  PERF-855 - Getting issue details... STATUS

Compared with results in previous test report: Data Import Create MARC holdings records [Poppy]

Summary

• Data import jobs duration decreased significantly compared to the Orchid release results for all tests
• Top CPU utilization: nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%,  mod-source-record-storage-b - 25%
• Top Memory consumption: mod-users-b - 60%, mod-data-import-b - 60%, okapi-b - 48%. There're no issues with memory.
• DI jobs duration is directly proportional to the size of the files being processed.

Recommendations & Jiras

• Autovacuum as background process increase DI job duration for 10%-25% so it's necessary to check DB activity before DI jobs start
• SQL queries are the possible place where performance may be improved:
  • SQL DB Load with WAITS - Lock:tuple, Lock:transactionid take a major time in queries: 

    INSERT INTO [tenant]_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

    UPDATE [tenant]_mod_inventory_storage.holdings_record SET jsonb = $1::jsonb WHERE id = 'UUID'

Test Results

Profile used for testing - Default - Create Holdings and SRS MARC Holdings

Set 1 - Files used to test DI create Holdings had 1 instance HRID for all created Holdings

Set 2 - Files used to test DI create Holdings had 1 unique instance HRID for every 1000 created Holdings (new approach)

Comparison

Table contains comparison between Quesnelia and Poppy

Service CPU Utilization

Set #1: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%

Set #2: mod-inventory-b - 33%, nginx-okapi - 23%, mod-source-record-storage-b - 11%, mod-quick-marc-b - 7%

Set #1

Set #2

Memory Utilization

Set #1

Set #2

DB CPU Utilization

DB CPU was about 80% for all tests except of 1k - 23%.

DB Connections

DB connections number- 505

DB Load

SQL queries

Infrastructure

PTF - environment qcon

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 1 database  instances, writer

• MSK ptf-mobius-testing2
  • 2 m5.2xlarge brokers in 2 zones
  • Apache Kafka version 2.8.0
  • EBS storage volume per broker 300 GiB
  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2