Overview

• This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

PERF-818 - Getting issue details... STATUS  

Summary

• Three tests of DI with update jobs with 25k records were carried out on the qcp1 cluster.
• Comparing the Quesnelia and Poppy releases, we observed improved DI update job durations in the Quesnelia environment. On average, these durations were 1 minute and 23 seconds shorter for a single tenant and 2 minutes shorter for two and three concurrent tenants in the Quesnelia environment.
• Memory consumption for mod-data-import was 60%, mod-inventory - 128%. No memory leaks are suspected.
• CPU utilization for mod-data-import spiked to 70%, mod-inventory - 150%.
• DB CPU utilization was 93%.
• DB connections - 1273 on average.

Test Results

Service CPU Utilization

Here we can see that mod-inventory-b module used 171% CPU, mod-di-converter-storage-b used 83% CPU and nginx-okapi used 66% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU in the average was 88%.

DB Connections

DB connections was 1139.

DB load

Top SQL-queries

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

set SEARCH_PATH to 'fs09000000_mod_source_record_manager';
SELECT * FROM get_job_log_entries('b7d9c003-eff8-42a7-8a77-1473defa0a84','source_record_order', 'ASC', 100, 0, false, 'ALL')

Service CPU Utilization

Here we can see that mod-inventory-b module used 140% CPU in average, mod-inventory-b used 117% CPU and nginx-okapi used 109% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 93%.

DB Connections

DB connections was 1273.

DB load

Top SQL-queries

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

• 10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instances, writer

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GiB	4 vCPUs

• MSK ptf-mobius-testing2
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=2

Methodology/Approach

DI tests scenario (DI MARC Bib Update) were started from UI on Quesnelia (qcp1) env with Kafka consolidated topics and file splitting features enabled on a non-ecs-enabled environment..

Test set

• Test 1: Manually tested 25k records files DI started on one tenant only.
• Test 2: Manually tested 25k records files DI started on 2 tenants concurrently, step 30%.
• Test 3: Manually tested 25k records files DI started on 3 tenants concurrently, step 30%.