Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Poppy release. PERF-712 - Getting issue details... STATUS  

Summary

• Duration for DI increases correlates to the number of the records imported. 
• The increase in memory utilization was due to the scheduled cluster shutdown. No memory leak is suspected for DI modules.
• Average CPU utilization of modules for all Create and Update jobs did not exceed 150 %. Spikes at the beginning because of mod-data-import module are expected.
• Approximate DB CPU usage is close to 95% and this numbers goes for all jobs with files more than 10k records. 
• Poppy release has higher average CPU resource utilization of DI related services comparing with Orchid. Specially mod-inventory. 
• Errors occurred for DI create job with 100k file and for DI update jobs with 25k file because of timeout issue (Opening SQLConnection failed: Timeout)

Recommendations and Jiras

1. Investigate Timeout issues. Ticket created MODINV-924 - Getting issue details... STATUS
2. Check memory trends for mod-source-record-storage-b and mod-inventory during additional DI tests without cluster night shut down
3. Increase CPU units allocation for mod-inventory, mod-di-converter-storage, mod-quick-marc services
4. Use higher DB instance type (scale up from db.r6g.xlarge to db.r6g.2xlarge). 

Results

 * - for all jobs completed with errors or cancelled there was the same issue in UI: io.vertx.core.impl.NoStackTraceThrowable: Timeout

Test #14 was stopped manually from UI. 2 tests with 25k MARC BIB Update were carried out to confirm that 25k doesn't work properly and has the same issue. 

Memory Utilization

The increase in memory utilization was due to the scheduled cluster shutdown. no memory leak is suspected for DI modules.

MARC BIB CREATE

Tests #1-7

1k, 2k, 5k, 10k, 25k, 50k, 100k records

MARC BIB UPDATE

Tests #8-14

1k, 2k, 5k, 10k, 25k, 25k, 25k records

Service CPU Utilization 

MARC BIB CREATE

Tests #1-7

1k, 2k, 5k, 10k, 25k, 50k, 100k records

CPU utilization for all modules came back to by default numbers after all tests. The highest 170% of resource utilization was observed for mod-quick-marc-b module in 5k DI create job.

Average for mod-inventory-b - 125%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 60%, mod-source-record-manager-b - 35%, mod-di-converter-storage-b - 80%, , mod-data-import - 200% spike for 100k job.

MARC BIB UPDATE

Tests #8-14

1k, 2k, 5k, 10k, 25k, 25k, 25k records

Average for mod-inventory-b - 220%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 50%, mod-source-record-manager-b - 45%, mod-di-converter-storage-b - 90%, , mod-data-import - 96% spike for 25k job.

RDS CPU Utilization 

MARC BIB CREATE

Average 95% for DI jobs with more than 10k records

MARC BIB UPDATE

RDS Database Connections

MARC BIB CREATE
 For DI  job Create maximum 275 and for Update - 260 connections

Average active sessions (AAS)

MARC BIB CREATE

Top SQL

MARC BIB UPDATE

Top SQL

INSERT INTO fs09000000_mod_source_record_manager.events_processed 

INSERT INTO fs09000000_mod_source_record_manager.journal_records 

MSK CPU utilization (Percent) OpenSearch

Utilization is not higher than 20%

CPU (User) usage by broker

Errors

15:58:44 [478677/data-import] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [] ERROR PostgresClient       Opening SQLConnection failed: Timeout

15:24:19 [] [fs09000000] [] [] ERROR KafkaConsumerWrapper businessHandlerCompletionHandler:: Error while processing a record - id: 3 subscriptionPattern: SubscriptionDefinition(eventType=DI_COMPLETED, subscriptionPattern=pcp1\.Default\.\w{1,}\.DI_COMPLETED) offset: 541469

io.vertx.pgclient.PgException: ERROR: Cannot update record 511ea771-3dd0-49a7-a14d-c4187c94aff7 because it has been changed (optimistic locking): Stored _version is 2, _version of request is 1 (23F09)

Additional tests were analysed to investigate "(optimistic locking)" issue

Issue found only in create jobs in mod-inventory-storage and mod-inventory modules - it happens in completed and failed jobs as well and also depends on files.

Table shows the amount of optimistic locking messages in tests. Hyphen '-' means no test performed.

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology

1. Prepare files for DI Create job

   • 1K, 2K, 5K, 10K, 25K, 50K, 100K files.
2. Run DI Create on a single tenant one by one with delay with files using PTF - Create 2 profile.
3. Prepare files for DI Update with Data export app
4. Run DI Update on a single tenant one by one with delay with prepared files using PTF - Update Success 1 profile