Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Quesnelia release [non-ECS].
Ticket: PERF-836 - Getting issue details... STATUS on QCP1 environment.

Summary

• All Data-imports jobs finished successfully.

• Duration for DI increases correlates to the number of records imported. 

• The average CPU utilization of modules for all Create and Update jobs did not exceed 150 %. Spikes at the beginning on mod-data-import module are expected because of large file uploading.

• No memory leak is suspected for DI modules. During DI of 500K on both profiles, mod-pubsub has a sawtooth-like memory usage within the range of 40-60%.

• Approximate DB CPU usage is close to 95% and this number goes for all jobs with files more than 10k records. 

Comparison with previous testing results Data Import test report (Poppy)

• Duration for Data-import with PTF - Create 2 has not increased significantly. In Quesnelia release there were some changes in the update profile so the new one was created PTF - Updates Success - 6, these durations will be the baseline for the next tests.

• Services CPU utilization, Service memory utilization, and DB CPU utilization have the same utilization trend and values as in the Poppy release.

Recommendations and Jiras

1. Investigate data-import job details failed to load MODDATAIMP-1044 - Getting issue details... STATUS

Results

Test Runs 

MARC BIB CREATE

Tests #1-7 1k, 5k, 10k, 25k, 50k, 100k, 500k records

MARC BIB UPDATE

Tests #8-15 1k, 2k, 5k, 10k, 25k, 50k, 100k, 500k records

Service CPU Utilization 

MARC BIB CREATE

Tests #1-7

1k, 5k, 10k, 25k, 50k, 100k, 500k records

CPU utilization for all modules returned by default numbers after all tests. Average for mod-inventory-b - 130%, mod-inventory-storage-b - 25%, mod-source-record-storage-b - 40%, mod-source-record-manager-b - 35%, mod-di-converter-storage-b - 70%, , mod-data-import - 350% spike for 500k job(same behaviour on Poppy version).

MARC BIB UPDATE

Tests #8-15

1k, 2k, 5k, 10k, 25k, 50k, 100k, 500k records

Memory Utilization

No memory leak is suspected for DI modules.

MARC BIB CREATE

Tests #1-7

1k, 5k, 10k, 25k, 50k, 100k, 500k records

MARC BIB UPDATE

Tests #8-15

1k, 2k, 5k, 10k, 25k, 50k, 100k, 500k records

RDS CPU Utilization 

MARC BIB CREATE

Average 95% for DI jobs with more than 10k records for Create and Update profiles

MARC BIB UPDATE

RDS Database Connections

MARC BIB CREATE
When the SUT Data-Import job Create profile typically uses an average of 675 connections, whereas for Update profile, it uses around 640 connections. When the SAR number of the DB connection is about 500.

MARC BIB Update

Average active sessions (AAS)

MARC BIB CREATE

Top SQL

MARC BIB UPDATE

Top SQL

MSK CPU utilization (Percent) OpenSearch

Avarage CPU Utilization is about 9%

CPU (User) usage by broker

MARC BIB Create

MARC BIB Update

The use of resources on Kafka clusters will not be valid, since testing was carried out on another environment, as well as one connected to the tenant cluster.

Errors

fields @timestamp, @message, @logStream, @log
############       DISABLED MODULES
| filter @logStream not like "mod-remote-storage" and @logStream not like "edge-caiasoft" and @logStream not like "mod-graphql" and @logStream not like "edge-ncip" and @logStream not like "mod-rtac"
| filter @logStream not like "mod-finance"  and @logStream not like "mod-inn-reach"

############       DISABLED MODULES
| filter @message like "Error" or @message like "error" or @message like "ERROR"
#| filter @message like "Exception" or @message like "exception" or @message like "WARN" 

############       mod-authtoken/ 
| filter @message not like "Invalid token" and  @message not like "Token validation failure: Access token has expired" and @message not like "Access token has expired"

############       okapi-b , nginx-okapi
| filter @message not like "statusAny=ERROR" and  @message not like "No suitable module found for path and tenant" and  @message not like "Removing connection to endpoint" and  @message not like "ErrorTypeException: 503:"
| filter @message not like "error_page" 

############       mod-search
| filter @message not like "invokeBatchErrorHandler" and @message not like "ErrorHandlingUtils" and @message not like "FallbackBatchErrorHandler" and @message not like "DefaultErrorHandler" and @message not like "errorFromXContent"

############       mod-data-export-spring
| filter @message not like "java.lang.NoClassDefFoundError: io/vertx/core/Vertx" and @message not like "ERROR StatusConsoleListener Resolver failed to lookup FolioLoggingContext" 

############       mod-inventory ---- Instances contains error in title 
| filter @message not like "INFO  cceedingTitlesHelper createPrecedingSucceedingTitles" 

###########       mod-pub-sub 
| filter @message not like "TopicExistsException" and @message not like "Some of the topics" 

###########       OTHER ERRORS
| filter @message not like "Failed to index resource event" 
| filter @message not like "FeignException.errorStatus" and @message not like "FeignException.clientErrorStatus"  and @message not like "/var/log/nginx/error"
| filter @message not like "ErrorDecoder$Default.decode" and @message not like "InvocationContext.decodeError" and @message not like "Records indexed to elasticsearch"
| filter @message not like "HeapDumpOnOutOfMemoryError" and @message not like "while connecting to upstream" and @message not like "TopicExistsException" and @message not like "main ERROR Unrecognized"
| sort @timestamp desc
| limit 5000

Appendix

Infrastructure

PTF -environment qcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  NameMemory GIBvCPUsmax_connections

  db.r6g.xlarge

  32 GiB4 vCPUs2731

• MSK tenant
  4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

Methodology

1. Pregenerated files were used for DI Create job profile

   • 1K, 2K, 5K, 10K, 25K, 50K, 100K and 500K files.

2. Run DI Create on a single tenant one by one with the delay with files using PTF - Create 2 profile.

3. Prepare files for DI Update with the Data export app, using previous imported items

4. Run DI Update on a single tenant one by one with delay with prepared files using PTF - Update Success 6 profile

• 1K, 2K, 5K, 10K, 25K, 50K, 100K and 500K files.

5. Data-import durations were obtained from DB using SQL query

SELECT (completed_date-started_date) as duration, *
	FROM fs09000000_mod_source_record_manager.job_execution
where subordination_type = 'COMPOSITE_PARENT'
order by started_date desc
limit 10