Overview

• This document contains the results of testing marc-migration for over 12 million records on Ramsons ECS environment. In the Ramsons release, Spitfire created mod-marc-migrations to separate out the data migration process from upgrading the module. In this test, we'll analyze the performance of the newly created module, mod-marc-migrations, with respect to migrating 12 million Authority records. 

PERF-1005 - Getting issue details... STATUS  

Summary

• During the tests, we collected the mapping_duration and saving_duration for the central tenant, along with the total_saving_duration for all member tenants, specifically during the final test. When the saving process started for the central tenant, it triggered the update and saving processes asynchronously for all member tenants. This behavior is specific to the ECS environment.
• The saving process encountered the status DATA_SAVING_FAILED, and not all records were updated. This issue occurred because the central tenant contained record IDs that were not present in the member tenants. Percentage of Unsaved Records for Test №1 was 0.69% but for last Test №7 it was 5.35%, this issue should be investigated. 
• We gather baseline performance metrics for the marc-migration process across the central tenant. However, our recommendation is to collect results for both the central and member tenants, and separate metrics for each individual member tenant.

Recommendations and Jiras

• Repeat tests to collect results for both the central and member tenants.
• Run tests to collect separate metrics for each individual member tenant.

Test Results

This table contains duration time for Migrated and saved Marc Authority records. 

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-entities-links  module had spikes up to 90% Instances CPU power and mod-marc-migrations module used 20% Instances CPU power.

Service Memory Utilization

Here we can see that mod-entities-links had spikes up to 90% memory.

Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU had spikes up to 99%

DB Connections

Max number of DB connections was 1250.

DB load

Top SQL-queries

Appendix

Infrastructure

PTF - Baseline RCON environment configuration

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  Name	Memory GIB	vCPUs
  db.r6g.xlarge	32 GB	4 vCPUs

• Open Search ptf-test 
  • Data nodes
    • Instance type - r6g.2xlarge.search
    • Number of nodes - 4
    • Version: OpenSearch_2_7_R20240502
  • Dedicated master nodes
    • Instance type - r6g.large.search
    • Number of nodes - 3
• MSK fse-tenant
  
  • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x 

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Methodology/Approach

Baseline RCON Environment configuration

Test preparation: 

• 12 million Marc Authority records present in database

• Background 'Deleting old records from marc_indexers' jobs were disabled:
  added configuration to mod-source-record-storage

                  {
                      "name": "srs.marcIndexers.delete.interval.seconds",
                      "value": "86400"
                  }

• Check Migration Operation for central tenant from the database:

  select end_time_mapping - start_time_mapping as mapping_duration,end_time_saving - start_time_saving as saving_duration,* 
  from cs00000int_mod_marc_migrations.operation order by start_time_mapping desc

• Check Migration Operation for member tenants from the database:

  WITH date_range AS (
  SELECT 
  '10/18/2024 6:22:40 AM'::timestamp AS start_date, 
  '10/18/2024 3:23:28 PM'::timestamp AS end_date
  )
  SELECT 
  AGE(MAX(updated_date), MIN(updated_date)) AS total_saving_duration,
  MIN(updated_date) AS start_total_saving_process, 
  MAX(updated_date) AS end_total_saving_process 
  FROM (
  SELECT updated_date 
  FROM cs00000int_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  UNION ALL
  SELECT updated_date 
  FROM cs00000int_0001_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  UNION ALL
  SELECT updated_date 
  FROM cs00000int_0002_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  UNION ALL
  SELECT updated_date 
  FROM cs00000int_0003_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  UNION ALL
  SELECT updated_date 
  FROM cs00000int_0004_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  UNION ALL
  SELECT updated_date 
  FROM cs00000int_0005_mod_entities_links.authority, date_range 
  WHERE updated_date >= date_range.start_date AND updated_date <= date_range.end_date
  ) AS combined;

Test execution:

• Marc-Migration tests were initiated by a JMeter script from a local machine with the following configurations for the mod-marc-migrations service. The migration of Marc Authority records was started for the central tenant, and automatic migrations were triggered for the member tenants.