Versions Compared

Old Version 26

changes.mady.by.user Stanislav Nehrii

Saved on

compared with

New Version 27

changes.mady.by.user Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • During the tests, we collected the mapping_duration and saving_duration for the central tenant, along with the total_saving_duration for all member tenants, specifically during the final test. When the saving process started for the central tenant, it triggered the update and saving processes asynchronously for all member tenants. This behavior is specific to the ECS environment.
  • The saving process encountered the status DATA_SAVING_FAILED, and not all records were updated. This issue occurred because the central tenant contained record IDs that were not present in the member tenants. Percentage of Unsaved Records for Test №1 was 0.69% but for last Test №7 it was 5.35%, this issue should be investigated. 
  • We gather baseline performance metrics for the marc-migration process across the central tenant. However, our recommendation is to collect results for both the central and member tenants, and separate metrics for each individual member tenant.

Recommendations and Jiras

  • Repeat tests to collect results for both the central and member tenants.
  • Run tests to collect separate metrics for each individual member tenant.
  • Fix test data set to avoid issue that central tenant contained record IDs which not present in the member tenants.


Test №statustotal_num_of_recordsmapped_num_of_recordssaved_num_of_recordspercentage of Unsaved Records
Test №1DATA_SAVING_FAILED1206725012067250119836920.69%
Test №2DATA_SAVING_FAILED1206725012067250119592810.89%
Test №3DATA_SAVING_FAILED1206725012067250119214421.21%
Test №4DATA_SAVING_FAILED1206725012067250118119762.12%
Test №5DATA_SAVING_FAILED1206725012067250118159272.08%
Test №6DATA_SAVING_FAILED1206725012067250116726973.27%
Test №7DATA_SAVING_FAILED1206725012067250114217435.35%

...

Here we can see that mod-entities-links  module had spikes up to 90% Instances CPU power and mod-marc-migrations module used 20% Instances CPU power.

...

Here we can see that mod-entities-links had spikes up to 90% memory.


Kafka metrics

OpenSearch Data Nodes metrics

...

PTF - Baseline RCON environment configuration

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.xlarge

    		32 GB4 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK fse-tenant
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...