Versions Compared

Old Version 28

changes.mady.by.user Stanislav Nehrii

Saved on

compared with

New Version Current

changes.mady.by.user Roman_Fedynyshyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

Overview

  • This document contains the results of testing marc-migration for over 12 million records on Ramsons ECS environment. In the Ramsons release, Spitfire created mod-marc-migrations to separate out the data migration process from upgrading the module. In this test, we'll analyze the performance of the newly created module, mod-marc-migrations, with respect to migrating 12 million Authority records and collect Baseline performance measurements for the marc-migration process across central tenant. 

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-1005
 

Summary

  • During the tests, we collected the mapping_duration and saving_duration for the central tenant, along with the total_saving_duration for all member tenants, specifically during the final test. When the saving process started for the central tenant, it triggered the update and saving processes asynchronously for all member tenants. This behavior is specific to the ECS environment.
  • The saving process encountered the status DATA_SAVING_FAILED, and not all records were updated. This issue occurred because the central tenant contained record IDs that were not present in the member tenants. Percentage of Unsaved Records for Test №1 was 0.69% but for last Test №7 it was 5.35%, this issue should be investigated. 
  • We gather baseline performance metrics for the marc-migration process across the central tenant. However, our recommendation is to collect results for both the central and member tenants, and separate metrics for each individual member tenant.

Recommendations and Jiras

  • Repeat tests to collect results for both the central and member tenants.
  • Run tests to collect separate metrics for each individual member tenant.
  • Fix test data set to avoid issue that central tenant contained record IDs which not present in the member tenants.


Test №statustotal_num_of_recordsmapped_num_of_recordssaved_num_of_recordspercentage of Unsaved Records
Test №1DATA_SAVING_FAILED1206725012067250119836920.69%
Test №2DATA_SAVING_FAILED1206725012067250119592810.89%
Test №3DATA_SAVING_FAILED1206725012067250119214421.21%
Test №4DATA_SAVING_FAILED1206725012067250118119762.12%
Test №5DATA_SAVING_FAILED1206725012067250118159272.08%
Test №6DATA_SAVING_FAILED1206725012067250116726973.27%
Test №7DATA_SAVING_FAILED1206725012067250114217435.35%


Test Results

This table contains duration time for Migrated and saved Marc Authority records. 

Test №CHUNK_FETCH_IDS_COUNTRECORDS_CHUNK_SIZEmapping_duration
Central Tenant		saving_duration
Central Tenant		total_saving_duration
Central and Member Tenants
Test №15005003:01:491:18:09
Test №2200010002:22:241:06:29


Test №3400020002:07:420:50:57
Test №4500025002:00:051:26:05
Test №5700035002:01:270:52:20
Test №61000050002:06:320:50:41
Test №71200040002:21:130:54:052:04:41

*Total saving duration time for Central and Member Tenants have to collected after each test run, results for Test №7 were collected only from automatic migrations triggered from the central tenant. Separate tests for each Member Tenants was not run.

Test №1-2-3-4-5-6-7

Introduction: The Baseline RCON Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The objective of these tests was to collect performance measurements for the marc-migration process across central tenants.

Results: Results were collect for central tenant and only for last test for member tenants. 

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-entities-links  module had spikes up to 90% Instances CPU power and mod-marc-migrations module used 20% Instances CPU power.

Service Memory Utilization

Here we can see that mod-entities-links had spikes up to 90% memory.


Kafka metrics

OpenSearch Data Nodes metrics

DB CPU Utilization

DB CPU had spikes up to 99%

DB Connections

Max number of DB connections was 1250.

DB load

                                                                                                                    

Top SQL-queries



Appendix

Infrastructure

PTF - Baseline RCON environment configuration

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.xlarge

    		32 GB4 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK fse-tenant
    • 2 brokers, kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...

Expand
titlercon modules memory and CPU parameters

Cluster Resources - rcon-pvt
Fri Oct 18 05:45:21 UTC 2024


ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage2mod-remote-storage:3.2.1-SNAPSHOT.17124920447203960512512
mod-finance-storage2mod-finance-storage:8.7.0-SNAPSHOT.18321024896070088128
mod-ncip2mod-ncip:1.14.6-SNAPSHOT.23321024896076888128
mod-agreements2mod-agreements:7.1.0-SNAPSHOT.2372159214880000
mod-ebsconet2mod-ebsconet:2.3.0-SNAPSHOT.802124810240700128256
mod-organizations2mod-organizations:2.0.0-SNAPSHOT.9521024896070088128
mod-consortia2mod-consortia:1.2.0-SNAPSHOT.2225136477604416384512
edge-sip22edge-sip2:3.3.0-SNAPSHOT.26421024896076888128
mod-serials-management2mod-serials-management:1.1.0-SNAPSHOT.4622480231201792384512
mod-settings2mod-settings:1.0.4-SNAPSHOT.6721024896076888128
mod-data-import2mod-data-import:3.2.0-SNAPSHOT.18912048184401292384512
mod-search6mod-search:4.0.0-SNAPSHOT.278225922480014405121024
edge-dematic2edge-dematic:2.3.0-SNAPSHOT.14311024896076888128
mod-inn-reach2mod-inn-reach:3.2.1-SNAPSHOT.102236003240028805121024
mod-record-specifications2mod-record-specifications:1.0.0-SNAPSHOT.421024896076888128
mod-tags2mod-tags:2.2.1-SNAPSHOT.13821024896076888128
mod-authtoken3mod-authtoken:2.16.0-SNAPSHOT.303214401152092288128
edge-courses2edge-courses:1.5.0-SNAPSHOT.116021024896076888128
mod-notify2mod-notify:3.2.1-SNAPSHOT.26821024896076888128
mod-inventory-update2mod-inventory-update:3.4.2-SNAPSHOT.10021024896076888128
mod-configuration2mod-configuration:5.11.0-SNAPSHOT.35521024896076888128
mod-orders-storage2mod-orders-storage:13.8.0-SNAPSHOT.24521024896070088128
edge-caiasoft2edge-caiasoft:2.3.0-SNAPSHOT.10521024896076888128
mod-login-saml2mod-login-saml:2.9.0-SNAPSHOT.29121024896076888128
mod-erm-usage-harvester2mod-erm-usage-harvester:5.0.0-SNAPSHOT.13321024896076888128
mod-gobi2mod-gobi:2.9.0-SNAPSHOT.25921024896070088128
mod-licenses2mod-licenses:6.1.0-SNAPSHOT.16322480231201792384512
mod-password-validator2mod-password-validator:3.2.2-SNAPSHOT.1492144012980768384512
edge-dcb3edge-dcb:1.2.0-SNAPSHOT.3921024896076888128
mod-bulk-operations2mod-bulk-operations:2.1.0-SNAPSHOT.11923072260001536384512
mod-fqm-manager2mod-fqm-manager:2.1.0-SNAPSHOT.1189230002600076888128
mod-graphql2mod-graphql:1.12.1.41221024896076888128
mod-finance2mod-finance:5.0.0-SNAPSHOT.17421024896070088128
mod-erm-usage2mod-erm-usage:5.0.0-SNAPSHOT.10922800255001800384512
mod-batch-print2mod-batch-print:1.2.0-SNAPSHOT.1521024896076888128
mod-tlr2mod-tlr:1.0.0-SNAPSHOT.821024896076888128
mod-lists2mod-lists:2.1.0-SNAPSHOT.130323000260002048384512
mod-copycat2mod-copycat:1.6.1-SNAPSHOT.20821024512076888128
mod-entities-links5mod-entities-links:3.1.0-SNAPSHOT.1132259224800144001024
mod-permissions3mod-permissions:6.6.0-SNAPSHOT.38621684154401024384512
pub-edge2pub-edge:2023.06.1421024896076800
mod-orders2mod-orders:12.9.0-SNAPSHOT.40122048174001024384512
edge-patron2edge-patron:5.2.0-SNAPSHOT.17421024896076888128
mod-marc-migrations5mod-marc-migrations:1.0.0-SNAPSHOT.821024896076888128
edge-ncip3edge-ncip:1.10.2-SNAPSHOT.11621024896076888128
edge-inn-reach3edge-inn-reach:3.3.0-SNAPSHOT.6921024896076888128
mod-users-bl2mod-users-bl:7.9.0-SNAPSHOT.318214401152092288128
mod-oa2mod-oa:2.1.0-SNAPSHOT.6621024896076888128
mod-inventory-storage3mod-inventory-storage:27.2.0-SNAPSHOT.773240963690030765121024
mod-invoice2mod-invoice:5.9.0-SNAPSHOT.205214401152092288128
mod-user-import2mod-user-import:3.9.0-SNAPSHOT.19721024896076888128
mod-sender2mod-sender:1.12.1-SNAPSHOT.10021024896076888128
edge-oai-pmh2edge-oai-pmh:2.10.0-SNAPSHOT.17421512136001440384512
mod-data-export-worker3mod-data-export-worker:3.3.0-SNAPSHOT.23323072204802048384512
mod-rtac2mod-rtac:3.7.0-SNAPSHOT.20321024896076888128
mod-circulation-storage2mod-circulation-storage:17.3.0-SNAPSHOT.41222880259201814384512
mod-source-record-storage3mod-source-record-storage:5.9.0-SNAPSHOT.40525600500003500384512
mod-calendar2mod-calendar:3.2.0-SNAPSHOT.22921024896076888128
mod-event-config2mod-event-config:2.7.2-SNAPSHOT.11021024896076888128
mod-courses3mod-courses:1.4.11-SNAPSHOT.15521024896076888128
mod-circulation-item2mod-circulation-item:1.0.1-SNAPSHOT.26210248960000
mod-inventory2mod-inventory:20.3.0-SNAPSHOT.54622880259201814384512
mod-email2mod-email:1.17.1-SNAPSHOT.12822800255001800384512
mod-requests-mediated2mod-requests-mediated:1.0.0-SNAPSHOT.421024896076888128
mod-circulation2mod-circulation:24.3.0-SNAPSHOT.68522880259201814384512
mod-pubsub2mod-pubsub:2.15.0-SNAPSHOT.2242153614400922384512
mod-di-converter-storage2mod-di-converter-storage:2.3.0-SNAPSHOT.12421024896076888128
edge-rtac2edge-rtac:2.7.3-SNAPSHOT.23421024896076888128
edge-orders2edge-orders:3.1.0-SNAPSHOT.15621024896076888128
mod-users2mod-users:19.3.3-SNAPSHOT.70221024896076888128
mod-template-engine2mod-template-engine:1.20.1-SNAPSHOT.11821024896076888128
mod-patron-blocks2mod-patron-blocks:1.11.0-SNAPSHOT.16321024896076888128
mod-audit2mod-audit:2.9.1-SNAPSHOT.21621024896076888128
edge-fqm3edge-fqm:2.1.0-SNAPSHOT.10021024896076888128
mod-source-record-manager2mod-source-record-manager:3.9.0-SNAPSHOT.35325600500003500384512
nginx-edge2nginx-edge:2023.06.14210248960000
mod-quick-marc2mod-quick-marc:5.2.0-SNAPSHOT.15312288217601664384512
nginx-okapi2nginx-okapi:2023.06.14210248960000
okapi-b2okapi:5.3.03168414400922384512
mod-feesfines2mod-feesfines:19.2.0-SNAPSHOT.19821024896076888128
mod-invoice-storage2mod-invoice-storage:5.9.0-SNAPSHOT.13821872153601024384512
mod-reading-room2mod-reading-room:1.0.0-SNAPSHOT.1221024896076888128
mod-dcb2mod-dcb:1.1.3-SNAPSHOT.4021024896076888128
mod-service-interaction2mod-service-interaction:4.1.0-SNAPSHOT.12622048184401290384512
mod-patron2mod-patron:6.2.0-SNAPSHOT.16521024896076888128
mod-data-export5mod-data-export:5.1.0-SNAPSHOT.24012592248001440881024
mod-oai-pmh2mod-oai-pmh:3.14.0-SNAPSHOT.40224096369003076384512
edge-connexion2edge-connexion:1.4.0-SNAPSHOT.8821024896076888128
mod-notes2mod-notes:5.3.0-SNAPSHOT.325210248960952384512
mod-kb-ebsco-java3mod-kb-ebsco-java:4.2.0-SNAPSHOT.17821024896076888128
mod-organizations-storage2mod-organizations-storage:4.8.0-SNAPSHOT.13321024896070088128
mod-data-export-spring2mod-data-export-spring:3.3.1-SNAPSHOT.16712048184401536384512
mod-login2mod-login:7.12.0-SNAPSHOT.3262144012980768384512
pub-okapi2pub-okapi:2023.06.1421024896076800
mod-eusage-reports2mod-eusage-reports:3.0.0-SNAPSHOT.7321024896076888128




Methodology/Approach

Baseline RCON Environment configuration

...

  • Marc-Migration tests were initiated by a JMeter script from a local machine with the following configurations for the mod-marc-migrations service. The migration of Marc Authority records was started for the central tenant, and automatic migrations were triggered for the member tenants.
Test №CHUNK_FETCH_IDS_COUNTRECORDS_CHUNK_SIZE
Test №1500500
Test №220001000
Test №340002000
Test №450002500
Test №570003500
Test №6100005000
Test №7120004000