Versions Compared

Old Version 25

changes.mady.by.user Stanislav Nehrii

Saved on

compared with

New Version Current

changes.mady.by.user Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • Data Export tests finished successfully on the Eureka environment using the "Default instances export job profile" and "srs - holdings and items job profile." Data Export tests were run on Central tenants.
  • During Test №1-2When comparing QECP1 Eureka with Quesnelia ECS, NON-ECS environments:
      • Data Export duration for the "Default instances export job profile" was almost the same across environments.
      • Data Export duration for the "srs - holdings and items job profile" showed a significant degradation: around 2.5 times longer for the 1k and 100k files, and around 5 times longer for the 500k file, and no clear reasons were identified.
  • During Tests №1 and №2, we noticed some background processes for Database, so we deactivated mod-fqm-manager. As a result, the repeated test showed a twofold improvement in performance.
  • Data Export jobs were getting stuck and returned a [401 Unauthorized] error with the message: "errors [{"type":"UnauthorizedException","code":"authorization_error","message":"Unauthorized"}]. To resolve this, we increased the token lifespan by modifying the KC_CONFIG_TTL parameter for mod-login-keycloak from 360s to 3600s and KC_ADMIN_TOKEN_TTL from 410s to 4100s:
      • {"name": "KC_CONFIG_TTL","value": "3600s"}
      • {"name": "KC_ADMIN_TOKEN_TTL", "value": "4100s"}
    When comparing QECP1 Eureka with Quesnelia ECS, NON-ECS environments:
      • Data Export duration for the "Default instances export job profile" was almost the same across environments.Data Export duration for the "srs - holdings and items job profile" showed a significant degradation: around 2.5 times longer for the 1k and 100k files, and around 5 times longer for the 500k file., "value": "4100s"}


Test Results

This table contains durations for Data Export with 2 job profiles. 

Profile

CSV  FileCentral Tenant (fs09000000)
Result
Test Set 1
Result
Test Set 2
Status
DE MARC Bib (Default instances export job profile)

1k.csv0:00:070:00:03COMPLETED
100k.csv0:06:030:02:17COMPLETED
500k.csv0:09:030:04:25COMPLETED
DE MARC Bib (srs - holdings and items)

1k.csv0:00:130:00:10COMPLETED
100k.csv0:20:250:12:41COMPLETED
500k.csv0:38:210:38:53COMPLETED

...


ProfileNumber of recordsQuesnelia NON-ECS QCP1Quesnelia ECS
QCON		Quesnelia Eureka
QECP1		DE Duration, DELTA QCP1/QECP1
hhh:mm:sshhh:mm:sshhh:mm:sshhh:mm:ss / percent
DE MARC Bib (Default instances export job profile)1k000:00:02000:00:050:00:03

+000:00:01
+50%

100k000:02:17000:04:240:02:17

0:00:00
0%

500k000:05:10000:06:170:04:25

-000:00:45
-14.5%

DE MARC Bib (srs - holdings and items)1k000:00:04000:00:050:00:10

+000:00:06
+150%

100k000:05:13000:05:580:12:41

+000:07:28
+143.2%

500k000:08:58000:08:280:38:53

+000:29:55
+333.7%



Test №1 - №2

Introduction: The Baseline QECP1 Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The objective of these tests was to collect performance measurements for the data-export process across central tenant.

Results: During the test, we observed that the database was running a background process .  for Database, so we deactivated mod-fqm-manager. As a result, the repeated test showed a twofold improvement in performance.

Service CPU Utilization

Here we can see that mod-data-export used 28% CPU Instance Power in spike.

...

DB connections was 830.

Kafka metrics

Image Modified

Image Modified

OpenSearch Data Nodes metrics

...

DB connections was 850.

Kafka metrics


OpenSearch Data Nodes metrics

...

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: ptf-test

    • Number of nodes: 7

    • Version: OpenSearch_2_13_R20240520-P5

  • MSK - fse-tenant

    • 4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

...

  • Test 1: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using Default instances export job profile.
  • Test 2: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using srs - holdings and items job profile.
  • Test 3: Set Instance Count=0 for mod-fqm-manager. Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using Default instances export job profile.
  • Test 4: Set Instance Count=0 for mod-fqm-manager. Manually tested 1k, 100k and 500k records files Data Export started on Main  tenant(fs09000000) using srs - holdings and items job profile.

...