PTF - Data Export Test Report (Quesnelia - Eureka)

PTF - Data Export Test Report (Quesnelia - Eureka)

Overview

  • This document contains the results of testing Data Export (MARC BIB) on the Eureka release of Quesnelia FOLIO.  The goal is to compare the performances of Data Export on Eureka and non-Eureka environments and to highlight any observable differences in the KPIs.

https://folio-org.atlassian.net/browse/PERF-866 

Summary

  • Data Export tests finished successfully on the Eureka environment using the "Default instances export job profile" and "srs - holdings and items job profile." Data Export tests were run on Central tenants.

  • When comparing QECP1 Eureka with Quesnelia ECS, NON-ECS environments:

      • Data Export duration for the "Default instances export job profile" was almost the same across environments.

      • Data Export duration for the "srs - holdings and items job profile" showed a significant degradation: around 2.5 times longer for the 1k and 100k files, and around 5 times longer for the 500k file, and no clear reasons were identified.

  • During Tests №1 and №2, we noticed some background processes for Database, so we deactivated mod-fqm-manager. As a result, the repeated test showed a twofold improvement in performance.

  • Data Export jobs were getting stuck and returned a [401 Unauthorized] error with the message: "errors [{"type":"UnauthorizedException","code":"authorization_error","message":"Unauthorized"}]. To resolve this, we increased the token lifespan by modifying the KC_CONFIG_TTL parameter for mod-login-keycloak from 360s to 3600s and KC_ADMIN_TOKEN_TTL from 410s to 4100s:

      • {"name": "KC_CONFIG_TTL","value": "3600s"}

      • {"name": "KC_ADMIN_TOKEN_TTL", "value": "4100s"}

 

Test Results

This table contains durations for Data Export with 2 job profiles. 

Profile

CSV  File

Central Tenant (fs09000000)

Result
Test Set 1

Result
Test Set 2

Status

DE MARC Bib (Default instances export job profile)

1k.csv

0:00:07

0:00:03

COMPLETED

100k.csv

0:06:03

0:02:17

COMPLETED

500k.csv

0:09:03

0:04:25

COMPLETED

DE MARC Bib (srs - holdings and items)

1k.csv

0:00:13

0:00:10

COMPLETED

100k.csv

0:20:25

0:12:41

COMPLETED

500k.csv

0:38:21

0:38:53

COMPLETED

Comparison

This table contains durations comparison between Quesnelia ECS, NON-ECS and Eureka environments.

Profile

Number of records

Quesnelia NON-ECS QCP1

Quesnelia ECS
QCON

Quesnelia Eureka
QECP1

DE Duration, DELTA QCP1/QECP1

h:mm:ss

h:mm:ss

h:mm:ss

h:mm:ss / percent

DE MARC Bib (Default instances export job profile)

1k

0:00:02

0:00:05

0:00:03

+0:00:01
+50%

100k

0:02:17

0:04:24

0:02:17

0:00:00
0%

500k

0:05:10

0:06:17

0:04:25

-0:00:45
-14.5%

DE MARC Bib (srs - holdings and items)

1k

0:00:04

0:00:05

0:00:10

+0:00:06
+150%

100k

0:05:13

0:05:58

0:12:41

+0:07:28
+143.2%

500k

0:08:58

0:08:28

0:38:53

+0:29:55
+333.7%

 

Test №1 - №2

Introduction: The Baseline QECP1 Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The objective of these tests was to collect performance measurements for the data-export process across central tenant.

Results: During the test, we observed that the database was running a background process for Database, so we deactivated mod-fqm-manager. As a result, the repeated test showed a twofold improvement in performance.

Service CPU Utilization

Here we can see that mod-data-export used 28% CPU Instance Power in spike.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU spike was 68%.

DB Connections

DB connections was 830.

Kafka metrics

OpenSearch Data Nodes metrics

DB load

Top SQL-queries

 

 

Test №3 - №4

Introduction: The Baseline QECP1 Environment configuration was applied, and CPU=0 was set for all modules.

Objective: The objective of these tests has to repeat previous tests after deactivated mod-fqm-manager.

Results: Results were collect for central tenant without any background processes for Database. 

Instance CPU Utilization

Service CPU Utilization

Here we can see that mod-data-export used 28% CPU Instance Power in spike.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU spike was 22%.

DB Connections

DB connections was 850.

Kafka metrics

 

OpenSearch Data Nodes metrics

DB load

Top SQL-queries

 

 

Appendix

Infrastructure

PTF - environment Quesnelia [Eureka] (qecp1)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: ptf-test

    • Number of nodes: 7

    • Version: OpenSearch_2_13_R20240520-P5

  • MSK - fse-tenant

    • 4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size mod-remote-storage 3 mod-remote-storage:3.2.0 2 4920 4472 0 3960 512 512 mod-remote-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-ncip 3 mod-ncip:1.14.5 2 1024 896 0 768 88 128 mod-ncip - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-finance-storage 3 mod-finance-storage:8.6.1 2 1024 896 0 700 88 128 mod-finance-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-agreements 3 mod-agreements:7.0.6 2 1592 1488 0 0 0 0 mod-agreements - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-ebsconet 3 mod-ebsconet:2.2.0 2 1248 1024 0 700 128 256 mod-ebsconet - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-organizations 3 mod-organizations:1.9.2 2 1024 896 0 768 88 128 mod-organizations - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-sip2 2 edge-sip2:3.2.5 2 1024 896 0 768 88 128 mod-settings 3 mod-settings:1.0.3 2 1024 896 0 768 88 128 mod-settings - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-serials-management 3 mod-serials-management:1.0.3 2 2480 2312 0 1792 384 512 mod-serials-management - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-search 3 mod-search:3.2.6 2 2592 2480 0 1440 512 1024 mod-search - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-data-import 3 mod-data-import:3.1.1 1 2048 1844 0 1292 384 512 mod-data-import - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-dematic 2 edge-dematic:2.2.3 1 1024 896 0 768 88 128 mod-tags 3 mod-tags:2.2.0 2 1024 896 0 768 88 128 mod-tags - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-courses 2 edge-courses:1.4.3 2 1024 896 0 768 88 128 mod-notify 3 mod-notify:3.2.0 2 1024 896 0 768 88 128 mod-notify - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-inventory-update 3 mod-inventory-update:3.3.1 2 1024 896 0 768 88 128 mod-inventory-update - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-configuration 3 mod-configuration:5.10.0 2 1024 896 0 768 88 128 mod-configuration - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-orders-storage 3 mod-orders-storage:13.7.4 2 1024 896 0 700 88 128 mod-orders-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 folio-keycloak 3 folio-keycloak:25.0.3 1 2400 2048 0 2048 0 0 edge-caiasoft 2 edge-caiasoft:2.2.4 2 1024 896 0 768 88 128 mod-licenses 3 mod-licenses:6.0.3 2 2480 2312 0 1792 384 512 mod-licenses - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-gobi 3 mod-gobi:2.8.1 2 1024 896 0 768 88 128 mod-gobi - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-password-validator 3 mod-password-validator:3.2.1 2 1440 1298 0 768 384 512 mod-password-validator - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-fqm-manager 3 mod-fqm-manager:2.0.5 0 1024 896 0 768 88 128 mod-fqm-manager - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-bulk-operations 3 mod-bulk-operations:2.0.2 2 3072 2600 0 1536 384 512 mod-bulk-operations - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-graphql 3 mod-graphql:1.12.1 2 1024 896 0 768 88 128 mod-graphql - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-finance 3 mod-finance:4.9.0 2 1024 896 0 768 88 128 mod-finance - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-roles-keycloak 3 mod-roles-keycloak:1.4.5 2 1024 896 0 512 88 256 mod-roles-keycloak - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-batch-print 3 mod-batch-print:1.1.0 2 1024 896 0 768 88 128 mod-batch-print - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-copycat 3 mod-copycat:1.6.0 2 1024 896 0 768 88 128 mod-copycat - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-lists 4 mod-lists:2.0.6 2 1024 896 0 768 88 128 mod-lists - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-entities-links 3 mod-entities-links:3.0.2 2 2592 2480 0 1440 0 1024 mod-entities-links - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-permissions 6 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 mod-permissions - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 pub-edge 1 pub-edge:2023.06.14 2 1024 896 0 768 0 0 mod-orders 3 mod-orders:12.8.8 2 2048 1740 0 1024 384 512 mod-orders - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 kong 9 kong:3.7.1 1 6144 6144 2048 768 88 128 edge-patron 2 edge-patron:5.1.2 2 1024 896 256 768 88 128 mod-users-keycloak 3 mod-users-keycloak:1.5.3 2 1024 896 0 512 88 256 mod-users-keycloak - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-ncip 2 edge-ncip:1.10.1 2 1024 896 0 768 88 128 mgr-applications 2 mgr-applications:1.3.3 1 1024 896 512 512 88 256 mod-users-bl 3 mod-users-bl:7.8.0 2 1440 1152 0 922 88 128 mod-users-bl - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mgr-tenants 2 mgr-tenants:1.3.3 1 1024 896 512 512 88 256 mod-invoice 3 mod-invoice:5.8.2 2 1440 1152 0 922 88 128 mod-invoice - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-inventory-storage 4 mod-inventory-storage:27.1.4 2 4096 3690 0 3076 384 512 mod-inventory-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 256 0 0 mod-user-import 3 mod-user-import:3.8.0 2 1024 896 0 768 88 128 mod-user-import - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-sender 3 mod-sender:1.12.0 2 1024 896 0 768 88 128 mod-sender - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-oai-pmh 2 edge-oai-pmh:2.9.2 2 1512 1360 1024 1440 384 512 mod-login-keycloak 8 mod-login-keycloak:1.4.4 2 1024 896 0 512 88 256 mod-login-keycloak - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mgr-tenant-entitlements 2 mgr-tenant-entitlements:1.2.4 1 1024 896 512 512 88 256 mod-data-export-worker 3 mod-data-export-worker:3.2.4 2 3072 2800 0 2048 384 512 mod-data-export-worker - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-rtac 4 mod-rtac:3.6.0 2 1024 896 0 768 88 128 mod-rtac - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 256 0 0 mod-circulation-storage 4 mod-circulation-storage:17.2.3 2 2880 2592 0 1814 384 512 mod-circulation-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 256 0 0 mod-source-record-storage 4 mod-source-record-storage:5.8.6 2 5600 5000 0 3500 384 512 mod-source-record-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-scheduler 3 mod-scheduler:1.2.3 2 1024 896 0 512 88 256 mod-scheduler - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-calendar 4 mod-calendar:3.1.0 2 2048 1536 0 768 88 128 mod-calendar - Sidecar 1 N/A folio-module-sidecar:1.0.0.317 N/A 665 512 0 128 0 0 mod-event-config 3 mod-event-config:2.7.1 2 1024 896 0 768 88 128 mod-event-config - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-courses 3 mod-courses:1.4.10 2 1024 896 0 768 88 128 mod-courses - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-inventory 5 mod-inventory:20.2.8-SNAPSHOT.580 2 2881 2592 0 1814 384 512 mod-inventory - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-email 3 mod-email:1.17.0 2 2800 2550 0 1800 384 512 mod-email - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-di-converter-storage 3 mod-di-converter-storage:2.2.2 2 1024 896 0 768 88 128 mod-di-converter-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-pubsub 3 mod-pubsub:2.14.0 2 1536 1440 0 922 384 512 mod-pubsub - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-circulation 4 mod-circulation:24.2.5 2 2880 2592 0 1814 384 512 mod-circulation - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 256 0 0 edge-orders 2 edge-orders:3.0.3 2 1024 896 0 768 88 128 edge-rtac 2 edge-rtac:2.7.3 2 1024 896 0 768 88 128 mod-template-engine 3 mod-template-engine:1.20.0 2 1024 896 0 768 88 128 mod-template-engine - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-users 3 mod-users:19.3.1 2 1024 896 0 768 88 128 mod-users - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-patron-blocks 3 mod-patron-blocks:1.10.0 2 1024 896 0 768 88 128 mod-patron-blocks - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-audit 3 mod-audit:2.9.0 2 1024 896 0 768 88 128 mod-audit - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-fqm 2 edge-fqm:2.0.2 2 1024 896 0 768 88 128 mod-source-record-manager 3 mod-source-record-manager:3.8.6 2 5600 5000 0 3500 384 512 mod-source-record-manager - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 nginx-edge 1 nginx-edge:2023.06.14 1 1024 896 0 0 0 0 mod-quick-marc 3 mod-quick-marc:5.1.1 1 2288 2176 0 1664 384 512 mod-quick-marc - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-feesfines 3 mod-feesfines:19.1.0 2 1024 896 0 768 88 128 mod-feesfines - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-invoice-storage 3 mod-invoice-storage:5.8.2 2 1872 1536 0 1024 384 512 mod-invoice-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-service-interaction 3 mod-service-interaction:4.0.2 2 2048 1844 0 1290 384 512 mod-service-interaction - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-data-export 11 mod-data-export:5.0.4 1 2592 2480 0 1440 88 1024 mod-data-export - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-patron 3 mod-patron:6.1.0 2 1024 896 0 768 88 128 mod-patron - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-oai-pmh 3 mod-oai-pmh:3.13.2 2 4096 3690 0 3076 384 512 mod-oai-pmh - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-connexion 2 edge-connexion:1.3.1 2 1024 896 0 768 88 128 mod-kb-ebsco-java 3 mod-kb-ebsco-java:4.0.0 2 1024 896 0 768 88 128 mod-kb-ebsco-java - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-notes 3 mod-notes:5.2.0 2 1024 896 0 952 384 512 mod-notes - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-organizations-storage 3 mod-organizations-storage:4.7.0 2 1024 896 0 768 88 128 mod-organizations-storage - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 mod-data-export-spring 3 mod-data-export-spring:3.2.2 1 2048 1844 0 1536 384 512 mod-data-export-spring - Sidecar 1 N/A folio-module-sidecar:1.0.0.312 N/A 665 512 0 128 0 0 edge-erm 2 edge-erm:1.2.1 2 1024 896 0 768 88 128

 

Methodology/Approach

Data Export tests scenario using the profiles Default instances export job profile and srs - holdings and items were started from UI on Quesnelia (qecp1) Eureka environment.

Test set:

  • Test 1: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using Default instances export job profile.

  • Test 2: Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using srs - holdings and items job profile.

  • Test 3: Set Instance Count=0 for mod-fqm-manager. Manually tested 1k, 100k and 500k records files Data Export started on Main tenant(fs09000000) using Default instances export job profile.

  • Test 4: Set Instance Count=0 for mod-fqm-manager. Manually tested 1k, 100k and 500k records files Data Export started on Main  tenant(fs09000000) using srs - holdings and items job profile.

To get status and time range for export jobs the query used: 

SQL Query
select jsonb->'exportedFiles'->0->>'fileName' as fileName, job_profile_name,exported, started_date,completed_date, completed_date - started_date as duration ,status from fs09000000_mod_data_export.job_executions where started_date > '2024-10-22' order by started_date desc limit 10;