/
PTF - Data Export Test Report (Quesnelia) [non-ECS]

PTF - Data Export Test Report (Quesnelia) [non-ECS]

Jun 07, 2024

Overview

This document contains the results of testing Data Export (MARC BIB) on Quesnelia release with Data Export tests for 1k, 100k, 500k. Three csv files were prepared to run Data Export with Default instances export job profile and srs - holdings and items job profiles.

Ticket: https://folio-org.atlassian.net/browse/PERF-822

 

Summary

Recommendations & Jiras

  • Consider increasing of cpu for mod-data-export module to smoothen spikes with big size files.

Test Results

This table contains durations for jobs with 3 job profiles. 

Profile

CSV  File

DE Duration/Status Quesnelia 

 

 

Result

Status

DE MARC Bib (Default instances export job profile)  

1kDE.csv

00:00:02

COMPLETED

100kDE.csv

00:02:17

COMPLETED

500kDE.csv

00:05:10

COMPLETED

DE MARC Bib (srs - holdings and items)  

1kDE.csv

00:00:04

COMPLETED

100kDE.csv

00:05:13

COMPLETED

500kDE.csv

00:08:58

COMPLETED

Export for Data Import updates(created by script)  

1kDE.csv

00:00:04

COMPLETED

100kDE.csv

00:05:08

COMPLETED

500kDE.csv

00:10:41

COMPLETED

Comparison

This table contains durations comparison between Poppy and Quesnelia releases

Profile

CSV  File

DE Duration/Status Poppy 2 set 

DE Duration/Status Quesnelia 

DE Duration, DELTA Poppy/Quesnelia

 

 

Result

Status

Result

Status

hh:mm:ss

DE MARC Bib (Default instances export job profile)

1kDE.csv

00:00:08

COMPLETED

00:00:02

COMPLETED

00:00:06 - 4 times improvement

 

100kDE.csv

00:15:36

COMPLETED

00:02:17

COMPLETED

00:13:19 - 7 times improvement

 

500kDE.csv

00:57:25

FAIL

00:05:10

COMPLETED

 

DE MARC Bib (srs - holdings and items)

1kDE.csv

00:00:29

COMPLETED

00:00:04

COMPLETED

00:00:25 - 7 times improvement

 

100kDE.csv

00:47:23

COMPLETED

00:05:13

COMPLETED

00:42:10 - 9 times improvement

 

500kDE.csv

04:11:09

FAIL

00:08:58

COMPLETED

 

Service CPU Utilization

Default instances export job profile with 500k file Module CPU mod-data-export-b 434.91 mod-inventory-b 10.79 mod-source-record-manager-b 1.74 mod-source-record-storage-b 1.5 okapi-b 0.96 mod-users-bl-b 0.63 mod-authtoken-b 0.61 mod-inventory-storage-b 0.39 nginx-okapi 0.24 pub-okapi 0.16 srs - holdings and items with 500k file Module CPU mod-data-export-b 296.31 mod-inventory-b 12.79 mod-source-record-manager-b 1.85 mod-source-record-storage-b 1.49 okapi-b 1.1 mod-authtoken-b 0.95 mod-users-bl-b 0.69 mod-inventory-storage-b 0.54 nginx-okapi 0.32 pub-okapi 0.21 TOP 20 modules Module CPU mod-data-export-b 296.31 mod-consortia-b 19.03 mod-inventory-b 12.79 mod-dcb-b 8.17 mod-quick-marc-b 7.99 mod-pubsub-b 7.05 mod-users-b 6.85 mod-audit-b 6.18 mod-kb-ebsco-java-b 5.78 edge-dematic-b 4.5 mod-erm-usage-harvester-b 4.49 mod-organizations-storage-b 4.04 mod-licenses-b 3.51 mod-permissions-b 3.06 mod-data-export-spring-b 2.78 mod-lists-b 2.77 mod-organizations-b 2.74 mod-user-import-b 2.7 mod-tags-b 2.57 mod-sender-b 2.5

Mod-data-export-b

For Default instances export job profile with 100k file used 92% , during exporting 500k file - 434%.

For job profile "Export for Data Import updates" (created by script) - exporting with 100k - 33%, 500k - 202%. 

For srs - holdings and items job profile - 100k - 63%, 500k - 296%.

Memory Utilization

Module Memory mod-data-export-b 97 mod-inventory-b 55 okapi-b 42 mod-source-record-manager-b 42 mod-users-bl-b 32 mod-source-record-storage-b 27 mod-authtoken-b 25 mod-inventory-storage-b 14 nginx-okapi 5 pub-okapi 4 TOP 20 modules srs - holdings and items with 500k file Module Memory mod-data-export-b 97.51 mod-data-export-worker-b 89.71 mod-dcb-b 78.72 mod-consortia-b 78.13 mod-oa-b 76.73 mod-orders-b 65.1 mod-copycat-b 62.06 mod-calendar-b 56.01 mod-agreements-b 55.19 mod-invoice-b 55.1 mod-permissions-b 55.07 mod-circulation-item-b 53.84 mod-erm-usage-harvester-b 52.36 mod-service-interaction-b 51.65 mod-notes-b 50.5 mod-orders-storage-b 50.49 mod-users-b 49.88 mod-tags-b 47.66 mod-inventory-b 45.58 mod-audit-b 44.97

 

This graph contains DE related modules.

Mod-data-export-b use 97%, mod-inventory - 55%. The rest was under 50%

All modules memory consumption graph. Here we see a lot of modules that go with more than 50% consumption. 

DB CPU Utilization

DB utilized with Default instances export job profile 100k file - 17%, 500k file - 25%. For Export for Data Import updates(created by script) job profile - 100k file - 17%, 500k file - 38%. 

For srs - holdings and items job profile - 100k file - 17%, 500k file - 33%. 

DB Connections

DB connections - 1260 in Average. High peaks go with 1360.

DB Load

Default instances export job profile

Export for Data Import updates(test)1 job profile

srs - holdings and items job profile

SQL queries

Default instances export job profile

Export for Data Import updates(test)1 job profile

srs - holdings and items job profile

Top-SQL statement: 

Default instances export job profile

autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

srs - holdings and items job profile

autovacuum: VACUUM fs09000000_mod_data_export.job_executions_export_ids

select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)

select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

Errors / Additional information

Note: inconsistency of data or duplicated records in the file with instances may lead to job failures. Zip file generates on UI side if number of records more than 100.000. 

Methodology/Approach

3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;

All tests were carried out sequentially with each job profile on main tenant fs09000000. 

To get status and time range for export jobs the query used: 

SQL Query
select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'srs - holdings and items' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Default instances export job profile' ORDER BY completed_date DESC select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date FROM [tenant]_mod_data_export.job_executions where job_profile_name = 'Export for Data Import updates(test)1' ORDER BY completed_date DESC

Infrastructure

PTF -environment qcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 2 database  instances, writer/reader

    Name

    Memory GIB

    vCPUs

    max_connections

    db.r6g.xlarge

    32 GiB

    4 vCPUs

    2731

    Data set for fs09000000

    • Instances - 25606331

    • Items       - 26779913

    • Holdings - 25576735

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

QCP1 modules

Module Task Def. Revision Module Version Task Count Mem Hard Limit Mem Soft limit CPU units Xmx MetaspaceSize MaxMetaspaceSize qcp1-pvt Tue Jun 04 07:31:53 UTC 2024 mod-remote-storage 4 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 mod-ncip 4 mod-ncip:1.14.4 2 1024 896 128 768 88 128 mod-finance-storage 4 mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 mod-agreements 4 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 mod-ebsconet 4 mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 mod-organizations 4 mod-organizations:1.9.0 2 1024 896 128 700 88 128 mod-consortia 2 mod-consortia:1.1.0 2 3072 2048 128 2048 512 1024 edge-sip2 2 edge-sip2:3.2.0-SNAPSHOT.209 2 1024 896 128 768 88 128 mod-serials-management 4 mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 mod-settings 4 mod-settings:1.0.3 2 1024 896 200 768 88 128 mod-data-import 7 mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 edge-dematic 4 edge-dematic:2.2.0 1 1024 896 128 768 88 128 mod-search 4 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 mod-inn-reach 2 mod-inn-reach:3.2.0-SNAPSHOT.86 2 3600 3240 1024 2880 512 1024 mod-tags 4 mod-tags:2.2.0 2 1024 896 128 768 88 128 edge-courses 4 edge-courses:1.4.0 2 1024 896 128 768 88 128 mod-authtoken 5 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 mod-inventory-update 4 mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 mod-notify 4 mod-notify:3.2.0 2 1024 896 128 768 88 128 mod-configuration 4 mod-configuration:5.10.0 2 1024 896 128 768 88 128 mod-orders-storage 4 mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 edge-caiasoft 4 edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 mod-login-saml 4 mod-login-saml:2.8.0 2 1024 896 128 768 88 128 mod-erm-usage-harvester 4 mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 mod-licenses 4 mod-licenses:6.0.0 2 2480 2312 128 1792 384 512 mod-gobi 4 mod-gobi:2.8.0 2 1024 896 128 700 88 128 mod-password-validator 4 mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 mod-bulk-operations 4 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 mod-fqm-manager 4 mod-fqm-manager:2.0.1 2 3000 2600 128 2048 384 512 edge-dcb 4 edge-dcb:1.1.0 2 1024 896 128 768 88 128 mod-graphql 5 mod-graphql:1.12.1 2 1024 896 128 768 88 128 mod-finance 4 mod-finance:4.9.0 2 1024 896 128 700 88 128 mod-erm-usage 4 mod-erm-usage:4.7.0 2 1024 896 128 768 88 128 mod-batch-print 5 mod-batch-print:1.1.0 2 1024 896 128 768 88 128 mod-copycat 4 mod-copycat:1.6.0 2 1024 512 128 768 88 128 mod-lists 4 mod-lists:2.0.0 2 3000 2600 128 2048 384 512 mod-entities-links 5 mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 mod-permissions 8 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 pub-edge 3 pub-edge:2023.06.14 2 1024 896 128 768 0 0 mod-orders 4 mod-orders:12.8.0 2 2048 1440 1024 1024 384 512 edge-patron 4 edge-patron:5.1.0 2 1024 896 256 768 88 128 edge-ncip 4 edge-ncip:1.9.2 2 1024 896 128 768 88 128 edge-inn-reach 2 edge-inn-reach:3.1.1-SNAPSHOT.45 2 1024 896 128 768 88 128 mod-users-bl 4 mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 mod-oa 2 mod-oa:2.1.0-SNAPSHOT.62 2 1024 896 128 768 88 128 mod-inventory-storage 4 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 mod-invoice 5 mod-invoice:5.8.0 2 1440 1152 512 922 88 128 mod-user-import 4 mod-user-import:3.8.0 2 1024 896 128 768 88 128 mod-sender 5 mod-sender:1.12.0 2 1024 896 128 768 88 128 edge-oai-pmh 4 edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 mod-data-export-worker 4 mod-data-export-worker:3.2.1 2 3072 2048 1024 2048 384 512 mod-rtac 4 mod-rtac:3.6.0 2 1024 896 128 768 88 128 mod-circulation-storage 4 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 mod-calendar 4 mod-calendar:3.1.0 2 1024 896 128 768 88 128 mod-source-record-storage 4 mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 mod-event-config 4 mod-event-config:2.7.0 2 1024 896 128 768 88 128 mod-courses 4 mod-courses:1.4.10 2 1024 896 128 768 88 128 mod-circulation-item 4 mod-circulation-item:1.0.0 2 1024 896 128 0 0 0 mod-inventory 4 mod-inventory:20.2.0 2 2880 2592 1024 1814 384 512 mod-email 4 mod-email:1.17.0 2 1024 896 128 768 88 128 mod-pubsub 4 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 mod-circulation 4 mod-circulation:24.2.0 2 2880 2592 1536 1814 384 512 mod-di-converter-storage 4 mod-di-converter-storage:2.2.0 2 1024 896 128 768 88 128 edge-rtac 4 edge-rtac:2.7.1 2 1024 896 128 768 88 128 edge-orders 4 edge-orders:3.0.0 2 1024 896 128 768 88 128 mod-users 5 mod-users:19.3.1 2 1024 896 128 768 88 128 mod-template-engine 4 mod-template-engine:1.20.0 2 1024 896 128 768 88 128 mod-patron-blocks 4 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 mod-audit 4 mod-audit:2.9.0 2 1024 896 128 768 88 128 edge-fqm 4 edge-fqm:2.0.0 2 1024 896 128 768 88 128 mod-source-record-manager 5 mod-source-record-manager:3.9.0-SNAPSHOT.330 2 5600 5000 2048 3500 384 512 nginx-edge 3 nginx-edge:2023.06.14 2 1024 896 128 0 0 0 mod-quick-marc 4 mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 nginx-okapi 3 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 okapi-b 4 okapi:5.3.0 3 1684 1440 1024 922 384 512 mod-feesfines 4 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 mod-invoice-storage 4 mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 mod-dcb 5 mod-dcb:1.1.0 2 1024 896 128 768 88 128 mod-service-interaction 4 mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 mod-data-export 11 mod-data-export:5.0.4 1 2048 1524 1024 0 0 0 mod-patron 4 mod-patron:6.1.0 2 1024 896 128 768 88 128 mod-oai-pmh 4 mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 edge-connexion 4 edge-connexion:1.2.0 2 1024 896 128 768 88 128 mod-kb-ebsco-java 4 mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 mod-notes 4 mod-notes:5.2.0 2 1024 896 128 952 384 512 mod-data-export-spring 4 mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 mod-organizations-storage 4 mod-organizations-storage:4.7.0 2 1024 896 128 700 88 128 mod-login 4 mod-login:7.11.0 2 1440 1298 1024 768 384 512 pub-okapi 3 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 mod-eusage-reports 4 mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128

Data Export related modules

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

qcp1-pvt

 

 

 

 

 

 

 

 

 

Tue Jun 04 07:31:53 UTC 2024

 

 

 

 

 

 

 

 

 

mod-authtoken

5

mod-authtoken:2.15.1

2

1440

1152

512

922

88

128

mod-users-bl

4

mod-users-bl:7.7.0

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.1.0

2

4096

3690

2048

3076

384

512

mod-source-record-storage

4

mod-source-record-storage:5.8.0

2

5600

5000

2048

3500

384

512

mod-inventory

4

mod-inventory:20.2.0

2

2880

2592

1024

1814

384

512

mod-source-record-manager

5

mod-source-record-manager:3.9.0-SNAPSHOT.330

2

5600

5000

2048

3500

384

512

nginx-okapi

3

nginx-okapi:2023.06.14

2

1024

896

128

0

0

0

okapi-b

4

okapi:5.3.0

3

1684

1440

1024

922

384

512

mod-data-export

11

mod-data-export:5.0.4

1

2048

1524

1024

0

0

0

pub-okapi

3

pub-okapi:2023.06.14

2

1024

896

128

768

0

0