/
[Sunflower] [non-ECS] Multi-tenants DI

[Sunflower] [non-ECS] Multi-tenants DI

Overview

  • This document contains the results of testing Data Import for MARC Bibliographic records with an update job profile in the Sunflower release on secp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs Eureka environment

https://folio-org.atlassian.net/browse/PERF-1113

  • During the first run, tests #2 and #3 failed due to authentication issues for tenants fs07000001 and fs07000002. These authentication blockers have since been resolved. Detailed steps and actions taken to fix the issue can be found here documentations

Summary

Three DI update job tests with 25,000 records were performed on the secp1 cluster.

We compared the Sunflower and Quesnelia releases and observed both performance gains and losses in the DI update job durations:

  • Single tenant: On average, jobs ran 9 minutes and 40 seconds slower in Sunflower.

  • Two tenants: Jobs were 10 minutes slower in Sunflower.

  • Three tenants: Jobs ran 45 minutes faster in Sunflower compared to Quesnelia.

Resource Utilization (Sunflower):

  • mod-di-converter-storage CPU usage: 203%

  • mod-inventory CPU usage: 128%

  • Database CPU usage: 99%

  • Average DB connections: 1,542

 

Test Results

Profile

Test #

Tenant

MARC File

DI Duration Sunflower (hh:mm:ss)

Results

Profile

Test #

Tenant

MARC File

DI Duration Sunflower (hh:mm:ss)

Results

DI MARC Bib Update (PTF - Updates Success - 6)

1

fs09000000

25K.mrc

00:25:35

Completed

2

fs09000000

25K.mrc

00:30:15

Completed

fs07000001

25K.mrc

00:25:09

Completed

3

fs09000000

25K.mrc

00:26:35

Completed

fs07000001

25K.mrc

00:25:36

Completed

fs07000002

25K.mrc

00:15:15

Completed

 

Comparison

Test #

Sunflower

Quesnelia

DI Delta
per tenant
(hh:mm:ss) Sunflower/Quesnelia

DI Delta
per test
(hh:mm:ss) Sunflower/Quesnelia

Profile

Tenant

DI Duration Sunflower

per tenant

(hh:mm:ss)

DI Duration Sunflower

per test

(hh:mm:ss)

Profile

Tenant

DI Duration Quesnelia (hh:mm:ss)

DI Duration Quesnelia

per test

(hh:mm:ss)

№1

25K.mrc

DI MARC Bib Update

(PTF - Updates Success - 6)

fs09000000

00:25:35

00:25:35

DI MARC Bib Update

(PTF - Updates Success - 1)

fso9000000

00:15:55

00:15:55

+00:09:40

+00:09:40

№2

25K.mrc

fs09000000

00:30:15

00:55:24

fso9000000

00:17:06

00:45:23

+00:13:09

+0:10:01

fs07000001

00:25:09

fso7000001

00:28:17

-00:03:08

№3

25K.mrc

fs09000000

00:26:35

01:06:59

fso9000000

00:33:57

1:51:59

-00:07:22

-00:45:00

fs07000001

00:25:36

fso7000001

00:36:07

-00:10:31

fs07000002

00:15:15

fso7000002

00:41:55

-00:26:40

Resource utilization for Test #1 and Test #2

CPU RAM mod-di-converter-storage-b 65% mod-applications-b 82% mod-inventory-b 53% mod-tenant-entitlements-b 69% mod-audit-b 52% mod-scheduler-b 67% mod-source-record-storage-b 43% mod-roles-keycloak-b 64% mod-inventory-storage-b 37% mod-finance-b 62% mod-inn-reach-b 26% mod-tenants-b 61% mod-quick-marc-b 21% mod-audit-b 60% mod-remote-storage-b 18% mod-orders-storage-b 59% mod-circulation-storage-b 12.% mod-invoice-b 58% mod-source-record-manager-b 11% mod-users-keycloak-b 58%

Service CPU Utilization

Here we can see that mod-di-converter-storage-b module used 145% CPU, mod-source-record-storage-b used 88% CPU and mod-inventory-b used 81% CPU

image-20250508-100408.png

Service Memory Utilization

Here we can see that all modules show a stable trend.

image-20250507-113908.png

DB CPU Utilization

DB CPU in the average was 99%.

image-20250508-104949.png

DB Connections

DB connections was 1496.

image-20250508-105110.png

DB load

image-20250508-105428.png

Top SQL-queries

image-20250508-105601.png

 

#

TOP 5 SQL statements

#

TOP 5 SQL statements

1

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

2

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17) ON CONFLICT DO NOTHING

3

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

4

WITH cte AS (SELECT id, name, name_type_id, authority_id, last_updated_date FROM fs09000000_mod_search.contributor WHERE last_updated_date > $1 ORDER BY last_updated_date ) SELECT c.id, c.name, c.name_type_id, c.authority_id, c.last_updated_date, json_agg( CASE WHEN sub.instance_count

5

UPDATE fs09000000_mod_inventory_storage.holdings_record SET jsonb = $1::jsonb WHERE id=?

Resource utilization for Test #3

CPU RAM mod-di-converter-storage-b 203% mrg-applications-b 83% mod-inventory-b 128% mgr-tenant-entitlements-b 69% mod-source-record-storage-b 111% mod-scheduler-b 68% mod-inventory-storage-b 101% mod-roles-keycloak-b 64% mod-audit-b 65% mod-finance-b 63% mod-inn-reach-b 34% mrg-tenants-b 62% mod-quick-marc-b 32% mod-data-import-b 61% mod-remote-storage-b 18% mod-audit-b 61% mod-circulation-storage-b 16% mod-orders-storage-b 59% mod-orders-storage-b 15% mod-users-keycloak-b 59%

Service CPU Utilization

Here we can see that mod-di-converter-storage-b module used 203% CPU in average, mod-inventory-b used 129% CPU and mod-source-record-storage-b used 112% CPU

Untitled-20250508-110603.jpg

Service Memory Utilization

Here we can see that all modules show a stable trend.

image-20250507-131230.png

DB CPU Utilization

DB CPU was 97%.

image-20250507-131438.png

DB Connections

DB connections was 1588.

image-20250507-131609.png

DB load

image-20250508-110028.png

Top SQL-queries

image-20250508-110053.png

 

#

TOP 5 SQL statements

#

TOP 5 SQL statements

1

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

2

INSERT INTO fs07000002_mod_search.contributor (id, name, name_type_id, authority_id) VALUES ($1, $2, $3, $4) ON CONFLICT (id) DO UPDATE SET last_updated_date = CURRENT_TIMESTAMP

3

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

4

INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17) ON CONFLICT DO NOTHING

5

INSERT INTO fs09000000_mod_search.item (id, tenant_id, instance_id, holding_id, json) VALUES ($1::uuid, $2, $3::uuid, $4::uuid, $5::jsonb) ON CONFLICT (id, tenant_id) DO UPDATE SET instance_id = EXCLUDED.instance_id, holding_id = EXCLUDED.holding_id, tenant_id = EXCLUDED.tenant_id, json = EXCLUDED.json

Appendix

Infrastructure

PTF - environment Sunflower (secp1)

  • Name

Memory GIB

vCPUs

  • Name

Memory GIB

vCPUs

  • db.r7g.xlarge

32 GiB4

4 vCPUs

  • MSK fse-test

    • 4 kafka.m7g.xlarge brokers in 2 zones

    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

Module Task Def. Revision Module Version Task Count Mem Hard Limit Mem Soft limit CPU units Xmx MetaspaceSize MaxMetaspaceSize mod-remote-storage 3 mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 mod-ncip 3 mod-ncip:1.14.4 2 1024 896 128 768 88 128 mod-finance-storage 3 mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 mod-agreements 3 mod-agreements:7.0.0 2 1592 1488 128 0 0 0 mod-ebsconet 3 mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 mod-organizations 3 mod-organizations:1.9.0 2 1024 896 128 700 88 128 mod-consortia 2 mod-consortia:1.1.0 2 3072 2048 128 2048 512 1024 edge-sip2 2 edge-sip2:3.2.0-SNAPSHOT.209 2 1024 896 128 768 88 128 mod-serials-management 3 mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 mod-settings 3 mod-settings:1.0.3 2 1024 896 200 768 88 128 mod-data-import 6 mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 edge-dematic 3 edge-dematic:2.2.0 1 1024 896 128 768 88 128 mod-search 3 mod-search:3.2.0 2 2592 2480 2048 1440 512 1024 mod-inn-reach 2 mod-inn-reach:3.2.0-SNAPSHOT.86 2 3600 3240 1024 2880 512 1024 mod-tags 3 mod-tags:2.2.0 2 1024 896 128 768 88 128 edge-courses 3 edge-courses:1.4.0 2 1024 896 128 768 88 128 mod-authtoken 3 mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 mod-inventory-update 3 mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 mod-notify 3 mod-notify:3.2.0 2 1024 896 128 768 88 128 mod-configuration 3 mod-configuration:5.10.0 2 1024 896 128 768 88 128 mod-orders-storage 3 mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 edge-caiasoft 3 edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 mod-login-saml 3 mod-login-saml:2.8.0 2 1024 896 128 768 88 128 mod-erm-usage-harvester 3 mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 mod-licenses 3 mod-licenses:6.0.0 2 2480 2312 128 1792 384 512 mod-gobi 3 mod-gobi:2.8.0 2 1024 896 128 700 88 128 mod-password-validator 3 mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 mod-bulk-operations 3 mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 mod-fqm-manager 3 mod-fqm-manager:2.0.1 2 3000 2600 128 2048 384 512 edge-dcb 3 edge-dcb:1.1.0 2 1024 896 128 768 88 128 mod-graphql 4 mod-graphql:1.12.1 2 1024 896 128 768 88 128 mod-finance 3 mod-finance:4.9.0 2 1024 896 128 700 88 128 mod-erm-usage 3 mod-erm-usage:4.7.0 2 1024 896 128 768 88 128 mod-batch-print 4 mod-batch-print:1.1.0 2 1024 896 128 768 88 128 mod-copycat 3 mod-copycat:1.6.0 2 1024 512 128 768 88 128 mod-lists 3 mod-lists:2.0.0 2 3000 2600 128 2048 384 512 mod-entities-links 4 mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 mod-permissions 6 mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 pub-edge 3 pub-edge:2023.06.14 2 1024 896 128 768 0 0 mod-orders 3 mod-orders:12.8.0 2 2048 1440 1024 1024 384 512 edge-patron 3 edge-patron:5.1.0 2 1024 896 256 768 88 128 edge-ncip 3 edge-ncip:1.9.2 2 1024 896 128 768 88 128 edge-inn-reach 2 edge-inn-reach:3.1.1-SNAPSHOT.45 2 1024 896 128 768 88 128 mod-users-bl 3 mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 mod-oa 2 mod-oa:2.1.0-SNAPSHOT.62 2 1024 896 128 768 88 128 mod-inventory-storage 3 mod-inventory-storage:27.1.0 2 4096 3690 2048 3076 384 512 mod-invoice 4 mod-invoice:5.8.0 2 1440 1152 512 922 88 128 mod-user-import 3 mod-user-import:3.8.0 2 1024 896 128 768 88 128 mod-sender 4 mod-sender:1.12.0 2 1024 896 128 768 88 128 edge-oai-pmh 3 edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 mod-data-export-worker 3 mod-data-export-worker:3.2.1 2 3072 2048 1024 2048 384 512 mod-rtac 3 mod-rtac:3.6.0 2 1024 896 128 768 88 128 mod-circulation-storage 3 mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 mod-calendar 3 mod-calendar:3.1.0 2 1024 896 128 768 88 128 mod-source-record-storage 3 mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 mod-event-config 3 mod-event-config:2.7.0 2 1024 896 128 768 88 128 mod-courses 3 mod-courses:1.4.10 2 1024 896 128 768 88 128 mod-circulation-item 3 mod-circulation-item:1.0.0 2 1024 896 128 0 0 0 mod-inventory 3 mod-inventory:20.2.0 2 2880 2592 1024 1814 384 512 mod-email 3 mod-email:1.17.0 2 1024 896 128 768 88 128 mod-pubsub 3 mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 mod-circulation 3 mod-circulation:24.2.0 2 2880 2592 1536 1814 384 512 mod-di-converter-storage 3 mod-di-converter-storage:2.2.0 2 1024 896 128 768 88 128 edge-rtac 3 edge-rtac:2.7.1 2 1024 896 128 768 88 128 edge-orders 3 edge-orders:3.0.0 2 1024 896 128 768 88 128 mod-users 4 mod-users:19.3.1 2 1024 896 128 768 88 128 mod-template-engine 3 mod-template-engine:1.20.0 2 1024 896 128 768 88 128 mod-patron-blocks 3 mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 mod-audit 3 mod-audit:2.9.0 2 1024 896 128 768 88 128 edge-fqm 3 edge-fqm:2.0.0 2 1024 896 128 768 88 128 mod-source-record-manager 3 mod-source-record-manager:3.8.0 2 5600 5000 2048 3500 384 512 nginx-edge 3 nginx-edge:2023.06.14 2 1024 896 128 0 0 0 mod-quick-marc 3 mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 nginx-okapi 3 nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 okapi-b 4 okapi:5.3.0 3 1684 1440 1024 922 384 512 mod-feesfines 3 mod-feesfines:19.1.0 2 1024 896 128 768 88 128 mod-invoice-storage 3 mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 mod-dcb 4 mod-dcb:1.1.0 2 1024 896 128 768 88 128 mod-service-interaction 3 mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 mod-data-export 8 mod-data-export:5.0.1 1 2048 1524 1024 0 0 0 mod-patron 3 mod-patron:6.1.0 2 1024 896 128 768 88 128 mod-oai-pmh 3 mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 edge-connexion 3 edge-connexion:1.2.0 2 1024 896 128 768 88 128 mod-kb-ebsco-java 3 mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 mod-notes 3 mod-notes:5.2.0 2 1024 896 128 952 384 512 mod-data-export-spring 3 mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 mod-organizations-storage 3 mod-organizations-storage:4.7.0 2 1024 896 128 700 88 128 mod-login 3 mod-login:7.11.0 2 1440 1298 1024 768 384 512 pub-okapi 3 pub-okapi:2023.06.14 2 1024 896 128 768 0 0 mod-eusage-reports 3 mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128

Methodology/Approach

Before each test files for Marc Autority creation must created using python script https://github.com/folio-org/perf-testing/tree/master/workflows-scripts/data-import/MARC_authority

DI tests scenario (DI Autority Bib Create) were started from UI on Sunflower (secp1) env with Kafka consolidated topics and file splitting features enabled.

Test set

  • Test 1: Manually tested 25k records files DI started on one tenant only.

  • Test 2: Manually tested 25k records files DI started on 2 tenants concurrently, step 30%.

  • Test 3: Manually tested 25k records files DI started on 3 tenants concurrently, step 30%.