1 Overview
2 Summary
- 2.1 Recommendations & Jiras
3 Results
- 3.1 Memory Utilization
  - 3.1.1 Service memory usage for DI creates and updates
- 3.2 CPU Utilization
  - 3.2.1 Service CPU utilisation for DI creates and updates
- 3.3 RDS Metrics
  - 3.3.1 DB CPU utilization for DI creates and updates
  - 3.3.2 DB load for DI creates and updates
- 3.4 Slow query found in mod-search
- 3.5 MSK CPU usage
4 Appendix
- 4.1 Infrastructure
- 4.2 Cluster Resources - rcp1-pvt
- 4.3 Methodology/Approach

Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Sunflower release [non-ECS].

Note: Starting from Sunflower release all FOLIO environments will be Eureka.

In scope is testing of 5K,10K,25K,50K, 100K data import create and update.
Ticket: PERF-1108: [Sunflower] [non-ECS] [Data import] Update and Create MARC BIB RecordsAwaiting release

Summary

All tests passed successfully.
Visible significant improvements on mod-search side (in Ramsons long running query caused performance degradation 50-100%). Now duration of data imports is faster for creates and more or les the same for updates.
On mod-search schema side deadlocks observed during data import creates and updates, however deadlocks does not affect completion of DI.

Recommendations & Jiras

mod-search deadlocks ticketMSEARCH-932: Simplify bulk failure error logsClosed

Results

Test #	Data-import test	Profile	Duration Sunflower (secp1)	Duration Ramsons (rcp1) mod-search enabled	Duration Quesnelia (qcp1)	Status

Test #	Data-import test	Profile	Duration Sunflower (secp1)	Duration Ramsons (rcp1) mod-search enabled	Duration Quesnelia (qcp1)	Status
1	5k MARC BIB Create	PTF - Create 2	2 min 4 s	3 min 7 s	-	Completed
	10k MARC BIB Create	PTF - Create 2	4 min 43 s	6 min 15 s	6 minutes	Completed
2	25k MARC BIB Create	PTF - Create 2	10 min	17 min	13 min 41 sec	Completed
3	50k MARC BIB Create	PTF - Create 2	21 min	41 min 25 s	21 min 59 sec	Completed
4	100k MARC BIB Create	PTF - Create 2	42 min 46 s	1 hr 19 min	40 min 16 sec	Completed
	5k MARC BIB Update	PTF - Updates Success - 6	6 min 18 s	6 min 33 s	-	Completed
6	10k MARC BIB Update	PTF - Updates Success - 6	6 min 4 s	11 min 14 s	10 min 27 sec	Completed
7	25k MARC BIB Update	PTF - Updates Success - 6	31 min	28 min 43 s	23 min 16 sec	Completed
8	50k MARC BIB Update	PTF - Updates Success - 6	1 hr 8 min	58 min 30 s	40 min 52 sec	Completed
9	100k MARC BIB Update	PTF - Updates Success - 6	2 hr 5 min	2 hr 14 min	1 hrs 2 min	Completed

Memory Utilization

Memory utilization showed stable trend during DI creates and updates tests. No sudden crashes or unexpected growth of memory usage were observed.

All services didn’t exceed 80% of memory usage. Most used module is mod-permissions, however after each test it returns to normal condition.

Service memory usage for DI creates and updates

CPU Utilization

CPU utilization is stable and predictable for all modules during all tests.

Service CPU utilisation for DI creates and updates

RDS Metrics

As expected CPU usage of DB is high (as usual during data import process).

DB CPU utilization for DI creates and updates

DB load for DI creates and updates

Slow query detected from mod-search side that affecting performance significantly and it’s still a slowest query:

Data volume:

search.instance - 4 109 321

search.instance_contributor - 8 327 231

Slow query found in mod-search

WITH cte AS (SELECT id,
	                    name,
	                    name_type_id,
	                    authority_id,
	                    last_updated_date
	             FROM fs09000000_mod_search.contributor
	             WHERE last_updated_date > $1
	             ORDER BY last_updated_date
	             )
	SELECT c.id,
	       c.name,
	       c.name_type_id,
	       c.authority_id,
	       c.last_updated_date,
	       json_agg(
	               CASE
	                                WHEN sub.instance_count IS NULL THEN NULL
	                                ELSE json_build_object(
	                       'count', sub.instance_count,
	                       'typeId', sub.type_ids,
	                       'shared', sub.shared,
	                       'tenantId', sub.tenant_id
	               )
	               END
	       ) AS instances
	FROM cte c
	         LEFT JOIN
	     (SELECT cte.id,
	             ins.tenant_id,
	             ins.shared,
	             array_agg(DISTINCT ins.type_id) FILTER (WHERE ins.type_id <> '') AS type_ids,
	             count(DISTINCT ins.instance_id)                                  AS instance_count
	      FROM fs09000000_mod_search.instance_contributor ins
	               INNER JOIN cte
	                          ON ins.contributor_id = cte.id
	      GROUP BY cte.id,
	               ins.tenant_id,
	               ins.shared) sub ON c.id = sub.id
	GROUP BY c.id,
	         c.name,
	         c.name_type_id,
	         c.authority_id,
	         c.last_updated_date
	      ORDER BY last_updated_date ASC

MSK CPU usage

During all tests CPU usage haven’t exceed 60% on all brokers.

Appendix

Infrastructure

PTF -environment rcp1

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r6.xlarge database instances, writer
MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x (KRaft mode)
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch 2.13 ptf-test cluster
- r6g.2xlarge.search 4 data nodes
- r6g.large.search 3 dedicated master nodes

Cluster Resources - rcp1-pvt

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	1	mod-remote-storage:3.4.1	2	4920	4472	128	3960	512	512
mod-remote-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-finance-storage	1	mod-finance-storage:8.8.2	2	1024	896	128	700	88	128
mod-finance-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-ebsconet	1	mod-ebsconet:2.4.0	2	1248	1024	128	700	128	256
mod-ebsconet - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-sip2	1	edge-sip2:3.4.0	2	1024	896	128	768	88	128
mod-consortia-keycloak	1	mod-consortia-keycloak:1.7.1	2	5136	4776	512	4416	384	512
mod-consortia-keycloak - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-tags	1	mod-tags:2.4.0	2	1024	896	128	768	88	128
mod-tags - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-courses	1	edge-courses:1.6.0	2	1024	896	128	768	88	128
mod-inventory-update	1	mod-inventory-update:4.1.0	2	1024	896	128	768	88	128
mod-inventory-update - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-notify	1	mod-notify:3.4.0	2	1024	896	128	768	88	128
mod-notify - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-configuration	1	mod-configuration:5.12.0	2	1024	896	128	768	88	128
mod-configuration - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
folio-keycloak	6	folio-keycloak:26.1.2	1	5000	3000	2048	2048	0	0
edge-caiasoft	1	edge-caiasoft:2.4.0	2	1024	896	128	768	88	128
mod-licenses	1	mod-licenses:6.2.1	2	2480	2312	512	1792	384	512
mod-licenses - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-gobi	1	mod-gobi:3.0.0	2	1024	896	128	768	88	128
mod-gobi - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-graphql	1	mod-graphql:1.13.1	2	1024	896	128	768	88	128
mod-graphql - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-roles-keycloak	1	mod-roles-keycloak:3.0.0	2	1024	896	128	512	88	256
mod-roles-keycloak - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-erm-usage	1	mod-erm-usage:5.1.0	2	2800	2550	128	1800	384	512
mod-erm-usage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-batch-print	1	mod-batch-print:1.3.0	2	1024	896	128	768	88	128
mod-batch-print - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-copycat	1	mod-copycat:1.8.0	2	1024	896	128	768	88	128
mod-copycat - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-entities-links	1	mod-entities-links:4.0.1	2	2592	2483	512	1440	0	1024
mod-entities-links - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
pub-edge	4	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	1	mod-orders:13.0.2	2	2048	1740	1024	1024	384	512
mod-orders - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
kong	4	kong:3.7.4	1	6144	6144	2048	768	88	128
edge-patron	1	edge-patron:5.3.0	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.10.2	2	1024	896	128	768	88	128
edge-inn-reach	1	edge-inn-reach:3.4.0	2	1024	896	128	768	88	128
mod-users-bl	1	mod-users-bl:7.10.1	2	1440	1152	128	922	88	128
mod-users-bl - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-invoice	1	mod-invoice:6.0.0	2	1440	1152	512	922	88	128
mod-invoice - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-inventory-storage	1	mod-inventory-storage:29.0.3	2	4096	3690	1024	3067	384	512
mod-inventory-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	768	512	256	0	128
mod-user-import	1	mod-user-import:3.10.0	2	1024	896	128	768	88	128
mod-user-import - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-sender	1	mod-sender:1.14.1	2	1024	896	128	768	88	128
mod-sender - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mgr-tenant-entitlements	3	mgr-tenant-entitlements:3.0.0	1	1024	896	1024	512	88	256
mod-login-keycloak	1	mod-login-keycloak:3.0.0	2	1024	896	128	512	88	256
mod-login-keycloak - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-data-export-worker	2	mod-data-export-worker:3.5.0-SNAPSHOT.271	2	3072	2800	1024	2048	384	512
mod-data-export-worker - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	768	512	256	0	128
mod-circulation-storage	1	mod-circulation-storage:17.4.0	2	2880	2592	1024	1814	384	512
mod-circulation-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-calendar	1	mod-calendar:3.3.0	2	2048	1536	128	768	88	128
mod-calendar - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-event-config	1	mod-event-config:2.9.0	2	1024	896	128	768	88	128
mod-event-config - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-courses	1	mod-courses:1.4.11	2	1024	896	128	768	88	128
mod-courses - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-circulation-item	1	mod-circulation-item:1.2.0	2	1024	896	128	0	0	0
mod-circulation-item - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-email	1	mod-email:1.19.0	2	2800	2550	128	1800	384	512
mod-email - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-pubsub	1	mod-pubsub:2.16.0	2	1536	1440	1024	922	384	512
mod-pubsub - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-di-converter-storage	1	mod-di-converter-storage:2.4.0	2	1024	896	128	768	88	128
mod-di-converter-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-circulation	1	mod-circulation:24.4.1	2	2880	2592	1024	1814	384	512
mod-circulation - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
edge-orders	1	edge-orders:3.2.0	2	1024	896	128	768	88	128
edge-rtac	1	edge-rtac:2.9.1	2	1024	896	128	768	88	128
mod-users	1	mod-users:19.5.0	2	1024	896	256	768	88	128
mod-users - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-template-engine	1	mod-template-engine:1.22.0	2	1024	896	128	768	88	128
mod-template-engine - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-audit	1	mod-audit:2.11.0	2	1024	896	128	768	88	128
mod-audit - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-source-record-manager	1	mod-source-record-manager:3.10.1	2	4096	3688	1024	3076	384	512
mod-source-record-manager - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-quick-marc	1	mod-quick-marc:7.0.0	1	2288	2176	128	1664	384	512
mod-quick-marc - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-feesfines	1	mod-feesfines:19.3.1	2	1024	896	128	768	88	128
mod-feesfines - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-dcb	3	mod-dcb:1.3.0	2	2048	1792	128	1536	176	256
mod-dcb - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-service-interaction	1	mod-service-interaction:4.2.1	2	2048	1844	256	1290	384	512
mod-service-interaction - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-patron	1	mod-patron:6.3.0	2	1024	896	128	768	88	128
mod-patron - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-connexion	1	edge-connexion:1.4.0	2	1024	896	128	768	88	128
mod-data-export-spring	1	mod-data-export-spring:3.5.0	1	2048	1844	256	1536	384	512
mod-data-export-spring - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-organizations-storage	1	mod-organizations-storage:4.9.0	2	1024	896	128	700	88	128
mod-organizations-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-erm	1	edge-erm:1.4.0	2	1024	896	128	768	88	128
mod-ncip	1	mod-ncip:1.15.7	2	1024	896	128	768	88	128
mod-ncip - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-agreements	1	mod-agreements:7.2.1	2	4096	4096	1024	4096	0	0
mod-agreements - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-organizations	1	mod-organizations:2.1.0	2	1024	896	128	620	88	128
mod-organizations - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-settings	1	mod-settings:1.2.0	2	1024	896	200	768	88	128
mod-settings - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-serials-management	1	mod-serials-management:2.0.1	2	2480	2312	256	1792	384	512
mod-serials-management - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-search	1	mod-search:5.0.1	2	2592	2480	1024	1440	512	1024
mod-search - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-dematic	1	edge-dematic:2.4.1	1	1024	896	128	768	88	128
mod-data-import	1	mod-data-import:3.3.1	1	2048	1844	256	1292	384	512
mod-data-import - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-record-specifications	1	mod-record-specifications:2.0.0	2	1024	896	400	1440	256	512
mod-record-specifications - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-inn-reach	1	mod-inn-reach:3.4.0	2	3600	3240	128	2880	512	1024
mod-inn-reach - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-orders-storage	1	mod-orders-storage:13.9.1	2	1024	896	512	700	88	128
mod-orders-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-erm-usage-harvester	1	mod-erm-usage-harvester:5.1.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-password-validator	1	mod-password-validator:3.4.0	2	1440	1298	128	768	384	512
mod-password-validator - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-bulk-operations	1	mod-bulk-operations:2.2.1	2	3072	2600	1024	2048	384	512
mod-bulk-operations - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1500	768	512	256	0	128
mod-fqm-manager	1	mod-fqm-manager:4.0.2	2	3000	2600	128	2048	88	512
mod-fqm-manager - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
edge-dcb	1	edge-dcb:1.3.0	2	1024	896	128	768	88	128
mod-finance	1	mod-finance:5.1.0	2	1024	896	256	768	88	128
mod-finance - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-lists	1	mod-lists:3.1.1	2	6000	2600	128	2048	88	512
mod-lists - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	768	715	128	256	0	96
mod-permissions	1	mod-permissions:6.7.0	2	1684	1544	512	1024	384	512
mod-permissions - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-users-keycloak	1	mod-users-keycloak:3.0.1	2	1024	896	128	512	88	256
mod-users-keycloak - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-marc-migrations	1	mod-marc-migrations:2.0.0	2	2880	2592	1024	1814	384	512
mod-marc-migrations - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mgr-applications	3	mgr-applications:3.0.0	1	1024	896	1024	512	88	256
mgr-tenants	3	mgr-tenants:3.0.0	1	1024	896	1024	512	88	256
edge-oai-pmh	1	edge-oai-pmh:2.11.0	2	1512	1360	1024	1440	384	512
mod-okapi-facade	1	mod-okapi-facade:3.0.0	2	1024	896	128	768	88	128
mod-okapi-facade - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-rtac	1	mod-rtac:3.8.0	2	1024	896	128	768	88	128
mod-rtac - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-source-record-storage	1	mod-source-record-storage:5.10.1	2	4096	3688	1024	3076	384	512
mod-source-record-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-scheduler	1	mod-scheduler:3.0.0	2	1024	896	128	512	88	256
mod-scheduler - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-inventory	1	mod-inventory:21.1.1	2	4096	3688	1024	3076	384	512
mod-inventory - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	768	512	256	0	128
mod-linked-data	1	mod-linked-data:1.0.1	2	2048	1792	256	768	88	128
mod-linked-data - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-patron-blocks	1	mod-patron-blocks:1.12.1	2	1024	896	1024	620	88	128
mod-patron-blocks - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
edge-fqm	1	edge-fqm:4.0.0	2	1024	896	128	768	88	128
nginx-edge	4	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-invoice-storage	1	mod-invoice-storage:6.0.0	2	1872	1536	0	1024	384	512
mod-invoice-storage - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-reading-room	1	mod-reading-room:1.2.0	2	1024	896	128	768	88	128
mod-reading-room - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-data-export	1	mod-data-export:5.2.1	1	2592	2480	2048	1440	88	1024
mod-data-export - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-oai-pmh	1	mod-oai-pmh:3.15.0	2	4096	3690	2048	3076	384	512
mod-oai-pmh - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	256	256	0	96
mod-notes	1	mod-notes:7.0.0	2	1024	896	128	952	384	512
mod-notes - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-kb-ebsco-java	1	mod-kb-ebsco-java:6.0.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96
mod-eusage-reports	1	mod-eusage-reports:3.0.0	2	1024	896	128	768	88	128
mod-eusage-reports - Sidecar 1	N/A	folio-module-sidecar:3.0.1.410	N/A	1024	512	128	256	0	96

Dataset Size:

Record type	Number of records

Record type	Number of records
Instances	30 243 010
Holdings	30 301 825
Items	31 390 568
marc-records	49 547 275

Methodology/Approach

Pre-generated files were used for DI Create job profile
- 5K, 10K, 25K, 50K, 100K files.
Run DI Create on a single tenant one by one with the delay with files using PTF - Create 2 profile.
Prepare files for DI Update with the Data export app, using previously imported instances
Run DI Update on a single tenant one by one with the delay with prepared files using PTF - Update Success 6 profile

5K, 10K, 25K, 50K, 100K files.

Data-import durations were obtained from DB using SQL query

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from [tenant]_mod_source_record_manager.job_execution order by started_date desc limit 1000;

Folio Development Teams

Data Import MARC BIB Sunflower [non-ECS]