Overview

The purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in Poppy in scope of ticket PERF-700 - Getting issue details... STATUS

Compared with results in previous test report: Data Import Create MARC holdings records [Orchid]

Summary

Data import jobs duration decreased significantly compared to the Orchid release results for all tests
Top CPU utilization: nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%, mod-source-record-storage-b - 25%
Top Memory consumption: mod-users-b - 60%, mod-data-import-b - 60%, okapi-b - 48%. There're no issues with memory.
DI jobs duration is directly proportional to the size of the files being processed.

Recommendations & Jiras

Autovacuum as background process increase DI job duration for 10%-25% so it's necessary to check DB activity before DI jobs start

SQL queries are the possible place where performance may be improved:

SQL DB Load with WAITS - Lock:tuple, Lock:transactionid take a major time in queries:

INSERT INTO [tenant]_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

UPDATE [tenant]_mod_inventory_storage.holdings_record SET jsonb = $1::jsonb WHERE id = 'UUID'

Test Results

Profile used for testing - Default - Create Holdings and SRS MARC Holdings

Test	File	Duration: Morning_Glory (previous results)	Duration: Nolana (previous results)	Duration: Orchid (previous results)	Duration: Poppy
1	1k	28s	32s	45s	32s
2	5k	1m 48s	4m 20s	7m 47s	2m 14s
3	10k	4m 4s	3m 24s	19m 46s	4m 35s
4	80k	29m 6s	21m 22s	20m (error*)	36m 25s

Comparison

Table contains comparison between Poppy and Orchid

Test	File	Duration: Orchid	Duration: Poppy	Diff_absolute	Diff_percentage
1	1k	45s	32s	13s	- 40%
2	5k	7m 47s	2m 14s	5m 33s	- 248%
3	10k	19m 46s	4m 35s	15m 11s	- 331%
4	80k	20m (error*)	36m 25s	No data to compare in Orchid

Service CPU Utilization

CPU utilization, 1k, 5k, 10k, 80k

	1k		5k		10k		80k
mod-inventory-b	26.42	nginx-okapi	51.81	nginx-okapi	51.66	nginx-okapi	51.8
mod-quick-marc-b	23.78	mod-quick-marc-b	44.77	mod-quick-marc-b	44.05	mod-quick-marc-b	44.97
nginx-okapi	21.44	mod-inventory-b	44.21	mod-inventory-b	43.83	mod-inventory-b	41.39
mod-di-converter-storage-b	11.78	okapi-b	25.48	okapi-b	25.16	okapi-b	25.29
okapi-b	11.54	mod-source-record-storage-b	24.27	mod-source-record-storage-b	24.19	mod-source-record-storage-b	25.15
mod-source-record-storage-b	10.67	mod-di-converter-storage-b	20.29	mod-di-converter-storage-b	20.36	mod-di-converter-storage-b	19.79
mod-source-record-manager-b	8.93	mod-inventory-storage-b	17.44	mod-inventory-storage-b	17.11	mod-inventory-storage-b	17.79
mod-inventory-storage-b	7.53	mod-source-record-manager-b	16.93	mod-source-record-manager-b	16.8	mod-source-record-manager-b	16.92
mod-pubsub-b	6.73	mod-pubsub-b	6.67	mod-pubsub-b	6.88	mod-pubsub-b	6.79
mod-users-b	6.32	mod-users-b	6.5	mod-users-b	6.39	mod-users-b	6.37
mod-data-import-b	4.04	mod-permissions-b	2.85	mod-organizations-storage-b	2.61	mod-permissions-b	2.87
mod-organizations-storage-b	2.29	mod-organizations-storage-b	2.42	mod-notes-b	2.27	mod-organizations-storage-b	2.43
mod-notes-b	2.05	mod-notes-b	2.1	mod-gobi-b	2.01	mod-notes-b	2.14
mod-gobi-b	2.01	mod-gobi-b	2.01	mod-permissions-b	1.43	mod-gobi-b	2
mod-permissions-b	1.46	mod-data-import-b	1.41	mod-search-b	1.31	mod-search-b	1.48
mod-search-b	1.25	mod-search-b	1.26	mod-data-import-b	1.29	mod-data-import-b	1.32
mod-circulation-storage-b	0.57	mod-circulation-storage-b	0.59	mod-circulation-storage-b	0.6	mod-circulation-storage-b	0.58
mod-circulation-b	0.32	mod-circulation-b	0.47	mod-circulation-b	0.32	mod-circulation-b	0.32
pub-okapi	0.27	pub-okapi	0.19	pub-okapi	0.18	pub-okapi	0.18

nginx-okapi - 52%, mod-quick-marc-b - 45%, mod-inventory-b - 42%, mod-source-record-storage-b - 25%

Memory Utilization

Memory consumption

mod-inventory-b	61.01
mod-users-b	59.97
mod-data-import-b	59.38
okapi-b	48.24
mod-source-record-manager-b	37.31
mod-search-b	31.38
mod-di-converter-storage-b	31.19
mod-quick-marc-b	27.52
mod-pubsub-b	24.53
mod-remote-storage-b	21.59
mod-inventory-storage-b	16.71
nginx-okapi	4.68
pub-okapi	4.58

DB CPU Utilization

DB CPU was about 80% for all tests except of 1k - 23%.

DB Connections

DB connections number- 505

DB Load

SQL queries

Infrastructure

PTF - environment pcp1

10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instances, writer

Name	Memory GIB	vCPUs
db.r6g.4xlarge	128 GiB	16 vCPUs

MSK ptf-mobius-testing2
- 2 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2

Modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcp1-pvt
Thu Mar 14 14:24:59 UTC 2024
mod-remote-storage	11	mod-remote-storage:3.0.0	2	4920	4472	1024	3960	512	512	FALSE
mod-ncip	10	mod-ncip:1.14.4	2	1024	896	128	768	88	128	FALSE
mod-finance-storage	10	mod-finance-storage:8.5.0	2	1024	896	1024	700	88	128	FALSE
mod-agreements	10	mod-agreements:6.0.2	2	1592	1488	128	0	0	0	FALSE
mod-ebsconet	10	mod-ebsconet:2.1.1	2	1248	1024	128	700	128	256	FALSE
edge-sip2	8	edge-sip2:3.1.1	2	1024	896	128	768	88	128	FALSE
mod-organizations	10	mod-organizations:1.8.0	2	1024	896	128	700	88	128	FALSE
mod-settings	11	mod-settings:1.0.2	2	1024	896	200	768	88	128	FALSE
edge-dematic	10	edge-dematic:2.1.0	1	1024	896	128	768	88	128	FALSE
mod-data-import	18	mod-data-import:3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-search	20	mod-search:3.0.5	2	2592	2480	2048	1440	512	1024	FALSE
mod-tags	10	mod-tags:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-authtoken	16	mod-authtoken:2.14.1	2	1440	1152	512	922	88	128	FALSE
edge-courses	2	edge-courses:1.3.0	2	1024	896	128	768	88	128	FALSE
mod-notify	10	mod-notify:3.1.0	2	1024	896	128	768	88	128	FALSE
mod-inventory-update	10	mod-inventory-update:3.2.1	2	1024	896	128	768	88	128	FALSE
mod-configuration	10	mod-configuration:5.9.2	2	1024	896	128	768	88	128	FALSE
mod-orders-storage	10	mod-orders-storage:13.6.0	2	1024	896	512	700	88	128	FALSE
edge-caiasoft	10	edge-caiasoft:2.1.0	2	1024	896	128	768	88	128	FALSE
mod-login-saml	18	mod-login-saml:2.7.1	2	1024	896	128	768	88	128	FALSE
mod-erm-usage-harvester	11	mod-erm-usage-harvester:4.4.1	2	1024	896	128	768	88	128	FALSE
mod-password-validator	10	mod-password-validator:3.1.0	2	1440	1298	128	768	384	512	FALSE
mod-licenses	10	mod-licenses:5.0.2	2	2480	2312	128	1792	384	512	FALSE
mod-gobi	10	mod-gobi:2.7.1	2	1024	896	128	700	88	128	FALSE
mod-fqm-manager	9	mod-fqm-manager:1.1.0-SNAPSHOT.1078	2	1024	896	128	768	88	128	FALSE
mod-bulk-operations	9	mod-bulk-operations:1.1.7	2	3072	2600	1024	1536	384	512	FALSE
mod-graphql	16	mod-graphql:1.12.0	2	1024	896	128	768	88	128	FALSE
mod-finance	10	mod-finance:4.8.0	2	1024	896	128	700	88	128	FALSE
mod-erm-usage	13	mod-erm-usage:4.6.0	2	1024	896	128	768	88	128	FALSE
mod-copycat	10	mod-copycat:1.5.0	2	1024	896	128	768	88	128	FALSE
mod-lists	8	mod-lists:1.1.0-SNAPSHOT.1261	2	1024	896	128	768	88	128	FALSE
mod-entities-links	15	mod-entities-links:2.0.4	2	2592	2480	400	1440	0	1024	FALSE
mod-permissions	47	mod-permissions:6.5.0-SNAPSHOT.369	2	1684	1544	512	1024	384	512	FALSE
pub-edge	9	pub-edge:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-orders	10	mod-orders:12.7.1	2	2048	1440	1024	1024	384	512	FALSE
edge-patron	10	edge-patron:5.0.0	2	1024	896	256	768	88	128	FALSE
edge-ncip	11	edge-ncip:1.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	10	mod-users-bl:7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	15	mod-inventory-storage:27.0.4	2	4096	3690	2048	3076	384	512	FALSE
mod-invoice	10	mod-invoice:5.7.2	2	1440	1152	512	922	88	128	FALSE
mod-user-import	10	mod-user-import:3.8.0	2	1024	896	128	768	88	128	FALSE
mod-sender	10	mod-sender:1.11.0	2	1024	896	128	768	88	128	FALSE
edge-oai-pmh	8	edge-oai-pmh:2.7.1	2	1512	1360	1024	1440	384	512	FALSE
mod-data-export-worker	10	mod-data-export-worker:3.1.2	2	3072	2800	1024	2048	384	512	FALSE
mod-rtac	10	mod-rtac:3.5.0	2	1024	896	128	768	88	128	FALSE
mod-circulation-storage	14	mod-circulation-storage:17.1.7	2	2880	2592	1536	1814	384	512	FALSE
mod-calendar	10	mod-calendar:2.5.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-storage	18	mod-source-record-storage:5.7.5	2	5600	5000	2048	3500	384	512	FALSE
mod-event-config	10	mod-event-config:2.6.0	2	1024	896	128	768	88	128	FALSE
mod-courses	10	mod-courses:1.4.8	2	1024	896	128	768	88	128	FALSE
mod-inventory	15	mod-inventory:20.1.8	2	2880	2592	1024	1814	384	512	FALSE
mod-email	10	mod-email:1.16.0	2	1024	896	128	768	88	128	FALSE
mod-di-converter-storage	18	mod-di-converter-storage:2.1.5	2	1024	896	128	768	88	128	FALSE
mod-circulation	14	mod-circulation:24.0.11	2	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	13	mod-pubsub:2.11.3	2	1536	1440	1024	922	384	512	FALSE
edge-orders	10	edge-orders:2.9.1	2	1024	896	128	768	88	128	FALSE
edge-rtac	7	edge-rtac:2.6.2	2	1024	896	128	768	88	128	FALSE
mod-template-engine	10	mod-template-engine:1.19.1	2	1024	896	128	768	88	128	FALSE
mod-users	34	mod-users:19.3.0-SNAPSHOT.677	2	1024	896	128	768	88	128	FALSE
mod-patron-blocks	10	mod-patron-blocks:1.9.0	2	1024	896	1024	768	88	128	FALSE
edge-fqm	21	edge-fqm:1.0.1	2	1024	896	128	768	88	128	FALSE
mod-audit	10	mod-audit:2.8.0	2	1024	896	128	768	88	128	FALSE
mod-source-record-manager	17	mod-source-record-manager:3.7.8	2	5600	5000	2048	3500	384	512	FALSE
nginx-edge	9	nginx-edge:2023.06.14	2	1024	896	128	0	0	0	FALSE
mod-quick-marc	11	mod-quick-marc:5.0.1	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	9	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	11	okapi:5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	11	mod-feesfines:19.0.0	2	1024	896	128	768	88	128	FALSE
mod-invoice-storage	10	mod-invoice-storage:5.7.0	2	1872	1536	1024	1024	384	512	FALSE
mod-service-interaction	10	mod-service-interaction:3.0.2	2	2048	1844	256	1290	384	512	FALSE
mod-data-export	12	mod-data-export:4.8.7	1	1024	896	1024	768	88	128	FALSE
mod-patron	10	mod-patron:6.0.0	2	1024	896	128	768	88	128	FALSE
mod-oai-pmh	5	mod-oai-pmh:3.12.8	2	4096	3690	2048	3076	384	512	FALSE
edge-connexion	10	edge-connexion:1.1.0	2	1024	896	128	768	88	128	FALSE
mod-kb-ebsco-java	10	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128	FALSE
mod-notes	10	mod-notes:5.1.0	2	1024	896	128	952	384	512	FALSE
mod-organizations-storage	10	mod-organizations-storage:4.6.0	2	1024	896	128	700	88	128	FALSE
mod-data-export-spring	12	mod-data-export-spring:3.0.2	1	2048	1844	256	1536	384	512	FALSE
mod-login	10	mod-login:7.10.1	2	1440	1298	1024	768	384	512	FALSE
pub-okapi	9	pub-okapi:2023.06.14	2	1024	896	128	768	0	0	FALSE
mod-eusage-reports	13	mod-eusage-reports:2.0.0	2	1024	896	128	768	88	128	FALSE

Folio Development Teams

Data Import Create MARC Holdings Records [Poppy]