Table of Contents

outline	true

...

The previous results report:
- PTF - Data Import MARC BIB + CI/CO (Quesnelia) [ECS]

The mod-pubsub module memory leak investigation ticket:
Jira Legacy
server System Jira
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODPUBSUB-311
New behaviour from the mod_orders_storage.po_line table detected during data import create jobs. Consider also [tenant]_mod_orders_storage.internal_lock query. It should be investigated.

...

Profile	MARC File	DI Duration				DI Delta Quesnelia/Ramsons without CI/CO, %	DI Delta Quesnelia/Ramsons with CI/CO, %	DI Delta Ramsons/Ramsons without/with CI/CO, %	Check In, Check Out Response time (8 users)		Check In, Check Out Response time (8 users)		Delta, %
		without CI/CO		with CI/CO					Quesnelia		Ramsons		Quesnelia/Ramsons	Quesnelia/Ramsons
		Quesnelia	Ramsons	Quesnelia	Ramsons				CI Average sec	CO Average sec	CI Average sec	CO Average sec	CI	CO
DI MARC Bib Create (PTF - Create 2)	5K.mrc		00:04:11	00:03:21	00:04:11		24.88%	0.00%	0.831	1.357	1.01	2	21.54%	47.38%
	10K.mrc	00:04:14	00:05:10	00:06:51	00:06:39	22.05%	-2.92%	28.71%	0.845	1.41	0.95	1.88	12.43%	33.33%
	25K.mrc	00:09:41	00:10:30	00:12:41	00:16:13	8.43%	27.86%	54.44%	0.719	1.333	1.18	2.25	64.12%	68.79%
	50K.mrc	00:18:18	00:15:43	00:23:19	00:29:47	-14.12%	27.73%	89.50%	0.691	1.327	1.11	2.29	60.64%	72.57%
	100K.mrc	00:38:36	00:31:51	00:51:24	00:56:00	-17.49%	8.95%	75.82%	0.664	1.335	1.6	2.4	140.96%	79.78%
DI MARC Bib Update (PTF - Updates Success - 6)	5K.mrc			00:04:12	00:06:19		50.40%		0.764	1.458	0.99	2.2	29.58%	50.89%
	10K.mrc	00:05:59	00:07:10	00:08:15	00:12:10	19.78%	47.47%	69.77%	0.779	1.377	1.1	2.5	41.21%	81.55%
	25K.mrc	00:19:52	00:19:03	00:20:38	00:24:31	-4.11%	18.82%	28.70%	0.755	1.401	1.04	2.1	37.75%	49.89%
	50K.mrc	00:37:53	00:38:53	00:43:06	00:49:53	2.64%	15.74%	28.29%	0.75	1.444	1.03	2.2	37.33%	52.35%
	100K.mrc	01:14:00	01:23:00	01:29:09	01:48:00	12.16%	21.14%	30.12%	0.73	1.458	1	2.1	36.99%	44.03%

Detailed CICO response time comparison without DI

Scenario	Load level	Request	Response time, sec Quesnelia		Response time, sec Ramsons
Scenario	Load level	Request	95 perc	average	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.64	0.49	1	0.65
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1.24	1.08	1.9	1.2

Test №1

Response times

Expand

title	Response times

...

Top SQL-queries

MSK resources utilization

CPU (User) usage by broker

Maximum utilization was 60% with 50k file

...

Infrastructure

PTF -environment rcon

rcon 12 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.xlarge database instance: Writer instance
MSK fse-test
- - 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
  - Apache Kafka version 3.7.x, metadata mode - KRaft
  - EBS storage volume per broker 300 GiB
  - auto.create.topics.enable=true
  - log.retention.minutes=480
  - default.replication.factor=2
  - revision - 26
OpenSearch 2.13 ptf-test cluster
- r6g.2xlarge.search 4 data nodes
- r6g.large.search 3 dedicated master nodes

DB table records size:

[tenant]
- instances - 1163924
- items - 2091901
- holding records - 1348036

Modules:

Expand

title	All modules

Cluster Resources - rcon-pvt (Fri Jan 17 14:41:06 UTC 2025)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	8	mod-remote-storage:3.3.3	2	4920	4472	0	3960	512	512
mod-finance-storage	7	mod-finance-storage:8.7.3	2	1024	896	0	700	88	128
mod-ncip	7	mod-ncip:1.15.6	2	1024	896	0	768	88	128
mod-agreements	8	mod-agreements:7.1.4	2	1592	1488	0	0	0	0
mod-ebsconet	8	mod-ebsconet:2.3.1	2	1248	1024	0	700	128	256
mod-organizations	7	mod-organizations:2.0.0	2	1024	896	0	700	88	128
mod-consortia	10	mod-consortia:1.2.2	2	5136	4776	0	2048	512	1024
edge-sip2	7	edge-sip2:3.3.1	2	1024	896	0	768	88	128
mod-serials-management	8	mod-serials-management:1.1.2	2	2480	2312	0	1792	384	512
mod-settings	7	mod-settings:1.1.0	2	1024	896	0	768	88	128
mod-data-import	10	mod-data-import:3.2.4	1	2048	1844	0	1292	384	512
mod-search	18	mod-search:4.0.7	2	2592	2480	0	1440	512	1024
edge-dematic	7	edge-dematic:2.3.1	1	1024	896	0	768	88	128
mod-inn-reach	4	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024
mod-record-specifications	7	mod-record-specifications:1.0.2	2	1024	896	0	768	88	128
mod-tags	7	mod-tags:2.3.0	2	1024	896	0	768	88	128
mod-authtoken	9	mod-authtoken:2.16.1	2	1440	1152	0	922	88	128
edge-courses	8	edge-courses:1.5.1	2	1024	896	0	768	88	128
mod-notify	7	mod-notify:3.3.0	2	1024	896	0	768	88	128
mod-inventory-update	7	mod-inventory-update:4.0.0	2	1024	896	0	768	88	128
mod-configuration	7	mod-configuration:5.11.0	2	1024	896	0	768	88	128
mod-orders-storage	7	mod-orders-storage:13.8.3	2	1024	896	0	700	88	128
edge-caiasoft	7	edge-caiasoft:2.3.2	2	1024	896	0	768	88	128
mod-login-saml	7	mod-login-saml:2.9.3	2	1024	896	0	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:5.0.1	2	1024	896	0	768	88	128
mod-gobi	7	mod-gobi:2.9.0	2	1024	896	0	700	88	128
mod-licenses	7	mod-licenses:6.1.2	2	2480	2312	0	1792	384	512
mod-password-validator	7	mod-password-validator:3.3.0	2	1440	1298	0	768	384	512
edge-dcb	8	edge-dcb:1.2.1	2	1024	896	0	768	88	128
mod-bulk-operations	8	mod-bulk-operations:2.1.8	2	3072	2600	0	1536	384	512
mod-fqm-manager	10	mod-fqm-manager:3.0.7	2	3000	2600	0	768	88	128
mod-graphql	9	mod-graphql:1.13.1	2	1024	896	0	768	88	128
mod-finance	8	mod-finance:5.0.1	2	1024	896	0	700	88	128
mod-erm-usage	7	mod-erm-usage:5.0.0	2	2800	2550	0	1800	384	512
mod-batch-print	7	mod-batch-print:1.2.0	2	1024	896	0	768	88	128
mod-tlr	4	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	12	mod-lists:3.0.5	2	6000	2600	0	768	88	128
mod-copycat	7	mod-copycat:1.7.0	2	1024	512	0	768	88	128
mod-entities-links	11	mod-entities-links:3.1.3	2	2592	2480	0	1440	0	1024
mod-permissions	14	mod-permissions:6.6.1	2	1684	1544	0	1024	384	512
pub-edge	7	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	9	mod-orders:12.9.9	2	2048	1740	0	1024	384	512
edge-patron	8	edge-patron:5.2.1	2	1024	896	0	768	88	128
mod-marc-migrations	26	mod-marc-migrations:1.0.0	2	1024	896	0	768	88	128
edge-ncip	8	edge-ncip:1.10.1	2	1024	896	0	768	88	128
edge-inn-reach	5	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	7	mod-users-bl:7.9.3	2	1440	1152	0	922	88	128
mod-oa	4	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-inventory-storage	12	mod-inventory-storage:28.0.4	2	4096	3690	0	3076	512	1024
mod-invoice	8	mod-invoice:5.9.2	2	1440	1152	0	922	88	128
mod-user-import	7	mod-user-import:3.9.0	2	1024	896	0	768	88	128
mod-sender	7	mod-sender:1.13.0	2	1024	896	0	768	88	128
edge-oai-pmh	7	edge-oai-pmh:2.10.0	2	1512	1360	0	1440	384	512
mod-data-export-worker	10	mod-data-export-worker:3.3.6	2	3072	2048	0	2048	384	512
mod-rtac	7	mod-rtac:3.7.0	2	1024	896	0	768	88	128
mod-circulation-storage	8	mod-circulation-storage:17.3.3	2	2880	2592	0	1814	384	512
mod-source-record-storage	13	mod-source-record-storage:5.9.5	2	5600	5000	0	3500	384	512
mod-calendar	7	mod-calendar:3.2.0	2	1024	896	0	768	88	128
mod-event-config	7	mod-event-config:2.8.0	2	1024	896	0	768	88	128
mod-courses	8	mod-courses:1.4.11	2	1024	896	0	768	88	128
mod-circulation-item	7	mod-circulation-item:1.1.0	2	1024	896	0	0	0	0
mod-inventory	9	mod-inventory:21.0.5	2	2880	2592	0	1814	384	512
mod-email	8	mod-email:1.18.1	2	2800	2550	0	1800	384	512
mod-requests-mediated	4	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-circulation	8	mod-circulation:24.3.8	2	2880	2592	0	1814	384	512
mod-pubsub	8	mod-pubsub:2.15.3	2	1536	1440	0	922	384	512
mod-di-converter-storage	9	mod-di-converter-storage:2.3.1	2	1024	896	0	768	88	128
edge-rtac	7	edge-rtac:2.8.0	2	1024	896	0	768	88	128
edge-orders	7	edge-orders:3.1.0	2	1024	896	0	768	88	128
mod-users	8	mod-users:19.4.5	2	1024	896	0	768	88	128
mod-template-engine	7	mod-template-engine:1.21.0	2	1024	896	0	768	88	128
mod-patron-blocks	7	mod-patron-blocks:1.11.1	2	1024	896	0	768	88	128
mod-audit	8	mod-audit:2.10.2	2	1024	896	0	768	88	128
edge-fqm	9	edge-fqm:3.0.2	2	1024	896	0	768	88	128
mod-source-record-manager	8	mod-source-record-manager:3.9.5	2	5600	5000	0	3500	384	512
nginx-edge	7	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	7	mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512
nginx-okapi	7	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	9	okapi:6.1.1	3	1684	1440	0	922	384	512
mod-feesfines	7	mod-feesfines:19.2.1	2	1024	896	0	768	88	128
mod-invoice-storage	7	mod-invoice-storage:5.9.1	2	1872	1536	0	1024	384	512
mod-reading-room	7	mod-reading-room:1.0.0	2	1024	896	0	768	88	128
mod-dcb	8	mod-dcb:1.2.4	2	1024	896	0	768	88	128
mod-service-interaction	7	mod-service-interaction:4.1.1	2	2048	1844	0	1290	384	512
mod-patron	8	mod-patron:6.2.5	2	1024	896	0	768	88	128
mod-data-export	13	mod-data-export:5.1.5	1	2048	1844	0	0	0	0
mod-oai-pmh	7	mod-oai-pmh:3.14.3	2	4096	3690	0	3076	384	512
edge-connexion	7	edge-connexion:1.3.1	2	1024	896	0	768	88	128
mod-notes	7	mod-notes:6.0.0	2	1024	896	0	952	384	512
mod-kb-ebsco-java	8	mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128
mod-organizations-storage	7	mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128
mod-data-export-spring	8	mod-data-export-spring:3.4.3	1	2048	1844	0	1536	384	512
mod-login	7	mod-login:7.12.1	2	1440	1298	0	768	384	512
pub-okapi	7	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
edge-erm	5	edge-erm:1.3.0	2	1024	896	0	768	88	128
mod-eusage-reports	7	mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128

...

Update revision in source-record-storage module to exclude every 30 minutes SQL statements - delete rows in marc_indexers (mi) WITH deleted_rows

{
"name": "srs.marcIndexers.delete.interval.seconds",
"value": "86400"
},

Update mod-serials module. Set number of task with 0 to exclude significant database connection growth.

DB trigger setup in Ramsons

...

Version	Old Version 14	New Version Current
Changes made by	Leonid Kolesnykov	Leonid Kolesnykov
Saved on	Feb 03, 2025	Feb 03, 2025

Versions Compared

Key

Test №1

Response times

Top SQL-queries

MSK resources utilization

CPU (User) usage by broker

Infrastructure

Content Comparison

Versions Compared

Key

Test №1

Response times

Top SQL-queries

MSK resources utilization

CPU (User) usage by broker

Infrastructure