Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records on the Ramsons[NON-ECS] release environment.

PERF-969 - Getting issue details... STATUS

Summary

Data Import with Check In/Check Out tests finished successfully with PTF - Create 2 and PTF - Updates Success - 2 job profiles with files 5K, 10K, 25K, 50K, 100K records.
Instance type was updated from m6i.2xlarge to r7g.2xlarge
Comparing with previous testing results Quesnelia and Ramsons releases
- Check-in duration degraded by 20% and Check-out duration degraded by 40% on average without Data Import
- Data Import Create jobs duration without CI/CO degraded by 5%-30%, depending on file size
- Data Import Update jobs durations without CI/CO degraded by 44%-138%, depending on file size
- Data Import Create jobs duration with CI/CO degraded by 15%-40%, depending on file size
- Data Import Update jobs durations with CI/CO degraded by two times.
- Check-in duration degraded by 20% and Check-out duration degraded by 40% on average for Data Import Create
- Check-in duration degraded by 40% and Check-out duration degraded by 45% on average for Data Import Update
During the test, a slow SQL query was detected with a latency of 28,32 seconds.

Test Runs

Test №	Scenario	Test Conditions	Results
1	DI MARC Bib Create	5K, 10K, 25K, 50K, 100K consequentially	Completed
1	CICO	8 users	Completed
2	DI MARC Bib Update	5K, 10K, 25K, 50K, 100K consequentially	Completed
2	CICO	8 users	Completed

Test Results

This table contains durations for Data Import with Check-in/Check-out.

Profile	MARC File	DI Duration Ramsons (hh:mm:ss)	Check In, Check Out Response time (8 users) Ramsons
Profile	MARC File	DI Duration Ramsons (hh:mm:ss)	CI Average, ms	CO Average, ms
DI MARC Bib Create (PTF - Create 2)	5K.mrc	0:02:53	837	1.628
	10K.mrc	0:05:35	809	1.460
	25K.mrc	0:15:13	895	1.545
	50K.mrc	0:27:17	901	1.495
	100K.mrc	1:04:31	896	1.459
DI MARC Bib Update (PTF - Updates Success - 2)	5K.mrc	0:07:25	992	1.767
	10K.mrc	0:12:52	1.100	1.903
	25K.mrc	0:37:02	1.225	1.957
	50K.mrc	1:13:16	1.299	1.957
	100K.mrc	2:32:25	1.297	2.027

Check-in/Check-out without Data Import

Scenario	Load level	Request	Response time, ms Quesnelia
Scenario	Load level	Request	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	767	625
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1347	1138

Comparison

This table contains DI durations with CICO comparison between Quesnelia and Ramsons releases.

Profile

MARC File

DI Duration

DI Delta
Quesnelia/Ramsons

without CI/CO
(hh:mm:ss)

DI Delta
Quesnelia/Ramsons

with CI/CO
(hh:mm:ss)

Check In, Check Out
Response time
(8 users)

Delta, %

without CI/CO

with CI/CO

Quesnelia

Ramsons

Quesnelia/Ramsons

Quesnelia

Ramsons

Quesnelia

Ramsons

CI Average sec

CO Average sec

CI Average sec

CO Average sec

DI MARC Bib Create (PTF - Create 2)

5K.mrc

00:03:20

00:02:32

0:02:53

0:00:21
+14%

0.645

0.901

0.837

1.628

+23%

+45%

10K.mrc

00:06:00

00:05:33

00:05:03

0:05:35

00:00:27
+7%

0:00:32
+11%

0.628

0.922

0.809

1.460

+22%

+37%

25K.mrc

00:13:41

0:14:16

00:11:58

0:15:13

00:00:35
+4%

0:03:15
+27%

0.639

0.96

0.895

1.545

+29%

+38%

50K.mrc

00:21:59

0:28:23

00:23:29

0:27:17

00:06:24
+29%

0:03:48
+16%

0.678

1.003

0.901

1.495

+25%

+33%

100K.mrc

00:40:16

0:49:03

00:46:07

1:04:31

00:08:47
+22%

0:18:24
+40%

0.686

0.998

0.896

1.459

+23%

+32%

DI MARC Bib Update (PTF - Updates Success - 2)

5K.mrc

00:07:10

00:03:24

0:07:25

0:04:01
+118%

0.628

0.975

0.992

1.767

+37%

+45%

10K.mrc

00:10:27

0:15:00

00:06:29

0:12:52

00:04:33
+43%

0:04:01
+98%

0.664

1.018

1.100

1.903

+40%

+47%

25K.mrc

00:23:16

0:34:41

00:16:15

0:37:02

00:11:27
+49%

0:20:47
+128%

0.717

1.062

1.225

1.957

+41%

+46%

50K.mrc

00:40:52

1:10:01

00:33:33

1:13:16

00:29:19
+71%

0:39:43
+118%

0.721

1.071

1.299

1.957

+44%

+45%

100K.mrc

01:02:00

2:27:05

01:10:14

2:32:25

01:05:05
+137%

1:22:11
+117%

0.739

1.081

1.297

2.027

+43%

+47%

Detailed CICO response time comparison without DI

Scenario	Load level	Request	Response time, ms Quesnelia		Response time, ms Ramsons		Delta, %
Scenario	Load level	Request	95 perc	average	95 perc	average	Quesnelia/Ramsons average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	609	521	767	625	+19.96%
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1.070	803	1.347	1.138	+41.72%

Test №1-2

Resource utilization

Resource utilization table

CPU		RAM
okapi-b	36%	mod-oa-b	127%
mod-inventory-b	5.74%	mod-dcb-b	64%
mod-source-record-storage-b	4.21%	mod-inventory-b	63%
mod-inventory-storage-b	3.67%	mod-source-record-storage-b	46%
mod-pubsub-b	1.20%	okapi-b	44%
mod-source-record-manager-b	1.20%	mod-quick-marc-b	41%
nginx-okapi	0.85%	mod-pubsub-b	41%
mod-circulation-storage-b	0.82%	mod-data-import-b	41%
mod-users-bl-b	0.59%	mod-users-b	40%
mod-di-converter-storage-b	0.57%	mod-users-bl-b	35%
mod-search-b	0.48%	mod-entities-links-b	35%
mod-quick-marc-b	0.43%	mod-feesfines-b	35%
mod-authtoken-b	0.29%	mod-di-converter-storage-b	33%
mod-circulation-b	0.26%	mod-source-record-manager-b	33%
mod-configuration-b	0.16%	mod-patron-blocks-b	30%
mod-dcb-b	0.14%	mod-inventory-storage-b	30%
mod-feesfines-b	0.11%	mod-circulation-storage-b	30%
mod-entities-links-b	0.11%	mod-configuration-b	29%
mod-patron-blocks-b	0.05%	mod-patron-b	22%
pub-okapi	0.05%	mod-circulation-b	22%
mod-oa-b	0.04%	mod-authtoken-b	21%
mod-data-import-b	0.04%	edge-patron-b	17%
edge-patron-b	0.03%	nginx-okapi	4%
mod-patron-b	0.03%	pub-okapi	4%
pub-edge	0.00%	pub-edge	4%

Response times

Service CPU Utilization

Here we can see that okapi-b used 36% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend

DB CPU Utilization

DB Connections

Kafka metrics

OpenSearch Data Nodes metrics

DB load

Top SQL-queries

Top applications

# TOP SQL statement

#	TOP SQL statement
1	`insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)`
#	TOP SLOW statement
1	WITH cte AS (SELECT id, name, name_type_id, authority_id, last_updated_date FROM fs09000000_mod_search.contributor WHERE last_updated_date > $1 ORDER BY last_updated_date ) SELECT c.id, c.name, c.name_type_id, c.authority_id, c.last_updated_date, json_agg( CASE WHEN sub.instance_count IS NULL THEN NULL ELSE json_build_object( 'count', sub.instance_count, 'typeId', sub.type_ids, 'shared', sub.shared, 'tenantId', sub.tenant_id ) END ) AS instances FROM cte c LEFT JOIN (SELECT cte.id, ins.tenant_id, ins.shared, array_agg(DISTINCT ins.type_id) FILTER (WHERE ins.type_id <> '') AS type_ids, count(DISTINCT ins.instance_id) AS instance_count FROM fs09000000_mod_search.instance_contributor ins INNER JOIN cte ON ins.contributor_id = cte.id GROUP BY cte.id, ins.tenant_id, ins.shared) sub ON c.id = sub.id GROUP BY c.id, c.name, c.name_type_id, c.authority_id, c.last_updated_date ORDER BY last_updated_date ASC

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

# TOP SLOW statement

WITH cte AS (SELECT id,
                    name,
                    name_type_id,
                    authority_id,
                    last_updated_date
             FROM fs09000000_mod_search.contributor
             WHERE last_updated_date > $1
             ORDER BY last_updated_date
             )
SELECT c.id,
       c.name,
       c.name_type_id,
       c.authority_id,
       c.last_updated_date,
       json_agg(
               CASE
                                WHEN sub.instance_count IS NULL THEN NULL
                                ELSE json_build_object(
                       'count', sub.instance_count,
                       'typeId', sub.type_ids,
                       'shared', sub.shared,
                       'tenantId', sub.tenant_id
               )
               END
       ) AS instances
FROM cte c
         LEFT JOIN
     (SELECT cte.id,
             ins.tenant_id,
             ins.shared,
             array_agg(DISTINCT ins.type_id) FILTER (WHERE ins.type_id <> '') AS type_ids,
             count(DISTINCT ins.instance_id)                                  AS instance_count
      FROM fs09000000_mod_search.instance_contributor ins
               INNER JOIN cte
                          ON ins.contributor_id = cte.id
      GROUP BY cte.id,
               ins.tenant_id,
               ins.shared) sub ON c.id = sub.id
GROUP BY c.id,
         c.name,
         c.name_type_id,
         c.authority_id,
         c.last_updated_date
      ORDER BY last_updated_date ASC

Appendix

Infrastructure

PTF -environment rcp1
rcp1 5 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 1 instance of db.r6g.xlarge database instance: Writer instance MSK fse-test 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone) Apache Kafka version 3.7.x, metadata mode - KRaft EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=2 revision - 26 OpenSearch 2.13 ptf-test cluster r6g.2xlarge.search 4 data nodes r6g.large.search 3 dedicated master nodes

PTF -environment rcp1

rcp1 5 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.xlarge database instance: Writer instance
MSK fse-test
- - 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
  - Apache Kafka version 3.7.x, metadata mode - KRaft
  - EBS storage volume per broker 300 GiB
  - auto.create.topics.enable=true
  - log.retention.minutes=480
  - default.replication.factor=2
  - revision - 26
OpenSearch 2.13 ptf-test cluster
- r6g.2xlarge.search 4 data nodes
- r6g.large.search 3 dedicated master nodes

DB tables records size:

Tennent	instance count	holdings count	item count	authority count
fs09000000	30.587.260	30.646.075	31.734.818	16.535.572

Methodology/Approach

DI tests scenario (DI MARC Bib Create and Update) were started on Ramsons (rcp1) env with file splitting features enabled on a NON-ECS environment.

To run CI/CO - Ubuntu AWS instance was used as load generator
DI tests were started from UI

Test runs:

Test 1: Manually tested 5K, 10K, 25K, 50K, 100K consequentially records files, DI (DI MARC Bib Create ) started on Main tenant (fs09000000), and CICO with 8 users on background.
Test 2: Manually tested 5K, 10K, 25K, 50K, 100K consequentially records files, DI (DI MARC Bib Update) started on Main tenant (fs09000000), and CICO with 8 users on background.

All RCP1 modules

Cluster Resources - rcp1-pvt (Wed Jan 22 12:28:18 UTC 2025)

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	9	mod-remote-storage:3.3.3	2	4920	4472	0	3960	512	512
mod-ncip	7	mod-ncip:1.15.6	2	1024	896	0	768	88	128
mod-finance-storage	8	mod-finance-storage:8.7.3	2	1024	896	0	700	88	128
mod-agreements	9	mod-agreements:7.1.4	2	3184	2976	0	0	0	0
mod-ebsconet	9	mod-ebsconet:2.3.1	2	1248	1024	0	700	128	256
mod-organizations	7	mod-organizations:2.0.0	2	1024	896	0	700	88	128
mod-consortia	3	mod-consortia:1.2.2	2	5136	4776	0	4416	512	1024
edge-sip2	7	edge-sip2:3.3.1	2	1024	896	0	768	88	128
mod-settings	8	mod-settings:1.1.0	2	1024	896	0	768	88	128
mod-serials-management	9	mod-serials-management:1.1.2	2	2480	2312	0	1792	384	512
edge-dematic	7	edge-dematic:2.3.1	1	1024	896	0	768	88	128
mod-data-import	8	mod-data-import:3.2.4	1	2048	1844	0	1292	384	512
mod-search	21	mod-search:4.0.7	2	2592	2480	0	1440	512	1024
mod-inn-reach	4	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024
mod-record-specifications	16	mod-record-specifications:1.0.2	2	1024	896	0	768	88	128
mod-tags	7	mod-tags:2.3.0	2	1024	896	0	768	88	128
mod-authtoken	8	mod-authtoken:2.16.1	2	1440	1152	0	922	88	128
edge-courses	9	edge-courses:1.5.1	2	1024	896	0	768	88	128
mod-notify	7	mod-notify:3.3.0	2	1024	896	0	768	88	128
mod-inventory-update	7	mod-inventory-update:4.0.0	2	1024	896	0	768	88	128
mod-configuration	7	mod-configuration:5.11.0	2	1024	896	0	768	88	128
mod-orders-storage	8	mod-orders-storage:13.8.3	2	1024	896	0	700	88	128
edge-caiasoft	7	edge-caiasoft:2.3.2	2	1024	896	0	768	88	128
mod-login-saml	7	mod-login-saml:2.9.3	2	1024	896	0	768	88	128
mod-erm-usage-harvester	7	mod-erm-usage-harvester:5.0.1	2	1024	896	0	768	88	128
mod-password-validator	7	mod-password-validator:3.3.0	2	1440	1298	0	768	384	512
mod-gobi	7	mod-gobi:2.9.0	2	1024	896	0	700	88	128
mod-licenses	7	mod-licenses:6.1.2	2	2480	2312	0	1792	384	512
edge-dcb	7	edge-dcb:1.2.1	2	1024	896	0	768	88	128
mod-bulk-operations	9	mod-bulk-operations:2.1.8	2	3072	2600	0	1536	384	512
mod-fqm-manager	21	mod-fqm-manager:3.0.7	2	1024	896	0	768	88	128
mod-graphql	9	mod-graphql:1.13.1	2	1024	896	0	768	88	128
mod-finance	9	mod-finance:5.0.1	2	1024	896	0	768	88	128
mod-erm-usage	7	mod-erm-usage:5.0.0	2	1024	896	0	768	88	128
mod-batch-print	7	mod-batch-print:1.2.0	2	1024	896	0	768	88	128
mod-tlr	4	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	16	mod-lists:3.0.5	2	1024	896	0	768	88	128
mod-copycat	7	mod-copycat:1.7.0	2	1024	512	0	768	88	128
mod-entities-links	13	mod-entities-links:3.1.3	2	2592	2480	0	1440	0	1024
mod-permissions	17	mod-permissions:6.6.1	2	1684	1544	0	1024	384	512
pub-edge	5	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	9	mod-orders:12.9.9	2	2048	1740	0	1024	384	512
edge-patron	9	edge-patron:5.2.1	2	1024	896	0	768	88	128
edge-ncip	7	edge-ncip:1.10.1	2	1024	896	0	768	88	128
mod-marc-migrations	30	mod-marc-migrations:1.0.0	2	1024	896	0	768	88	128
edge-inn-reach	3	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	7	mod-users-bl:7.9.3	2	1440	1152	512	922	88	128
mod-oa	5	mod-oa:2.1.0-SNAPSHOT.66	2	2048	896	0	1800	88	512
mod-invoice	9	mod-invoice:5.9.2	2	1440	1152	0	922	88	128
mod-inventory-storage	13	mod-inventory-storage:28.0.4	2	4096	3690	0	3076	384	512
mod-user-import	7	mod-user-import:3.9.0	2	1024	896	0	768	88	128
mod-sender	7	mod-sender:1.13.0	2	1024	896	0	768	88	128
edge-oai-pmh	8	edge-oai-pmh:2.10.0	2	1512	1360	0	1440	384	512
mod-data-export-worker	9	mod-data-export-worker:3.3.6	2	3072	2800	0	2048	384	512
mod-rtac	7	mod-rtac:3.7.0	2	1024	896	0	768	88	128
mod-circulation-storage	9	mod-circulation-storage:17.3.3	2	2880	2592	0	1814	384	512
mod-source-record-storage	13	mod-source-record-storage:5.9.5	2	5600	5000	0	3500	384	512
mod-calendar	8	mod-calendar:3.2.0	2	1024	896	0	768	88	128
mod-event-config	7	mod-event-config:2.8.0	2	1024	896	0	768	88	128
mod-courses	7	mod-courses:1.4.11	2	1024	896	0	768	88	128
mod-circulation-item	7	mod-circulation-item:1.1.0	2	1024	896	0	0	0	0
mod-inventory	8	mod-inventory:21.0.5	2	2880	2592	0	1814	384	512
mod-email	9	mod-email:1.18.1	2	2800	2550	0	1800	384	512
mod-requests-mediated	12	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-di-converter-storage	8	mod-di-converter-storage:2.3.1	2	1024	896	0	768	88	128
mod-pubsub	9	mod-pubsub:2.15.3	2	1536	1440	0	922	384	512
mod-circulation	9	mod-circulation:24.3.8	2	2880	2592	0	1814	384	512
edge-orders	7	edge-orders:3.1.0	2	1024	896	0	768	88	128
edge-rtac	7	edge-rtac:2.8.0	2	1024	896	0	768	88	128
mod-template-engine	7	mod-template-engine:1.21.0	2	1024	896	0	768	88	128
mod-users	9	mod-users:19.4.5	2	1024	896	0	768	88	128
mod-patron-blocks	8	mod-patron-blocks:1.11.1	2	1024	896	0	768	88	128
mod-audit	9	mod-audit:2.10.2	2	1024	896	0	768	88	128
edge-fqm	9	edge-fqm:3.0.2	2	1024	896	0	768	88	128
mod-source-record-manager	13	mod-source-record-manager:3.9.5	2	5600	5000	0	3500	384	512
nginx-edge	5	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	7	mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512
nginx-okapi	5	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	7	okapi:6.1.0	3	1684	1440	0	922	384	512
mod-feesfines	7	mod-feesfines:19.2.1	2	1024	896	0	768	88	128
mod-invoice-storage	8	mod-invoice-storage:5.9.1	2	1872	1536	0	1024	384	512
mod-reading-room	8	mod-reading-room:1.0.0	2	1024	896	0	768	88	128
mod-service-interaction	8	mod-service-interaction:4.1.1	2	2048	1844	0	1290	384	512
mod-dcb	9	mod-dcb:1.2.4	2	1024	896	0	768	88	128
mod-data-export	10	mod-data-export:5.1.5	1	2592	2480	0	1440	88	1024
mod-patron	9	mod-patron:6.2.5	2	1024	896	0	768	88	128
mod-oai-pmh	7	mod-oai-pmh:3.14.3	2	4096	3690	2048	3076	384	512
edge-connexion	7	edge-connexion:1.3.1	2	1024	896	0	768	88	128
mod-notes	7	mod-notes:6.0.0	2	1024	896	0	952	384	512
mod-kb-ebsco-java	7	mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128
mod-login	8	mod-login:7.12.1	2	1440	1298	0	768	384	512
mod-organizations-storage	7	mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128
mod-data-export-spring	9	mod-data-export-spring:3.4.3	1	2048	1844	0	1536	384	512
pub-okapi	5	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
edge-erm	6	edge-erm:1.3.0	2	1024	896	0	768	88	128
mod-eusage-reports	7	mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128

Test artifacts:

Browser not supported