Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Quesnelia release.

Ticket: PERF-824 - Getting issue details... STATUS

Summary

Data import tests finished successfully for all files. Duration of DI grew in correspondence with the number of records in files.
Check-in and Check-out with 8 virtual users was performed during DI Create and Update jobs. No issues.
Data Import in Quesnelia with CICO perform faster than without it.
Comparing Poppy and Quesnelia releases
- Check-in / Check-out perform better in Quesnelia. Response time improved during Create jobs - 30% in Average, and during DI Update jobs - 15% in Average.
- DI durations do not differ much
Resource utilization
- Average CPU utilization did not exceed 150% for all modules. The highest consumption observed from mod-inventory - 144% in DI Update job with 25k file. The same maximum level as in Poppy.
- No memory leaks observed during tests.
- Average DB CPU usage during data import has 5% decreasing in Quesnelia - close to 90%.
- Average connection count during data import is about 750 connections for create jobs that is 450 connections higher than in Poppy. For update jobs - 730 connections.

Test Runs

Test #	Scenario	Load level
1	DI MARC Bib Create	5K, 10K, 25K, 50K, 100K consequentially
1	CICO	8 users
2	DI MARC Bib Update	5K, 10K, 25K, 50K, 100K consequentially
2	CICO	8 users

Test Results

Data import

Files for Data Import update jobs prepared during previous tests. So no need to run Data Export.

Profile	MARC File	DI Duration Quesnelia (hh:mm:ss)	Check In, Check Out Response time (8 users) Quesnelia
Profile	MARC File	DI Duration Quesnelia (hh:mm:ss)	CI Average sec	CO Average sec
DI MARC Bib Create (PTF - Create 2)	5K.mrc	00:02:32	0.645	0.901
	10K.mrc	00:05:03	0.628	0.922
	25K.mrc	00:11:58	0.639	0.960
	50K.mrc	00:23:29	0.678	1.003
	100K.mrc	00:46:07	0.686	0.998
DI MARC Bib Update (PTF - Updates Success - 1)	5K.mrc	00:03:24	0.628	0.975
	10K.mrc	00:06:29	0.664	1.018
	25K.mrc	00:16:15	0.717	1.062
	50K.mrc	00:33:33	0.721	1.071
	100K.mrc	01:10:14	0.739	1.081

Check-in/Check-out without DI

Scenario	Load level	Request	Response time, sec Quesnelia
Scenario	Load level	Request	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.609	0.521
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	1.070	0.803

Comparison

CICO with DI comparison

DI duration results without Check-In and Check-Out for Quesnelia were taken from the report Data Import test report (Quesnelia)[non-ECS].

Profile

MARC File

DI Duration

Deviation, % (compared DI Quesnelia without CICO and with CICO)

DI Delta, (hh:mm:ss) Poppy/Quesnelia (with CICO)

Check In, Check Out Response time (8 users)

Delta, %

without CI/CO

with CI/CO

Poppy

Quesnelia

Poppy/Quesnelia

Poppy

Quesnelia

Poppy

Quesnelia

CI Average sec

CO Average sec

CI Average sec

CO Average sec

CI

CO

DI MARC Bib Create (PTF - Create 2)

5K.mrc

00:02:39

00:03:20

00:02:53

00:02:32

- 24% / 48 sec

- 00:00:21

0.901

1.375

0.645

0.901

-28.41%

-34.47%

10K.mrc

00:05:00

00:06:00

00:04:32

00:05:03

- 15% / 57 sec

+ 00:00:31

0.902

1.47

0.628

0.922

-30.38%

-37.28%

25K.mrc

00:11:15

00:13:41

00:11:14

00:11:58

- 12% / 1 min 43 sec

+ 00:00:44

1

1.571

0.639

0.96

-36.10%

-38.89%

50K.mrc

00:22:16

00:21:59

00:21:55

00:23:29

+ 6% / 1 min 34 sec

+ 00:01:34

0.981

1.46

0.678

1.003

-30.89%

-31.30%

100K.mrc

00:49:58

00:40:16

00:47:02

00:46:07

+ 14% / 5 min 51 sec

- 00:00:55

1.018

1.491

0.686

0.998

-32.61%

-33.07%

DI MARC Bib Update (PTF - Updates Success - 6)

5K.mrc

00:02:28

00:07:10

00:03:19

00:03:24

- 52% / 3 min 46 sec

+ 00:00:05

0.755

1.169

0.628

0.975

-16.82%

-16.60%

10K.mrc

00:05:31

00:10:27

00:06:20

00:06:29

- 37% / 3 min 58 sec

+ 00:00:09

0.75

1.307

0.664

1.018

-11.47%

-22.11%

25K.mrc

00:14:50

00:23:16

00:14:04

00:16:15

- 30% / 7 min 1 sec

+ 00:02:11

0.822

1.403

0.717

1.062

-12.77%

-24.31%

50K.mrc

00:32:53

00:40:52

00:29:59

00:33:33

- 17% / 7 min 19 sec

+ 00:03:34

0.893

1.424

0.721

1.071

-19.26%

-24.79%

100K.mrc

01:14:39

01:02:00

01:03:03

01:10:14

+ 13% / 8 min 14 sec

+ 00:07:11

0.908

1.51

0.739

1.081

-18.61%

-28.41%

The following table compares test results of current release (Quesnelia ) to the previous release (Poppy).

* Poppy DI and CICO results are taken from Data Import with Check-ins Check-outs Poppy

Detailed CICO response time comparison

Scenario	Load level	Request	Response time, sec Poppy		Response time, sec Quesnelia
Scenario	Load level	Request	95 perc	average	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.489	0.431	0.609	0.521
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	0.969	0.828	1.070	0.803

Detailed CICO response time for CICO with DI in Poppy

Request*	Response time (avg, sec)
Request*	Pure CICO	CICO + 100K MARC BIB Create	CICO + 100K MARC BIB Update
Request	Pure	Create	Update
Check-Out Controller	803.14	998.18	1081.19
Check-In Controller	521.62	686.69	739.48
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	289.16	397.54	432.5
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	210.45	318.34	331.21
GET_circulation/loans (Submit_barcode_checkout)	150.84	186.52	203.77
GET_users (Get_check_in_page)	76.98	90.13	113.47
GET_inventory/items (Submit_barcode_checkin)	59.48	89.73	99.01
GET_inventory/items (Submit_barcode_checkout)	55.23	79.96	85.51
GET_circulation/requests_status_openAwaitingPickup (Submit_patron_barcode)	20.51	22.54	24.33
GET_circulation/requests (Submit_barcode_checkin)	20.17	22.5	23.68
GET_circulation/loans (Submit_patron_barcode)	20.06	22.33	23.63

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI MARC BIB Update + CICO

Service CPU Utilization

Average CPU utilization did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. The same level as in Poppy.

Spike for mod-data-import module observed instantly after the start in DI Create job with 100k file by 150%. For other tests it didn't exceed 110%.

CPU Create, Update 100k

Service	100k DI Create	100k DI Update
mod-inventory-b	129	125.13
mod-quick-marc-b	110.24	96.74
mod-di-converter-storage-b	62.6	80.57
nginx-okapi	48.55	83.36
mod-source-record-storage-b	23.1	34.47
okapi-b	20.13	39.37
mod-inventory-storage-b	16.81	17.5
mod-source-record-manager-b	14.95	16.85
mod-pubsub-b	8.02	9.63
mod-dcb-b	7.89	8.99
mod-users-b	7.09	20.8
mod-configuration-b	2.61	9.46
mod-feesfines-b	2.61	7.75
mod-patron-b	2.59	2.46
mod-oa-b	2.26	2.24
mod-entities-links-b	1.89	2.46
mod-authtoken-b	1.89	14.56
mod-circulation-storage-b	1.81	2.05
mod-search-b	1.76	1.33
mod-data-import-b	1.57	1.61
edge-patron-b	0.98	0.94
mod-users-bl-b	0.66	0.72
mod-circulation-b	0.4	1.57
mod-patron-blocks-b	0.32	0.67
pub-okapi	0.13	3.78
pub-edge	0.09	0.13

DI MARC BIB Create and Update + CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

Top 5 modules with highest memory consumption: mod-inventory - 87%, mod-search - 87%, mod-oa - 75%, mod-dcb - 68%, mod-source-record-manager - 66%,

Mod-data-export-worker-b was no level of 93% before and after tests.

Memory consumption

Memory
mod-inventory-b	86.92
mod-search-b	86.83
mod-oa-b	75.51
mod-dcb-b	68.81
mod-source-record-manager-b	66.84
mod-source-record-storage-b	58.86
mod-users-b	42.52
mod-pubsub-b	40.83
mod-data-import-b	39.75
mod-di-converter-storage-b	38.99
mod-users-bl-b	38.92
okapi-b	37.63
mod-feesfines-b	34.15
mod-patron-blocks-b	32.59
mod-configuration-b	29.06
mod-quick-marc-b	27.48
mod-inventory-storage-b	25.77
mod-entities-links-b	24.85
mod-patron-b	24.7
mod-circulation-b	24.02
mod-circulation-storage-b	22.78
mod-authtoken-b	21.42
edge-patron-b	18.02
nginx-okapi	4.85
pub-okapi	4.58
pub-edge	4.46

DB CPU Utilization

Average DB CPU usage during data import is about 90% It decreased 5% from 95% in Poppy.

DB Connections

Average connection count during data import is about 750 connections for create jobs that is 450 connections higher than in Poppy. For update jobs - 730 connections.

DB load

Create jobs

Update jobs

Top SQL-queries:

Create jobs

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Update jobs

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

UPDATE fs09000000_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id='99b071cf-f789-4dfe-a238-d62c40bccfc0'

Appendix

Infrastructure

PTF -environment qcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Task count for modules mod-agreements, mod-serials-management, mod-graphql set to 0 during tests.

Modules

All modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcp1-pvt
Tue Apr 30 18:21:36 UTC 2024
mod-remote-storage	3	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-ncip	3	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-finance-storage	3	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-agreements	3	mod-agreements:7.0.0	0	1592	1488	128	0	0	0
mod-ebsconet	3	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-organizations	3	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-consortia	2	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
edge-sip2	2	edge-sip2:3.2.0-SNAPSHOT.209	2	1024	896	128	768	88	128
mod-serials-management	3	mod-serials-management:1.0.0	0	2480	2312	128	1792	384	512
mod-settings	3	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-data-import	6	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
edge-dematic	3	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-search	3	mod-search:3.2.0	2	2592	2480	2048	1440	512	1024
mod-inn-reach	2	mod-inn-reach:3.2.0-SNAPSHOT.86	2	3600	3240	1024	2880	512	1024
mod-tags	3	mod-tags:2.2.0	2	1024	896	128	768	88	128
edge-courses	3	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-authtoken	4	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
mod-inventory-update	3	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	3	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	3	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	3	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	3	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	3	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	3	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-licenses	3	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	3	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-password-validator	3	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-bulk-operations	3	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	3	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	3	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	4	mod-graphql:1.12.1	0	1024	896	128	768	88	128
mod-finance	3	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	3	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	4	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	3	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	3	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	4	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	6	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
pub-edge	3	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	3	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	3	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	3	edge-ncip:1.9.2	2	1024	896	128	768	88	128
edge-inn-reach	2	edge-inn-reach:3.1.1-SNAPSHOT.45	2	1024	896	128	768	88	128
mod-users-bl	3	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-oa	2	mod-oa:2.1.0-SNAPSHOT.62	2	1024	896	128	768	88	128
mod-inventory-storage	3	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-invoice	4	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-user-import	3	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	4	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	3	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	3	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	3	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	3	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-calendar	3	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-source-record-storage	3	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-event-config	3	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	3	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	3	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	3	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	3	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-pubsub	3	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-circulation	3	mod-circulation:24.2.0	2	2880	2592	1536	1814	384	512
mod-di-converter-storage	3	mod-di-converter-storage:2.2.0	2	1024	896	128	768	88	128
edge-rtac	3	edge-rtac:2.7.1	2	1024	896	128	768	88	128
edge-orders	3	edge-orders:3.0.0	2	1024	896	128	768	88	128
mod-users	4	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-template-engine	3	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	3	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	3	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	3	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	3	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
nginx-edge	3	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	3	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	3	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	4	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	3	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	3	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-dcb	4	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-service-interaction	3	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-data-export	10	mod-data-export:5.0.4	1	2048	1524	1024	0	0	0
mod-patron	3	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-oai-pmh	3	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	3	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-kb-ebsco-java	3	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-notes	3	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-data-export-spring	3	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
mod-organizations-storage	3	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-login	3	mod-login:7.11.0	2	1440	1298	1024	768	384	512
pub-okapi	3	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	3	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Methodology/Approach

To run CI/CO - Ubuntu AWS instance was used as load generator

DI tests were started from UI

Folio Development Teams

Data Import with Check-ins Check-outs (Quesnelia)[non-ECS]

Overview

Summary

Test Runs

Test Results

Comparison

Response time

DI MARC BIB Create + CICO

DI MARC BIB Update + CICO

Service CPU Utilization

DI MARC BIB Create and Update + CICO

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Create jobs

Update jobs

Appendix

Infrastructure

Methodology/Approach

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731