Overview

This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records in the Poppy release.

Ticket: PERF-697 - Getting issue details... STATUS

Summary

There is significant performance improvement for data import in Poppy comparing with Orchid. Durations are closer to Nolana release results. CI/CO response times improved 10% in average with all DI jobs. There's only CI response time degraded in 100k job with 17%.

Comparing Orchid and Poppy releases DI durations in create jobs are up to 10% higher with 5k file and 5% for bigger files with parallel Check-in/Check-out than pure Data import results. DI durations are higher up to 30% in update jobs.

Response times of CI/CO in Poppy release are twice higher with Data Import job with 100k compared with pure CI/CO.

DI create jobs with CI/CO for files 10k, 25k, 50k, 100k perform better in Poppy release and DI update jobs perform better with 25k, 50k, 100k with CI/CO.

No memory leaks are observed.

Average CPU utilization increased for mod-inventory up to 20% comparing with Orchid. So it did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. The rest of services were almost on the same level in the same test in Orchid Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

DB needs more connections (in average +20 more) needed for the same tests as in Orchid for all create and update jobs.

Average DB CPU usage is the same as in Orchid - 95%.

Upd: During previous tests on pcp1 and ocp3 there were problems with DI jobs running big files (Create jobs 100k and higher, Update jobs with 25k and higher). The problem was solved after new deployment and updates of 13 modules in scope of ticket RANCHER-1121 - Getting issue details... STATUS and RANCHER-1114 - Getting issue details... STATUS and the large DI jobs are completing successfully now.

Test Runs

Test #	Scenario	Load level
1	DI MARC Bib Create	5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)
1	CICO	8 users
2	DI MARC Bib Update	5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)
2	CICO	8 users

Test Results

Data import

Total time for all Data Export jobs - 1 hour 16 minutes 47 seconds.

Profile	MARC File	DI Duration Poppy (hh:mm:ss)	Check In, Check Out Response time (8 users) Poppy
Profile	MARC File	DI Duration Poppy (hh:mm:ss)	CI Average sec	CO Average sec
DI MARC Bib Create (PTF - Create 2)	5K.mrc	00:02:53	0.901	1.375
	10K.mrc	00:04:32	0.902	1.47
	25K.mrc	00:11:14	1	1.571
	50K.mrc	00:21:55	0.981	1.46
	100K.mrc	00:47:02	1.018	1.491
Data Export MARC Bib (Export for Data Import updates)	5K.mrc	00:02:09	0.495	0.836
	10K.mrc	00:04:19	0.468	0.917
	25K.mrc	00:10:30	0.497	0.935
	50K.mrc	00:20:11	0.509	0.923
	100K.mrc	00:39:38
DI MARC Bib Update (PTF - Updates Success - 1)	5K.mrc	00:03:19	0.755	1.169
	10K.mrc	00:06:20	0.75	1.307
	25K.mrc	00:14:04	0.822	1.403
	50K.mrc	00:29:59	0.893	1.424
	100K.mrc	01:03:03	0.908	1.51

Check-in/Check-out without DI

Scenario	Load level	Request	Response time, sec Poppy
Scenario	Load level	Request	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.489	0.431
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	0.969	0.828

Comparison

CICO with DI comparison

Profile

MARC File

DI Duration

Deviation, % (compared DI Poppy without CICO and with CICO)

DI Delta, (hh:mm:ss) Orchid/Poppy

Check In, Check Out Response time (8 users)

Delta, %

without CI/CO

with CI/CO

Orchid

Poppy

Orchid/Poppy

Orchid*

Poppy

Orchid*

Poppy

CI Average sec

CO Average sec

CI Average sec

CO Average sec

DI MARC Bib Create (PTF - Create 2)

5K.mrc

00:04:30

00:02:39

00:05:01

00:02:53

+8.5% / 14 sec

- 00:02:08

0.961

1.442

0.901

1.375

-6.24%

-4.65%

10K.mrc

00:09:25

00:05:00

00:09:06

00:04:32

-9.3% / 28 sec

- 00:04:35

1.058

1.624

0.902

1.47

-14.74%

-9.48%

25K.mrc

00:22:16

00:11:15

00:24:28

00:11:14

-0.2% / 1 sec

- 00:13:14

1.056

1.621

1.571

-5.30%

-3.08%

50K.mrc

00:39:27

00:22:16

00:43:03

00:21:55

-1.5% / 21 sec

- 00:21:09

0.936

1.519

0.981

1.46

4.81%

-3.88%

100K.mrc

01:38:00

00:49:58

01:35:50

00:47:02

-5.8% / 2 min 56 sec

- 00:48:49

0.868

1.468

1.018

1.491

17.28%

1.57%

DI MARC Bib Update (PTF - Updates Success - 1)

5K.mrc

00:04:02

00:02:28

00:04:52

00:03:19

+34% / 51 sec

- 00:01:33

0.855

1.339

0.755

1.169

-11.70%

-12.70%

10K.mrc

00:08:10

00:05:31

00:09:22

00:06:20

+15% / 49 sec

- 00:03:03

0.916

1.398

0.75

1.307

-18.12%

-6.51%

25K.mrc

00:19:39

00:14:50

00:24:02

00:14:04

-5.1% / 46 sec

- 00:09:58

0.922

1.425

0.822

1.403

-10.85%

-1.54%

50K.mrc

00:38:30

00:32:53

00:47:13

00:29:59

-8.8% / 2 min 54 sec

- 00:17:15

0.904

1.456

0.893

1.424

-1.22%

-2.20%

100K.mrc

01:33:00

01:14:39

01:40:25

01:03:03

-15.5% / 11 min 36 sec

- 00:37:23

0.838

1.415

0.908

1.51

8.35%

6.71%

The following table compares test results of current release (Orchid) to the previous release numbers (Orchid) and to the baselines Poppy results (CICO without DI and DI without CICO).

* Orchid DI and CICO results are taken from Data Import with Check-ins Check-outs Orchid.

*** Completed with errors

Detailed CICO response time comparison

Scenario	Load level	Request	Response time, sec Orchid		Response time, sec Poppy
Scenario	Load level	Request	95 perc	average	95 perc	average
Circulation Check-in/Check-out (without Data import)	8 users	Check-in	0.489	0.394	0.489	0.431
Circulation Check-in/Check-out (without Data import)	8 users	Check-out	0.793	0.724	0.969	0.828

Detailed CICO response time comparison for CICO with DI in Poppy

Request*	Response time (avg, sec)
Request*	Pure CICO	CICO + 100K MARC BIB Create	CICO + 100K MARC BIB Update
Check-Out Controller	0.828	1.491	1.51
Check-In Controller	0.431	1.018	0.908
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	0.266	0.647	0.718
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	0.187	0.57	0.477
GET_circulation/loans (Submit_barcode_checkout)	0.128	0.233	0.215
GET_inventory/items (Submit_barcode_checkin)	0.048	0.126	0.118
GET_inventory/items (Submit_barcode_checkout)	0.046	0.125	0.117
GET_note-links (Submit_barcode_checkout)	0.046	0.024	0.024
GET_users (Submit_patron_barcode)	0.037	0.041	0.037
GET_circulation/loans (Submit_patron_barcode)	0.028	0.03	0.049
GET_automated-patron-blocks (Submit_patron_barcode)	0.024	0.026	0.024
GET_users (Get_check_in_page)	0.023	0.054	0.051

*Top-10 requests were taken for analysis.

Response time

DI MARC BIB Create + CICO

DI Bib Update + CICO

Service CPU Utilization

Average CPU utilization did not exceed 150% for all the modules. The highest consumption observed from mod-inventory. That is 20% higher than in Orchid release. But the rest of services were almost on the same level in the same test in Orchid Data Import with Check-ins Check-outs Orchid and didn't exceed 60%.

Spikes of mod-data-import observed in Data Import jobs with 50k files up to 130%. for jobs and 320% spike for 100k. For Data Import jobs with 5k, 10k, 25k files CPU utilization didn't exceed 35%

DI MARC BIB Create + CICO

DI MARC BIB Update+ CICO

Service Memory Utilization

There is memory utilization increasing observed which is caused by previous modules restarting (everyday cluster shut down process).

DI MARC BIB Create + CICO

It was observed that during tests mod-source-record-storage grew every 30 minutes with 10% from 38% up to 67% but it's still less than in the same test in Orchid.

Memory consumption before tests for mod-search was 39% and for mod-inventory - 75%. During test with 100k file mod-search grew up to 75% and mod-inventory up to 91%.

MARC BIB Update + CICO

After 30 minutes of tests start mod-source-record-storage grew from 67% to 75% and didn't change during tests with 50k and 100k. So 75% of memory consumption can be defined as a baseline under load for this service. Mod-inventory grew from 91% to 99%.

DB CPU Utilization

Average DB CPU usage during data import is about 95% The same results if to compare with the same tests in Orchid.

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB Connections

Average connection count during data import is about 300 connections for create jobs that is 30 connections higher than in Orchid. For update jobs - 280 connections that is higher

DI MARC BIB Create + CICO

MARC BIB Update + CICO

DB load

DI MARC BIB Create + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *

INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

MARC BIB Update + CICO

Top SQL-queries:

INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)

insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

Appendix

Infrastructure

PTF -environment pcp1

10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections
db.r6g.xlarge
32 GiB 4 vCPUs 2731
MSK tenant
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize	R/W split enabled
pcp1-pvt

mod-remote-storage	10	3.0.0	2	4920	4472	1024	3960	512	512	FALSE
mod-data-import	18	3.0.7	1	2048	1844	256	1292	384	512	FALSE
mod-authtoken	13	2.14.1	2	1440	1152	512	922	88	128	FALSE
mod-configuration	9	5.9.2	2	1024	896	128	768	88	128	FALSE
mod-users-bl	9	7.6.0	2	1440	1152	512	922	88	128	FALSE
mod-inventory-storage	12	27.0.3	2	4096	3690	2048	3076	384	512	FALSE
mod-circulation-storage	12	17.1.3	2	2880	2592	1536	1814	384	512	FALSE
mod-source-record-storage	15	5.7.3	2	5600	5000	2048	3500	384	512	FALSE
mod-inventory	11	20.1.3	2	2880	2592	1024	1814	384	512	FALSE
mod-di-converter-storage	15	2.1.2	2	1024	896	128	768	88	128	FALSE
mod-circulation	12	24.0.8	2	2880	2592	1536	1814	384	512	FALSE
mod-pubsub	11	2.11.2	2	1536	1440	1024	922	384	512	FALSE
mod-patron-blocks	9	1.9.0	2	1024	896	1024	768	88	128	FALSE
mod-source-record-manager	14	3.7.4	2	5600	5000	2048	3500	384	512	FALSE
mod-quick-marc	9	5.0.0	1	2288	2176	128	1664	384	512	FALSE
nginx-okapi	9	2023.06.14	2	1024	896	128	0	0	0	FALSE
okapi-b	11	5.1.2	3	1684	1440	1024	922	384	512	FALSE
mod-feesfines	10	19.0.0	2	1024	896	128	768	88	128	FALSE
pub-okapi	9	2023.06.14	2	1024	896	128	768	0	0	FALSE

Methodology/Approach

DI tests were started from UI with 5 min pauses between the tests.

Additional links

Grafana dashboard:

MARC Bib Create + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1700738030629&to=1700749313428&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

MARC Bib Update + CICO

http://carrier-io.int.folio.ebsco.com/grafana/d/SqzWB26nk/jmeter-performance-check-in-check-out?orgId=1&from=1700755239000&to=1700766759000&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_3&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731

Browser not supported