20-users tests	Avg	Max
mod-users	21%	21%
mod-pubsub	5%	5%
okapi	14%	14%
mod-circulation	4%	4%
mod-circulation-storage	5%	5%
mod-inventory	7%	7%
mod-inventory-storage	7%	7%
mod-patron-blocks	1%	1%
mod-feesfines	17%	17%
mod-authtoken	31%	20-users tests	Avg	Max
		mod-users	21%	21%
		mod-pubsub	5%	5%
		okapi	14%	14%
		mod-circulation	4%	4%
		mod-circulation-storage	5%	5%
		mod-inventory	7%	7%
		mod-inventory-storage	7%	7%
		mod-patron-blocks	1%	1%
		mod-feesfines	17%	17%
		mod-authtoken	31%	78%

Overview

This is a report for a series of Check-in-check-out tests (CI/CO) + Data Import (DI) tests run against the Nolana release and with DB R/W Split enabled.

Infrastructure

PTF -environment ncp3

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-inventory	19.0.1	1	2	1024	2880	2592	512m	1814m
mod-inventory-storage	25.0.1	1-2	2	1024	2208 (1872 in MG)	1952 (1684 in MG)	512m	1440m
okapi	4.14.7	1-2	3	1024	1684 (1512 in MG)	1440 (1360 in MG)	512m	922m
mod-feesfines	18.1.0	1-2	2	128	1024	896	128	768
mod-patron-blocks	1.7.1	1-2	2	1024	1024	896	128	768
mod-pubsub	2.7.0	1-2	2	1024	1536 (1440 in MG)	1440 (1296 in MG)	512	922
mod-authtoken	2.12.0	1-2	2	512 (128 in MG)	1440 (1024 in MG)	1152 (896 in MG)	128	922 (768 in MG)
mod-circulation-storage	15.0.0	1-2	2	1024	1536 (1152 in MG)	1440 (1024 in MG)	512	896
mod-circulation	23.3.0	1	2	1024	1024	896	128	768
mod-configuration	5.9.0	1-2	2	128	1024	896	128m	768m
mod-users	19.0.0	1-2	2	258	1024	896	128m	768m
mod-remote-storage	1.7.0	1-2	2	128	1872	1684	512m	1178m
mod-data-import-cs	1.15.1	1-2	2	258	1024	896	128m	768m
mod-quick-marc	2.5.0	1-2	1	128	2288 (2098 in MG)	2176 (1920 in MG)	512m	1664m
mod-source-record-storage	5.5.2	1-2	2	1024	1536 (1440 in MG)	1440 (1296 in MG)	512m	908m
mod-source-record-manager	3.5.4	2-3	2	1024	4096 (2048 in MG)	3688 (1844 in MG)	512m (800M in MG)	2048m (1024m in MG)
mod-data-import	2.6.1	1-2	1	256	2048	1844	512m	1292m

MG- Morning Glory release

Front End:

Item Check-in (folio_checkin-7.2.0)
Item Check-out (folio_checkout-8.2.0)

Modules that had R/W split enabled

1	mod-inventory-storage
2	mod-feesfines
3	mod-patron-blocks
4	mod-pubsub
5	mod-authtoken
6	mod-circulation-storage
7	mod-configuration
8	mod-users
9	mod-remote-storage
10	mod-data-import-cs
11	mod-quick-marc
12	mod-source-record-storage
13	mod-source-record-manager
14	mod-data-import

High-Level Summary

Response times are better for DB R/W Split enabled compared to disabled, especially for 20 users of the CI/CO with the DI MARC BIB Create test, for both CI/CO and DI as well:
1. For 20 users CI/CO with DI MARC BIB Create with R/W Split enabled response time is 50% better compared to R/W Split disabled.
2. For 8 users CI/CO with DI MARC BIB Update with R/W Split enabled response time is 20% better compared to R/W Split disabled.
3. For DI MARC BIB Create with CI/CO 20 users (19 min 50 sec) for R/W Split enabled response time is 12,7% better compared to R/W Split disabled (22 min 40 sec).
4. For DI MARC BIB Update with CI/CO 8 users (22min) for R/W Split enabled response time is 12% better compared to R/W Split disabled (25 min).
5. For DI MARC BIB Create with CI/CO 20 users (19 min 50 sec) for R/W Split enabled response time is about 16.6% better compared to baseline DI MARC BIB Create (without CI/CO )(23 min 43 s).
6. For DI MARC BIB Update with CI/CO 8 users (22min) for R/W Split enabled response time is about 8.7% better compared to baseline DI MARC BIB Update (without CI/CO )(24 min 5 s).
When DB R/W split is enabled:
1. requests to the database are separated between 2 instances of DB
2. the database uses more connections in total
3. writer instance uses less CPU (load distribution between reader and writer and reader)
Services' memory utilization increases during the test run. Probably it is a result of the modules' restart right before the tests.
Modules and instances of CPU usage are similar for tests with DB R/W Split enabled and disabled.

Test Runs

Test #	CI/CO Virtual Users	CI/CO Duration	Data Import	Data Import profile	R/W Split
1.	20 users	30 mins	50K MARC BIB Create	PTF - Create 2	disabled
2.	20 users	30 mins	50K MARC BIB Create	PTF - Create 2	enabled
3.	8 users	30 mins	50K MARC BIB Update	PTF - Updates Success - 1	disabled
4.	8 users	30 mins	50K MARC BIB Update	PTF - Updates Success - 1	enabled

Results

Response Times (Average of CI/CO time with MARC BIB)

			Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)
Test #	Short test description	DI time	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out
1.	MARC BIB Create (R/W disabled) + CI/CO	22 min 40 sec	1.015	1.746	0.956	1.667	1.096	1.816	1.569	2.448
2.	MARC BIB Create (R/W enabled) + CI/CO r/w enabled	19 min 50 sec	0.484	0.877	0.453	0.822	0.491	0.869	0.564	0.966
3.	MARC BIB Update (R/W disabled)+ CI/CO	25 min	0.578	1.070	0.523	0.985	0.577	1.065	0.689	1.250
4.	MARC BIB Update (R/W enabled)+ CI/CO r/w enabled	22min	0.463	0.834	0.431	0.799	0.466	0.845	0.537	0.994

Response times are better for DB R/W Split enabled compared to disabled, especially for 20 users of the CI/CO with the DI MARC BIB Create test.

Comparisons

The following tables compare Nolana's test results of CI/CO + DI MARC BIB for tests with DB R/W Split enabled and disabled. For 20 users CI/CO with DI MARC BIB Create with R/W Split enabled response time is 50% better compared to R/W Split disabled. For 8 users CI/CO with DI MARC BIB Update with R/W Split enabled response time is 20% better compared to R/W Split disabled. For DI MARC BIB Update (with CI/CO 8 and 20 users) with R/W Split enabled response time is 12% better compared to R/W Split disabled

				Average (seconds)				50th %tile (seconds)				75th %tile (seconds)				95th %tile (seconds)
Test #	Short test description	DI time	Compared to MARC BIB baseline (without CI/CO)	Check-in	Compared to CI/CO baseline (without DI)	Check-out	Compared to CI/CO baseline (without DI)	Check-in	Compared to CI/CO baseline (without DI)	Check-out	Compared to CI/CO baseline (without DI)	Check-in	Compared to CI/CO baseline (without DI)	Check-out	Compared to CI/CO baseline (without DI)	Check-in	Compared to CI/CO baseline (without DI)	Check-out	Compared to CI/CO baseline (without DI)
1.	MARC BIB Create (R/W disabled) + CI/CO	22 min 40 sec	4.4%	1.015	-138%	1.746	-165%	0.956	-130%	1.667	-161%	1.096	-154%	1.816	-174%	1.569	-220%	2.448	-222%
2.	MARC BIB Create (R/W enabled) + CI/CO r/w enabled	19 min 50 sec	16.6%	0.484	-13%	0.877	-33%	0.453	-9%	0.822	-28%	0.491	-14%	0.869	-31%	0.564	-15%	0.966	-27%
Comparison R/W enabled and disabled		12.7%		52.3%		49.7%		52.6%		50.6%		55.2%		52.1%		64%		60.5%
3.	MARC BIB Update (R/W disabled)+ CI/CO	25 min	-3.73%	0.578	-26%	1.070	-53%	0.523	-18%	0.985	-46%	0.577	-22%	1.065	-50%	0.689	-29%	1.250	-48%
4.	MARC BIB Update (R/W enabled)+ CI/CO r/w enabled	22min	8.7%	0.463	-1.5%	0.834	-19%	0.431	2.2%	0.799	-18%	0.466	1%	0.845	-19%	0.537	-0.5%	0.994	-18%
Comparison R/W enabled and disabled		12%		19.9%		22%		17.5%		18.9%		19.2%		20.6%		22%		20.4%
5.	CI/CO baseline 8 users (without DI)	-	-	0.456		0.698		0.441		0.672		0.471		0.709		0.534		0.839
6.	CI/CO baseline 20 users (without DI)	-	-	0.425		0.658		0.414		0.638		0.430		0.661		0.489		0.758
7.	MARC BIB Create baseline (without CI/CO)	23 min 43 s	-	-		-		-		-		-		-		-		-
8.	MARC BIB Update baseline (without CI/CO)	24 min 5 s	-	-		-		-		-		-		-		-		-

Response Time Comparison Graph

MARC BIB Create

MARC BIB Update

Memory Utilization

Services' memory utilization increases during the test run. Probably it is a result of the modules' restart.

MARC BIB Create

Modules CPUs Utilization

MARC BIB Create

MARC BIB Update

Database and network

MARC BIB Create

MARC BIB Update

Database connections

MARC BIB Create

MARC BIB Update

SELECT queries DB writer instance while R/W split enabled:

SELECT save_event_and_decrease_flow_control_counter($1, $2)
select max("records_lb"."generation") as "generation" from "records_lb" join "snapshots_lb" on "records_lb"."snapshot_id" = "snapshots_lb"."id" where ("records_lb"."matched_id" = cast($1 as uuid) and "snapshots_lb"."status" in ($2::"job_execution_status", $3::"job_execution_status", $4::"job_execution_status") and "snapshots_lb"."updated_date" < (select "snapshots_lb"."processing_started_date" from "snapshots_lb" where "snapshots_lb"."id" = cast($5 as uuid)))
SELECT item.jsonb::text, holdings_record.jsonb::text FROM fs09000000_mod_inventory_storage.item LEFT JOIN fs09000000_mod_inventory_storage.holdings_record ON holdings_record.id = $2 WHERE item.id = $1
select "records_lb"."id", "records_lb"."snapshot_id", "records_lb"."matched_id", "records_lb"."generation", "records_lb"."record_type", "records_lb"."external_id", "records_lb"."state", "records_lb"."leader_record_status", "records_lb"."order", "records_lb"."suppress_discovery", "records_lb"."created_by_user_id", "records_lb"."created_date", "records_lb"."updated_by_user_id", "records_lb"."updated_date", "records_lb"."external_hrid" from "records_lb" where ("records_lb"."matched_id" = cast($1 as
select "id", "content" from "marc_records_lb" where "id" = cast($1 as uuid)
select "snapshots_lb"."id", "snapshots_lb"."status", "snapshots_lb"."processing_started_date", "snapshots_lb"."created_by_user_id", "snapshots_lb"."created_date", "snapshots_lb"."updated_by_user_id", "snapshots_lb"."updated_date" from "snapshots_lb" where "snapshots_lb"."id" = cast($1 as uuid)
select "raw_records_lb"."id", "raw_records_lb"."content" from "raw_records_lb" where "raw_records_lb"."id" = cast($1 as uuid)
SELECT jsonb FROM fs09000000_mod_inventory_storage.item WHERE holdingsRecordId = ?

Miscellaneous

Response time comparison in Grafana for
CICO + DI MARC BIB Create Tests with R/W Split enabled http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1671099937955&to=1671105782727&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
Response time comparison in Grafana for
CICO + DI MARC BIB Update Tests with R/W Split enabled http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1671622645242&to=1671628366157&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All

Browser not supported