Overview

This is a report for a series of Check-in-check-out test runs together with Bulk edits against the Nolana release.

Infrastructure

PTF -environment ncp3

10 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-inventory	19.0.1	1	2	1024	2880	2592	512m	1814m
mod-inventory-storage	25.0.1	1	2	1024	2208 (1872 in MG)	1952 (1684 in MG)	512m	1440m
okapi	4.14.7	1	3	1024	1684 (1512 in MG)	1440 (1360 in MG)	512m	922m
mod-feesfines	18.1.0	1	2	128	1024	896	128	768
mod-patron-blocks	1.7.1	1	2	1024	1024	896	128	768
mod-pubsub	2.7.0	1	2	1024	1536 (1440 in MG)	1440 (1296 in MG)	512	922
mod-authtoken	2.12.0	1	2	512 (128 in MG)	1440 (1024 in MG)	1152 (896 in MG)	128	922 (768 in MG)
mod-circulation-storage	15.0.0	1	2	1024	1536 (1152 in MG)	1440 (1024 in MG)	512	896
mod-circulation	23.3.0	1	2	1024	1024	896	128	768

MG- Morning Glory release

Front End:

Item Check-in (folio_checkin-7.2.0)
Item Check-out (folio_checkout-8.2.0)

High-Level Summary

In general, there are insignificant regressions in performance for Check-in and check-out during bulk editing. Bulk edit duration is shorter than without CI/CO.
Memory usage is stable for all modules, except for mod-data-export-worker increased from 57% to 60% (Stable CPU utilization was after reaching 98% in Morning Glory).
The relevant services overall seem to occupy CPU resources nominally. Only mod-authtoken seems to have the spikes but the processes did not crash. CPU usage of all modules did not exceed 41%, except for mod-users up to 75%.
RDS CPU utilization did not exceed 41%.

Test Runs

Test	Virtual Users	Duration of CI/CO	Load generator size (recommended)	Load generator Memory(GiB) (recommended)
1.	8 users CI/CO + Items editing	30 mins	t3.medium	3
2.	8 users CI/CO + Users editing	10 mins	t3.medium	3
3.	8 users CI/CO + Holdings editing	10 mins	t3.medium	3
4.	8 users CI/CO+ Items, Users & Holdings editing	30 mins	t3.medium	3

Results

Response Times (Average of all tests listed above, in seconds)

	Check-in-check-out								Bulk edit
	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile (seconds)		Items	Users	Holdings
	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	10k records	2500 records	10k records
Test 1	0.431	0.793	0.417	0.770	0.444	0.807	0.504	0.930	20 min	-	-
Test 2	0.451	0.805	0.428	0.767	0.456	0.807	0.580	1.001	-	1 min	-
Test 3	0.428	0.790	0.410	0.761	0.434	0.795	0.521	0.923	-	-	11 min 20 sec
Test 4	0.436	0.797	0.426	0.776	0.450	0.811	0.504	0.941	29 min	1 min	11min 20 sec
CI/CO 8 users to compare	0.456	0.698	0.441	0.672	0.471	0.709	0.534	0.839	-	-	-

Response times are consistently good.

Memory Utilization

Memory usage is stable for all modules, except for mod-data-export-worker increased from 57% to 60% (98% in Morning Glory).

Modules CPUs Utilization

The relevant services overall seem to occupy CPU resources nominally. Only mod-authtoken seems to have the spikes but the processes did not crash. CPU usage of all modules did not exceed 41%, except for mod-users up to 75%.

Database and network

RDS CPU usage did not exceed 46%

Folio Development Teams

Combined Bulk edit + Check-in-check-out Test Report (Nolana)