It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.

Overview

The target of the test is to investigate the impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations and to compare tested results with direct non-intersecting activities' baselines.

PERF-291 - Getting issue details... STATUS

Summary

Test results don't show a huge negative impact of Check-In/Check-Out activities during Data Import - MARC Holdings operations:
- Response time of CI/CO activities during Data Import - MARC Holdings was increased (degradation) by more than 25%.
- Duration of Data Import - MARC Holdings with CI/CO activities was decreased by 66%.
High Memory usage was detected for mod-inventory-storage (memory usage is 106%, it's >100%) but stable during the test
Memory usage of mod-source-record-storage was increased from 55% to 58%
All modules show expected CPU behaviour and typical workload, except 'mod-source-record-manager' and 'mod-authtoken'
During the test, there were detected several requests with more than a 1-second response time. (see Additional Screenshots of graphs or charts)

Recommendations & Jiras

During future tests, pay attention to:
- Memory usage of mod-inventory-storage, mod-source-record-storage
- CPU usage of mod-source-record-manager, mod-authtoken
To discuss Data Import - MARC Holdings' behaviour during Check-In/Check-Out activities Martin Tran Artem Vronik Roman_Fedynyshyn

Test Runs

Test #	Test Conditions	Duration	Load generator size (recommended)	Load generator Memory(GiB) (recommended)
1.	8 users CI/CO + DI (1k, 5k, 10k, 80k) MARC Holding	60 min	t3.medium	3

Results

Response Times

Grafana: http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&from=1672996256462&to=1673000063590

	Check-in/Check-out		Data Import
	Average, ms		MARC Holding
	Check-in Controller	Check-out Controller	1k Records	5k Records	10k Records	80k records
Test 1	573.79	924.19	"00:00:35.777"	"00:01:28.613"	"00:02:42.095"	"00:20:12.114"

Comparisons

Test #	Report link	Type of results	Check-In, ms	Check-Out, ms	DI MARC Holding
Test #	Report link	Type of results	Check-In, ms	Check-Out, ms	1k	5k	10k	80k
Test #1 - CI/CO 8 users only	Check-in-check-out Test Report (Nolana)	[Baseline]	456.00	698.00	x	x	x	x
Test #2 - DI MARC Holding only	Data Import Create MARC holdings records [Nolana]	[Baseline]	x	x	00:00:32.700	00:04:20.800	00:03:24.900	00:21:22.600
Test #3 - CI/CO + DI MARC Holding *CI/CO stable	Data Import (MARC Holdings) with CI/CO [Nolana]	[Tested]	573.79 461.17	924.19 711.94	00:00:35.777	00:01:28.613	00:02:42.095	00:20:12.114
Diff:			+25.83% *+1.13%	+32.41% *+1.99%	+9.41%	-66.02%	-20.89%	-5.50%

*Ci/CO stable:

Memory Utilization

mod-inventory-storage memory usage is 106%, it's >100% but stable during test

mod-source-record-storage memory usage was increased from 55% to 58%

CPU Utilization

All modules show expected and some workload, except 'mod-source-record-manager' and 'mod-authtoken':

mod-source-record-manager: MIN - 2.74%, MAX - 50.87%, AVG - 21.674%
mod-authtoken: MIN - 0.33%, MAX - 18.59%, AVG - 5.96%

TOP CPU usage modules:

nginx-okapi: MIN - 0.13%, MAX - 72.06%, AVG - 33.54%
mod-source-record-storage: MIN - 1.95%, MAX - 67.80%, AVG - 29.83%
mod-inventory: MIN - 6.07%, MAX - 59.05%, AVG - 27.44%

RDS CPU Utilization

MAX RDS CPU utilization is 49% under 80k DI Marc holding activity.

Appendix

Infrastructure

PTF -environment ncp3 [ environment name]

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters:

Modules	Version	Task Definition	Running Tasks	CPU	Memory (Soft/Hard Limits)	MaxMetaspaceSize	Xmx
mod-inventory	19.0.2	7	2	1024	2592/2880	512	1814
mod-inventory-storage	25.0.3	3	2	1024	1952/2208	512	1440
okapi	4.14.7	1	3	1024	1440/1684	512	922
mod-source-record-storage	5.5.2	4	2	1024	1440/1536	512	908
mod-source-record-manager	3.5.6	4	2	1024	3688/4096	512	2048
mod-data-import	2.6.2	4	1	256	1844/2048	512	1292
mod-data-import-cs	1.15.1	1	2	128	896/1024	128	768
mod-quick-marc	2.5.0	3	1	128	2176/2288	512	1664
mod-feesfines	18.1.1	3	2	128	896/1024	128	768
mod-patron-blocks	1.7.1	4	2	1024	896/1024	128	768
mod-pubsub	2.7.0	4	2	1024	1440/1536	512	922
mod-authtoken	2.12.0	3	2	512	1152/1440	128	922
mod-circulation-storage	15.0.2	3	2	1024	1440/1536	512	896
mod-circulation	23.3.2	3	2	1024	896/1024	128	768
mod-configuration	5.9.0	3	2	128	896/1024	128	768
mod-users	19.0.0	4	2	128	896/1024	128	768
mod-remote-storage	1.7.1	3	2	128	1692/1872	512	1178

Methodology/Approach

DB Refresh
Run a CI/CO circulation test by Jenkins job
Data Import MARC Holding job running with 5 minutes interval

Additional Screenshots of graphs or charts

During the test, there were detected several requests with more than a 1-second response time.

Data Import (MARC Holdings) with CI/CO [Nolana]