Overview

This document contains the results of testing Sharing local instances(SLI) for MARC Source records. PERF-904 - Getting issue details... STATUS

Summary

Duration on three tenants cs00000int_0001-cs00000int_0003 for 1 SLI process is about the same and the average value is about 16 seconds. Duration on the last tenant during test execution cs00000int_0004 is about 6 seconds. For 2 parallel SLIs duration is about 16 seconds on first cs00000int_0001 tenant and 6,5 seconds on second tenant cs00000int_0002. For 3 parallel SLIs and for 4 parallel SLIs is about 16 seconds for all except the last tenant - 6 seconds.
Duration increased significantly if to compare with approximate 2 seconds in Poppy release.
After adding parameter Dinventory.sharing.di.status.poll.interval.seconds=2 the duration decreased from 6 seconds to 2,9 seconds for SLI.
No memory leak is suspected for SLI modules. Memory consumption grew for mod-inventory to 56% during 3-rd test and didn't change.
Maximum CPU utilization was about 15% on mod-inventory and mod-consortia-b 1VU-23%, 2VU-25%, 3VU-28%, 4VU-35%.
RDS CPU utilization were from 8% for 1VU and up to 10% for 4VU. That is twice less than in Poppy.
All SLI finished successfully(without erros). Error rate = 0

Recommendations and Jiras

It's important to add parameters to task definition to reduce local instance sharing time
- ```
"name": "JAVA_OPTS",
"value": " -Dinventory.sharing.di.status.poll.interval.seconds=2"
```

Test results

Test 1. Virtual user working sequentially on each of the 4 tenants.

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15974	0.06	15560	15967	16369	16482
cs00000int_0002	100	16004	0.06	15594	16016	16351	16377
cs00000int_0003	100	15523	0.06	15281	15503	15760	15852
cs00000int_0004	100	5998	0.16	5714	5975	6285	6941

Test 2. Virtual users working parallel on 2 tenants.

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	ERRORS, COUNT	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15958	0.06	0	15571	15945	16311	16488
cs00000int_0002	100	6436	0.15	0	5982	6412	6802	8082

Test 3. Virtual users working parallel on 3 tenants

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15984	0.06	15554	15984	16269	16464
cs00000int_0002	100	15976	0.06	15613	15967	16323	16457
cs00000int_0003	100	5985	0.16	5655	5951	6308	7414

Test 4. Virtual users working parallel on 4 tenants

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15993	0.06	15630	15994	16302	16548
cs00000int_0002	100	15920	0.06	15554	15875	16315	16405
cs00000int_0003	100	15515	0.06	15281	15507	15729	15755
cs00000int_0004	100	5956	0.16	5656	5930	6288	7321

Results after mod-inventory module task definition update ("value": " -Dinventory.sharing.di.status.poll.interval.seconds=2")

Test 1. Virtual user working sequentially on each of the 4 tenants.

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15980	0.06	15613	15967	16332	16457
cs00000int_0002	100	15951	0.06	15583	15977	16321	16354
cs00000int_0003	100	15522	0.06	15263	15504	15755	15978
cs00000int_0004	100	2875	0.33	2603	2860	3130	3215

Test 2. Virtual users working parallel on 2 tenants.

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	ERRORS, COUNT	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15972	0.06	0	15592	15994	16296	16405
cs00000int_0002	100	3311	0.28	0	1035	3448	3739	4692

Test 3. Virtual users working parallel on 3 tenants

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15986	0.06	15643	15978	16305	16416
cs00000int_0002	100	15957	0.06	15573	15949	16264	16472
cs00000int_0003	100	2850	0.33	2579	2814	3095.2	4225

Test 4. Virtual users working parallel on 4 tenants

Tenant	TTL REQ, COUNT	Average, MS	THRGHPT, REQ/SEC	MIN, MS	MEDIAN, MS	PCT95, MS	MAX, MS
cs00000int_0001	100	15965	0.06	15580	15964	16346	16473
cs00000int_0002	100	15959	0.06	15617	15962	16286	16434
cs00000int_0003	100	15529	0.06	15263	15538	15704	15889
cs00000int_0004	100	2874	0.16	2601	2852	3101	4433

Resource Utilization Test 1,2,3,4.

Below are the resource utilization graphs for all tests.

Service CPU Utilization

CPU utilization increased only during the SLI process and all modules came back to default numbers after all SLI was finished.

mod-consortia-b

Test 1 - 25%, test 2 - 27%, test 3 - 30%, test 4 - 35%

CPU utilization table over the tests

CPU	Test 1	Test 2	Test 3	Test 4
mod-consortia-b	23.64	25.41	28.33	35.09
mod-inventory-b	13.95	14.7	13.71	13.9
mod-pubsub-b	6.62	6.65	6.54	6.62
mod-quick-marc-b	6	6.03	5.53	7.02
mod-di-converter-storage-b	2.94	4.77	4.78	5.17
mod-source-record-manager-b	2.4	2.68	2.85	3.01
mod-tags-b	2.25	2.23	2.28	2.27
mod-organizations-b	2.15	2.09	2.09	2.17
mod-search-b	1.6	1.69	1.81	1.85
mod-source-record-storage-b	1.57	1.63	1.64	1.72
okapi-b	1.53	2.47	3.19	3.59
nginx-okapi	1.08	2.34	3.15	3.81
mod-inventory-storage-b	0.46	0.85	1.02	1.19
mod-circulation-b	0.33	0.37	0.35	0.34

Memory Utilization

Memory usage was stable over 4 tests no memory leak is suspected for all modules, on the graph there are 10 most memory-consuming services. Mod-consortia - 74%, mod-inventory - 56%.

RDS CPU Utilization

For 1VU average RDS CPU Utilization was about 7% for all 4 tenants. 2VU ~ 8%; 3VU~9% and 4VU ~10%.

RDS Database Connections

The average number of DB connections before the test was about 500. During tests the connections spiked to 620.

Average active sessions (AAS)

Database load sliced by SQL

Errors

"error": "Sharing instance with InstanceId=[UUID] to the target tenant cs00000int. Error: Failed sending record data."

Fixed after deploy of mod-di-converter-storage:2.2.2 MODDICONV-379 - Getting issue details... STATUS

Appendix

Record parameters, on each of the Тenants


QCON	Instances	Source:	Records:
cs00000int	1,540,307	MARC	1436417
		FOLIO	103890

cs00000int_0001	3,147,248	MARC	3042890
		FOLIO	104358

cs00000int_0002	2,306,678	MARC	2180113
		FOLIO	126565

cs00000int_0003	1,540,307	MARC	1436417
		FOLIO	103890

cs00000int_0004	1,540,307	MARC	1436417
		FOLIO	103890

cs00000int_0005	1,540,307	MARC	1436417
		FOLIO	103890

PTF -environment: QCON

11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6.xlarge database instance: Writer instance
MSK - tenant
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
OpenSearch 2.7 domain: ptf-test
- Dedicated master nodes enabled
  - Instance type r6g.large.search
  - Number of nodes 3
- Data nodes
  - Instance type r6g.2xlarge.search
  - Number of nodes 4
Topics:
- { "name": "KAFKA_PRODUCER_TENANT_COLLECTION", "value": "ALL" }

Infrastructure

Share local instance modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcon-pvt
Fri May 10 18:49:16 UTC 2024
mod-consortia	1	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
mod-organizations	1	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-tags	1	mod-tags:2.2.0	2	1024	896	128	768	88	128
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-source-record-storage	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.2.2	2	1024	896	128	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
mod-quick-marc	1	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512

QCON all modules

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
qcon-pvt
Fri May 10 18:49:16 UTC 2024
mod-remote-storage	1	mod-remote-storage:3.2.0	2	4920	4472	1024	3960	512	512
mod-finance-storage	1	mod-finance-storage:8.6.0	2	1024	896	1024	700	88	128
mod-ncip	1	mod-ncip:1.14.4	2	1024	896	128	768	88	128
mod-agreements	1	mod-agreements:7.0.0	0	1592	1488	128	0	0	0
mod-ebsconet	1	mod-ebsconet:2.2.0	2	1248	1024	128	700	128	256
mod-consortia	1	mod-consortia:1.1.0	2	3072	2048	128	2048	512	1024
mod-organizations	1	mod-organizations:1.9.0	2	1024	896	128	700	88	128
mod-serials-management	1	mod-serials-management:1.0.0	0	2480	2312	128	1792	384	512
mod-settings	1	mod-settings:1.0.3	2	1024	896	200	768	88	128
mod-search	4	mod-search:3.2.0	2	5184	4960	2048	1440	512	1024
edge-dematic	1	edge-dematic:2.2.0	1	1024	896	128	768	88	128
mod-data-import	1	mod-data-import:3.1.0	1	2048	1844	256	1292	384	512
mod-tags	1	mod-tags:2.2.0	2	1024	896	128	768	88	128
mod-authtoken	3 non-expired token	mod-authtoken:2.15.1	2	1440	1152	512	922	88	128
edge-courses	1	edge-courses:1.4.0	2	1024	896	128	768	88	128
mod-inventory-update	1	mod-inventory-update:3.3.0	2	1024	896	128	768	88	128
mod-notify	1	mod-notify:3.2.0	2	1024	896	128	768	88	128
mod-configuration	1	mod-configuration:5.10.0	2	1024	896	128	768	88	128
mod-orders-storage	1	mod-orders-storage:13.7.0	2	1024	896	512	700	88	128
edge-caiasoft	1	edge-caiasoft:2.2.0	2	1024	896	128	768	88	128
mod-login-saml	1	mod-login-saml:2.8.0	2	1024	896	128	768	88	128
mod-erm-usage-harvester	1	mod-erm-usage-harvester:4.5.0	2	1024	896	128	768	88	128
mod-password-validator	1	mod-password-validator:3.2.0	2	1440	1298	128	768	384	512
mod-licenses	1	mod-licenses:6.0.0	2	2480	2312	128	1792	384	512
mod-gobi	1	mod-gobi:2.8.0	2	1024	896	128	700	88	128
mod-bulk-operations	1	mod-bulk-operations:2.0.0	2	3072	2600	1024	1536	384	512
mod-fqm-manager	1	mod-fqm-manager:2.0.1	2	3000	2600	128	2048	384	512
edge-dcb	1	edge-dcb:1.1.0	2	1024	896	128	768	88	128
mod-graphql	1	mod-graphql:1.12.1	2	1024	896	128	768	88	128
mod-finance	1	mod-finance:4.9.0	2	1024	896	128	700	88	128
mod-erm-usage	1	mod-erm-usage:4.7.0	2	1024	896	128	768	88	128
mod-batch-print	1	mod-batch-print:1.1.0	2	1024	896	128	768	88	128
mod-copycat	1	mod-copycat:1.6.0	2	1024	512	128	768	88	128
mod-lists	1	mod-lists:2.0.0	2	3000	2600	128	2048	384	512
mod-entities-links	1	mod-entities-links:3.0.0	2	2592	2480	400	1440	0	1024
mod-permissions	2	mod-permissions:6.5.0	2	1684	1544	512	1024	384	512
pub-edge	1	pub-edge:2023.06.14	2	1024	896	128	768	0	0
mod-orders	1	mod-orders:12.8.0	2	2048	1440	1024	1024	384	512
edge-patron	1	edge-patron:5.1.0	2	1024	896	256	768	88	128
edge-ncip	1	edge-ncip:1.9.2	2	1024	896	128	768	88	128
mod-users-bl	1	mod-users-bl:7.7.0	2	1440	1152	512	922	88	128
mod-invoice	1	mod-invoice:5.8.0	2	1440	1152	512	922	88	128
mod-inventory-storage	1	mod-inventory-storage:27.1.0	2	4096	3690	2048	3076	384	512
mod-user-import	1	mod-user-import:3.8.0	2	1024	896	128	768	88	128
mod-sender	1	mod-sender:1.12.0	2	1024	896	128	768	88	128
edge-oai-pmh	1	edge-oai-pmh:2.9.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	1	mod-data-export-worker:3.2.1	2	3072	2048	1024	2048	384	512
mod-rtac	1	mod-rtac:3.6.0	2	1024	896	128	768	88	128
mod-circulation-storage	1	mod-circulation-storage:17.2.0	2	2880	2592	1536	1814	384	512
mod-source-record-storage	1	mod-source-record-storage:5.8.0	2	5600	5000	2048	3500	384	512
mod-calendar	1	mod-calendar:3.1.0	2	1024	896	128	768	88	128
mod-event-config	1	mod-event-config:2.7.0	2	1024	896	128	768	88	128
mod-courses	1	mod-courses:1.4.10	2	1024	896	128	768	88	128
mod-circulation-item	1	mod-circulation-item:1.0.0	2	1024	896	128	0	0	0
mod-inventory	1	mod-inventory:20.2.0	2	2880	2592	1024	1814	384	512
mod-email	1	mod-email:1.17.0	2	1024	896	128	768	88	128
mod-circulation	2	mod-circulation:24.2.1	2	2880	2592	1536	1814	384	512
mod-pubsub	1	mod-pubsub:2.13.0	2	1536	1440	1024	922	384	512
mod-di-converter-storage	2	mod-di-converter-storage:2.2.2	2	1024	896	128	768	88	128
edge-orders	1	edge-orders:3.0.0	2	1024	896	128	768	88	128
edge-rtac	1	edge-rtac:2.7.1	2	1024	896	128	768	88	128
mod-users	1	mod-users:19.3.1	2	1024	896	128	768	88	128
mod-template-engine	1	mod-template-engine:1.20.0	2	1024	896	128	768	88	128
mod-patron-blocks	1	mod-patron-blocks:1.10.0	2	1024	896	1024	768	88	128
mod-audit	1	mod-audit:2.9.0	2	1024	896	128	768	88	128
edge-fqm	1	edge-fqm:2.0.0	2	1024	896	128	768	88	128
mod-source-record-manager	1	mod-source-record-manager:3.8.0	2	5600	5000	2048	3500	384	512
nginx-edge	1	nginx-edge:2023.06.14	2	1024	896	128	0	0	0
mod-quick-marc	1	mod-quick-marc:5.1.0	1	2288	2176	128	1664	384	512
nginx-okapi	1	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	1	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	1	mod-feesfines:19.1.0	2	1024	896	128	768	88	128
mod-invoice-storage	1	mod-invoice-storage:5.8.0	2	1872	1536	1024	1024	384	512
mod-service-interaction	1	mod-service-interaction:4.0.1	2	2048	1844	256	1290	384	512
mod-dcb	1	mod-dcb:1.1.0	2	1024	896	128	768	88	128
mod-patron	1	mod-patron:6.1.0	2	1024	896	128	768	88	128
mod-data-export	1	mod-data-export:5.0.0	1	2048	1524	1024	0	0	0
mod-oai-pmh	1	mod-oai-pmh:3.13.0	2	4096	3690	2048	3076	384	512
edge-connexion	1	edge-connexion:1.2.0	2	1024	896	128	768	88	128
mod-notes	1	mod-notes:5.2.0	2	1024	896	128	952	384	512
mod-kb-ebsco-java	1	mod-kb-ebsco-java:4.0.0	2	1024	896	128	768	88	128
mod-login	1	mod-login:7.11.0	2	1440	1298	1024	768	384	512
mod-organizations-storage	1	mod-organizations-storage:4.7.0	2	1024	896	128	700	88	128
mod-data-export-spring	1	mod-data-export-spring:3.2.0	1	2048	1844	256	1536	384	512
pub-okapi	1	pub-okapi:2023.06.14	2	1024	896	128	768	0	0
mod-eusage-reports	1	mod-eusage-reports:2.1.1	2	1024	896	128	768	88	128

Additional Files

source files for SLI CPU and Memory dashboard creation

Methodology

jMeter request to trigger instance sharing:

CSI_POST /consortia/{consortiumId}/sharing/instances

BODY: {"sourceTenantId":"${sourceTenantId}","instanceIdentifier":"${list_id1}","targetTenantId":"${targetTenantId}","id":"${__UUID()}"}

Test 1. The shared local instances process was started from the Jmeter script for Tenant cs00000int_0001, cs00000int_0002, cs00000int_0003, and cs00000int_0004 one after another.

Test 2. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001 and cs00000int_0002.

Test 3. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001, cs00000int_0002, and cs00000int_0003.

Test 4. The shared local instances process was started from the Jmeter script in parallel for Tenant cs00000int_0001, cs00000int_0002, cs00000int_0003, and cs00000int_0004.

Folio Development Teams

Member tenants sharing local instances (Quesnelia)