...

Central Tenant: DI_Importing MARC records workflow Transaction

...

DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for The table below contains metrics for single record Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Instance” and ”Inventory Single Record - Default Create Instance“ profile).

...

Tenant

...

Duration

...

Central Tenant: DI_Import MARC BIB create 2K records

...

1 minute

...

SDIU_(Update) cs00000001_0022

...

12 sec

...

SDIU_(Update) cs00000001_0016

...

13 sec

...

SDIU_(Update) cs00000001_0013

...

13,9 sec

...

SDIU_(Update) cs00000001_0010

...

13,3 sec

...

SDIU_(Update) cs00000001_0008

...

11 sec

...

SDIU_(Update) cs00000001_0006

...

11 sec

...

SDIU_(Update) cs00000001_0005

...

11 sec

...

SDIU_(Update) cs00000001_0002

...

11,5 sec

...

SDIC_(Create) cs00000001_0021

...

12 sec

...

SDIC_(Create) cs00000001_0020

...

12 sec

...

SDIC_(Create) cs00000001_0019

...

11 sec

...

SDIC_(Create) cs00000001_0018

...

12 sec

...

SDIC_(Create) cs00000001_0017

...

12 sec

...

SDIC_(Create) cs00000001_0015

...

12 sec

...

SDIC_(Create) cs00000001_0014

...

12,7 sec

...

SDIC_(Create) cs00000001_0012

...

12 sec

...

SDIC_(Create) cs00000001_0011

...

12 sec

...

SDIC_(Create) cs00000001_0009

...

10 sec

...

SDIC_(Create) cs00000001_0007

...

5 sec

...

SDIC_(Create) cs00000001_0004

...

5,3 sec

...

SDIC_(Create) cs00000001_0003

...

5,8 sec

The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

...

Metrics

...

Inventory Single Record - Default Update Instance

...

Inventory Single Record - Default Create Instance

...

Average

...

1,2 second

...

1,0 second

...

Max

...

1,4 second

...

1,4 second

...

Min

...

1 second

...

500 ms

...

75th percentile

...

1,3 second

...

1,2 second

...

90th percentile

...

1,35 second

...

1, 3 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”

...

DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

...

Tenant

...

Duration

Instance” profiles. More information on the duration of the DI on each tenant can be found in the section Appendix.

Metrics	Inventory Single Record - Default Update Instance	Inventory Single Record - Default Create Instance
Average	1,2 second	1,0 second
Max	1,4 second	1,4 second
Min	1 second	500 ms
75th percentile	1,3 second	1,2 second
90th percentile	1,35 second	1, 3 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, Data-import took less time for the first few tenants. This is probably because they were the first ones to start, followed by the others.

...

DI_T2(CT-10k_MT-SRI-50/50). The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

Metrics	Inventory Single Record - Default Update Instance	Inventory Single Record - Default Create Instance
Average	1,2 second	1,5 second
Max	1,65 second	1,8 second
Min	4,1 second	1 second
75th percentile	1,6 second	1,67 second
90th percentile	1,6 second	1,71 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, DI took less time for the first few tenants. This is probably because they were the first ones to start the data import process, followed by the others, like on the previous tests.

...

Resource utilization

This section presents a comprehensive collection of key graphs and metrics for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.

Service CPU Utilization

Service	DI_T1(CT-2k_MT-SRI-50/50)	DI_T2(CT-10k_MT-SRI-50/50)
mod-consortia-b	179%	189%
mod-quick-marc-b	152%	80%
mod-inventory-b	99%	73%
mod-di-converter-storage-b	95%	105%
mod-source-record-storage-b	49%	48%
mod-inventory-storage-b	36%	32%
okapi-b	28%	37%
mod-source-record-manager-b	26%	29%

...

Memory Utilization

During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns

Service	MU before the T1, T2	MU after the T1, T2
mod-data-import-b	36	50
mod-search-b	24	31
mod-inventory-b	15	28
mod-source-record-manager-b	14	22
mod-inventory-storage-b	13	23
mod-source-record-storage-b	12	16

...

DB CPU Utilization

CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.

...

DB Connections

The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.

...

DB Load

The graph on Database Load provides an illustration of the resource utilization database over a period of time when 2 tests were running. The key SQL queries that were sent are located in the right section of the figure

...

SQL queries

The SQL Queries section provides detailed information about the SQL queries executed on mcpt database. It includes metrics like the number of times each query was executed, the average time taken to execute each query, and the utilization of resources by each query.

...

Methodology/Approach

All tests were run using the JMeter script for single record import, large data import on the central tenant was started manually.

Appendix

DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Tenant	Duration
Central Tenant: DI_Import MARC BIB create 2K records	1 minute
SDIU_(Update) cs00000001_0022	12 sec
SDIU_(Update) cs00000001_0016	13 sec
SDIU_(Update) cs00000001_0013	13,9 sec
SDIU_(Update) cs00000001_0010	13,3 sec
SDIU_(Update) cs00000001_0008	11 sec
SDIU_(Update) cs00000001_0006	11 sec
SDIU_(Update) cs00000001_0005	11 sec
SDIU_(Update) cs00000001_0002	11,5 sec
SDIC_(Create) cs00000001_0021	12 sec
SDIC_(Create) cs00000001_0020	12 sec
SDIC_(Create) cs00000001_0019	11 sec
SDIC_(Create) cs00000001_0018	12 sec
SDIC_(Create) cs00000001_0017	12 sec
SDIC_(Create) cs00000001_0015	12 sec
SDIC_(Create) cs00000001_0014	12,7 sec
SDIC_(Create) cs00000001_0012	12 sec
SDIC_(Create) cs00000001_0011	12 sec
SDIC_(Create) cs00000001_0009	10 sec
SDIC_(Create) cs00000001_0007	5 sec
SDIC_(Create) cs00000001_0004	5,3 sec
SDIC_(Create) cs00000001_0003	5,8 sec

DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Create0018SDICCreate002112 SDICCreate0022SDICCreate002316 SDICCreate002416 SDICCreate0025165 SDICCreate002616SDICCreate0030156 SDICCreate0031157 SDICCreate0032157 SDICCreate0033151 SDICCreate0034155 SDICCreate0036148 SDICCreate0039156 SDICCreate004115,8 SDICCreate004715,1 SDICCreate0048149 SDICCreate004915,4 SDICCreate0052158 SDICCreate0055153 SDICCreate0061157 SDICCreate0062158

Tenant	Duration
Central Tenant: DI_Importing MARC records workflow Transaction	5 minutes
SDIC_Single Record Import (Create) cs00000001_0002	5 sec
SDIC_Single Record Import (Create) cs00000001_0003	5 sec
SDIC_Single Record Import (Create) cs00000001_0006	5,6 sec
SDIC_Single Record Import (Create) cs00000001_0007	4,2 sec
SDIC_Single Record Import (Create) cs00000001_0008	4,1 sec
SDIC_Single Record Import (Create) cs00000001_0010	10 sec
SDIC_Single Record Import (Create) cs00000001_0012	6 sec
SDIC_Single Record Import (Create) cs00000001_0015	10 sec
SDIC_Single Record Import (Create) cs00000001_0017	11,3 sec
SDIC_Single Record Import (Create) cs00000001_0018	11,6 sec
SDIC_Single Record Import (Create) cs00000001_0021	12 sec
SDIC_Single Record Import (Create) cs00000001_0022	11 sec
SDIC_Single Record Import (Create) cs00000001_0023	16 sec
SDIC_Single Record Import (Create) cs00000001_0024	16 sec
SDIC_Single Record Import (Create) cs00000001_0025	16,5 sec
SDIC_Single Record Import (Create) cs00000001_0026	16,1 sec
SDIC_Single Record Import (Create) cs00000001_0030	15,6 sec
SDIC_Single Record Import (Create) cs00000001_0031	15,7 sec
SDIC_Single Record Import (Create) cs00000001_0032	15,7 sec
SDIC_Single Record Import (Create) cs00000001_0033	15,1 sec
SDIC_Single Record Import (Create) cs00000001_0034	15,5 sec
SDIC_Single Record Import (Create) cs00000001_0036	14,8 sec
SDIC_Single Record Import (Create) cs00000001_00020039	5 15,6 sec
SDIC_Single Record Import (Create) cs00000001_00030041	5 15,8 sec
SDIC_Single Record Import (Create) cs00000001_00060047	515,6 1 sec
SDIC_Single Record Import (Create) cs00000001_00070048	414,2 9 sec
SDIC_Single Record Import (Create) cs00000001_00080049	15,4 ,1 sec
SDIC_Single Record Import (Create) cs00000001_00100052	10 15,8 sec
SDIC_Single Record Import (Create) cs00000001_00120055	6 15,3 sec
SDIC_Single Record Import (Create) cs00000001_00150061	10 15,7 sec
SDIC_Single Record Import (Create) cs00000001_00170062	1115,3 8 secSDIC
SDIU_Single Record Import (	Update) cs00000001_	0004	11,6 sec
SDIU_Single Record Import (	Update) cs00000001_	0005	10,3 sec
SDIU_Single Record Import (	Update) cs00000001_	0009	11,7 sec
SDIU_Single Record Import (	Update) cs00000001_	0011	11,8 sec
SDIU_Single Record Import (	Update) cs00000001_	0013	11,4 sec
SDIU_Single Record Import (	Update) cs00000001_	0014	11,	6 sec
SDIU_Single Record Import (	Update) cs00000001_	0016	12,1 sec
SDIU_Single Record Import (	Update) cs00000001_	0019	11,	3 sec
SDIU_Single Record Import (	Update) cs00000001_	0020	12,	1 sec
SDIU_Single Record Import (	Update) cs00000001_	0027	16,	1 sec
SDIU_Single Record Import (	Update) cs00000001_	0028	16,	9 sec
SDIU_Single Record Import (	Update) cs00000001_	0029	17,	1 sec
SDIU_Single Record Import (	Update) cs00000001_	0035	16,	9 sec
SDIU_Single Record Import (	Update) cs00000001_	0037	16,	5 sec
SDIU_Single Record Import (	Update) cs00000001_	0038	16 sec
SDIU_Single Record Import (	Update) cs00000001_	0040	18 sec
SDIU_Single Record Import (	Update) cs00000001_	0042	16,	8 sec
SDIU_Single Record Import (	Update) cs00000001_	0043	17 sec
SDIU_Single Record Import (	Update) cs00000001_	0044	17,	1 sec
SDIU_Single Record Import (	Update) cs00000001_	0045	16,	5 sec
SDIU_Single Record Import (	Update) cs00000001_	0046	17,	1 sec
SDIU_Single Record Import (	Update) cs00000001_	0050	16,	5 sec
SDIU_Single Record Import (Update) cs00000001_00040051	1116,6 5 sec
SDIU_Single Record Import (Update) cs00000001_00050053	10,3 16 sec
SDIU_Single Record Import (Update) cs00000001_00090054	1116,7 3 sec
SDIU_Single Record Import (Update) cs00000001_00110056	1115,8 5 sec
SDIU_Single Record Import (Update) cs00000001_00130057	1116,4 3 sec
SDIU_Single Record Import (Update) cs00000001_00140058	1116,6 sec
SDIU_Single Record Import (Update) cs00000001_00160059	1216,1 2 sec
SDIU_Single Record Import (Update) cs00000001_00190060	1115,3 sec
SDIU_Single Record Import (Update) cs00000001_0020	12,1 sec
SDIU_Single Record Import (Update) cs00000001_0027	16,1 sec
SDIU_Single Record Import (Update) cs00000001_0028	16,9 sec
SDIU_Single Record Import (Update) cs00000001_0029	17,1 sec
SDIU_Single Record Import (Update) cs00000001_0035	16,9 sec
SDIU_Single Record Import (Update) cs00000001_0037	16,5 sec
SDIU_Single Record Import (Update) cs00000001_0038	16 sec
SDIU_Single Record Import (Update) cs00000001_0040	18 sec
SDIU_Single Record Import (Update) cs00000001_0042	16,8 sec
SDIU_Single Record Import (Update) cs00000001_0043	17 sec
SDIU_Single Record Import (Update) cs00000001_0044	17,1 sec
SDIU_Single Record Import (Update) cs00000001_0045	16,5 sec
SDIU_Single Record Import (Update) cs00000001_0046	17,1 sec
SDIU_Single Record Import (Update) cs00000001_0050	16,5 sec
SDIU_Single Record Import (Update) cs00000001_0051	16,5 sec
SDIU_Single Record Import (Update) cs00000001_0053	16 sec
SDIU_Single Record Import (Update) cs00000001_0054	16,3 sec
SDIU_Single Record Import (Update) cs00000001_0056	15,5 sec
SDIU_Single Record Import (Update) cs00000001_0057	16,3 sec
SDIU_Single Record Import (Update) cs00000001_0058	16,6 sec
SDIU_Single Record Import (Update) cs00000001_0059	16,2 sec
SDIU_Single Record Import (Update) cs00000001_0060	15,9 sec

The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

...

Metrics

...

Inventory Single Record - Default Update Instance

...

Inventory Single Record - Default Create Instance

...

Average

...

1,2 second

...

1,5 second

...

Max

...

1,65 second

...

1,8 second

...

Min

...

4,1 second

...

1 second

...

75th percentile

...

1,6 second

...

1,67 second

...

90th percentile

...

1,6 second

...

1,71 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”

...

Resource utilization

This section presents a comprehensive collection of key graphs and metrics necessary for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.

Service CPU Utilization

Service	DI_T1(CT-2k_MT-SRI-50/50)	DI_T2(CT-10k_MT-SRI-50/50)
mod-consortia-b	179%	189%
mod-quick-marc-b	152%	80%
mod-inventory-b	99%	73%
mod-di-converter-storage-b	95%	105%
mod-source-record-storage-b	49%	48%
mod-inventory-storage-b	36%	32%
okapi-b	28%	37%
mod-source-record-manager-b	26%	29%

...

Memory Utilization

During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns

Service	MU before the T1, T2	MU after the T1, T2
mod-data-import-b	36	50
mod-search-b	24	31
mod-inventory-b	15	28
mod-source-record-manager-b	14	22
mod-inventory-storage-b	13	23
mod-source-record-storage-b	12	16

...

DB CPU Utilization

CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.

...

DB Connections

The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.

...

DB Load

Image Removed

SQL queries

...

Methodology/Approach

All tests were run using the JMeter script.

Infrastructure

PTF -environment mcpt

...

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

...

1 database instance, writer

Name Memory GIBv CPUsmax_connections

db.r6g.4xlarge 128 GiB; 16 vCPU

MSK tenant

...

2 m5.2xlarge brokers in 1 zones

...

Apache Kafka version 2.8.0

...

9 sec

Infrastructure

PTF -environment mcpt

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIBv CPUsmax_connections
db.r6g.4xlarge 128 GiB; 16 vCPU

Expand

title	Number of instances and holdings on each of the tenant

Tenant	Number of instance records	Number of holdings records
cs00000001_0001	65359	66698
cs00000001_0002	22370	22478
cs00000001_0003	136291	136680
cs00000001_0004	77785	78348
cs00000001_0005	110093	110670
cs00000001_0006	293580	354866
cs00000001_0007	279493	281880
cs00000001_0008	160024	183456
cs00000001_0009	110918	113452
cs00000001_0010	83254	85931
cs00000001_0011	94171	96351
cs00000001_0012	227489	234351
cs00000001_0013	19131	20386
cs00000001_0014	20746	20972
cs00000001_0015	26899	32560
cs00000001_0016	153333	153430
cs00000001_0017	16492	17053
cs00000001_0018	22263	22782
cs00000001_0019	5312	5960
cs00000001_0020	44259	45941
cs00000001_0021	83988	90207
cs00000001_0022	86900	87359
cs00000001_0023	91517	92098
cs00000001_0024	385123	393085
cs00000001_0025	84215	85690
cs00000001_0026	153912	155148
cs00000001_0027	48176	49393
cs00000001_0028	64556	66641
cs00000001_0029	5140	7959
cs00000001_0030	97589	102043
cs00000001_0031	71895	76244
cs00000001_0032	80074	86818
cs00000001_0033	45886	47652
cs00000001_0034	18240	19995
cs00000001_0035	135704	136564
cs00000001_0036	114170	115913
cs00000001_0037	24226	25970
cs00000001_0038	225563	244421
cs00000001_0039	128523	130263
cs00000001_0040	111992	116736
cs00000001_0041	61091	61473
cs00000001_0042	634675	650198
cs00000001_0043	2831	831
cs00000001_0044	70871	71478
cs00000001_0045	78320	92824
cs00000001_0046	109677	126252
cs00000001_0047	48531	49519
cs00000001_0048	53346	53597
cs00000001_0049	6952	7459
cs00000001_0050	190736	208837
cs00000001_0051	351325	363364
cs00000001_0052	137940	148890
cs00000001_0053	61834	62324
cs00000001_0054	16297	17156
cs00000001_0055	176009	182364
cs00000001_0056	160976	167864
cs00000001_0057	22596	23744
cs00000001_0058	123819	124396
cs00000001_0059	37715	38686
cs00000001_0060	38395	38776
cs00000001_0061	87117	88458
cs00000001_0062	1	1

MSK “ptf-mobius-testing2“
- 2 m5.2xlarge brokers in 1 zone
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB

Task definitions of related modules were changed to use shared Kafka topics.

Expand

title	Additional kafka configs

Code Block
{ "name": "KAFKA_PRODUCER_TENANT_COLLECTION", "value": "ALL" },

Kafka topics were created with the suffix “mcpt.Default.ALL.” and used 2 partitions for involved topics topics,

The table contains modules, memory, and CPU parameters

...

Version	Old Version 12	New Version Current
Changes made by	Mykhailo Petryshyn	Mykhailo Petryshyn
Saved on	Mar 14, 2024	Mar 14, 2024

Versions Compared

Key

Resource utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Methodology/Approach

Appendix

Resource utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Methodology/Approach

Infrastructure

Infrastructure

Page Comparison

Versions Compared

Key

Resource utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Methodology/Approach

Appendix

Resource utilization

Service CPU Utilization

Memory Utilization

DB CPU Utilization

DB Connections

DB Load

SQL queries

Methodology/Approach

Infrastructure

Infrastructure