Overview

This document contains the results of the testing Data Import (batch imports) on 62 tenants with the following scenarios on the MCPT cluster PERF-778: MOBIUS baseline tests: Data ImportClosed :

Summary

1. The average duration of single record imports on 61 member tenants (50% member tenants create, 50% member tenants update) during :

• DI_T1(CT-2k_MT-SRI-50/50). was 1,2 second for Create profile and 1 second for Update;

• DI_T2(CT-10k_MT-SRI-50/50). was 1,5 second for Create profile and 1,2 second for Update;

2. All Data-import finished with the status “Completed“

3. Most CPU consuming services were mod-consortia-b , mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b, mod-source-record-storage-b, mod-inventory-storage-b, okapi-b, mod-source-record-manager-b their values are in Service CPU Utilization section

4. During 2 tests no memory leaks.

5. DB CPU usage during DI_T1(CT-2k_MT-SRI-50/50) has increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) increased up to 95%.

Recommendations & Jiras

Add more CPU for 4 first services that have high CPU(>100%) utilization mod-consortia-b, mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b

Test Runs

1. DI_T1(CT-2k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 2K MARC BIB records; 30% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.

2. DI_T2(CT-10k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 10K MARC BIB records; 100% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.

Test Results

DI_T1(CT-2k_MT-SRI-50/50). The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles. More information on the duration of the DI on each tenant can be found in the section Appendix.

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, Data-import took less time for the first few tenants. This is probably because they were the first ones to start, followed by the others.

Open

DI_T2(CT-10k_MT-SRI-50/50). The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, DI took less time for the first few tenants. This is probably because they were the first ones to start the data import process, followed by the others, like on the previous tests.

Open

Resource utilization

This section presents a comprehensive collection of key graphs and metrics for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.

Service CPU Utilization

Memory Utilization

During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns

DB CPU Utilization

CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.

DB Connections

The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.

DB Load

The graph on Database Load provides an illustration of the resource utilization database over a period of time when 2 tests were running. The key SQL queries that were sent are located in the right section of the figure

SQL queries

The SQL Queries section provides detailed information about the SQL queries executed on mcpt database. It includes metrics like the number of times each query was executed, the average time taken to execute each query, and the utilization of resources by each query.

Methodology/Approach

All tests were run using the JMeter script for single record import, large data import on the central tenant was started manually.

Appendix

DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Infrastructure

PTF -environment mcpt

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance, writer

  Name Memory GIBv CPUsmax_connections

  db.r6g.4xlarge 128 GiB; 16 vCPU

• MSK “ptf-mobius-testing2“

  • 2 m5.2xlarge brokers in 1 zone

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

Task definitions of related modules were changed to use shared Kafka topics.

Kafka topics were created with the suffix “mcpt.Default.ALL.” and used 2 partitions for involved topics topics,

The table contains modules, memory, and CPU parameters