Skip to end of banner
Go to start of banner

Data Import (batch imports) on 62 tenants-MCPT

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

Overview

This document contains the results of the testing Data Import (batch imports) on 62 tenants with the following scenarios on the MCPT cluster PERF-778 - Getting issue details... STATUS :

Summary

  1. The average duration of single record imports on 61 member tenants (50% member tenants create, 50% member tenants update) during :

  • DI_T1(CT-2k_MT-SRI-50/50). was 1,2 second for Create profile and 1 second for Update;

  • DI_T2(CT-10k_MT-SRI-50/50). was 1,5 second for Create profile and 1,2 second for Update;

  1. All Data-import finished with the status “Completed“

  2. Most CPU consuming services were mod-consortia-b , mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b, mod-source-record-storage-b, mod-inventory-storage-b, okapi-b, mod-source-record-manager-b their values are in Service CPU Utilization section

  3. During 2 tests no memory leaks.

  4. DB CPU usage during DI_T1(CT-2k_MT-SRI-50/50) has increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) increased up to 95%.

Recommendations & Jiras

Add more CPU for 4 first services that have high CPU(>100%) utilization mod-consortia-b, mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b

Test Runs

  1. DI_T1(CT-2k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 2K MARC BIB records; 30% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.

  2. DI_T2(CT-10k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 10K MARC BIB records; 100% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.

Test Results

DI_T1(CT-2k_MT-SRI-50/50). The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles. More information on the duration of the DI on each tenant can be found in the section Appendix.

Metrics

Inventory Single Record - Default Update Instance

Inventory Single Record - Default Create Instance

Average

1,2 second

1,0 second

Max 

1,4 second

1,4 second

Min

1 second

500 ms

75th percentile

1,3 second

1,2 second

90th percentile

1,35 second

1, 3 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, Data-import took less time for the first few tenants. This is probably because they were the first ones to start, followed by the others.

image-20240229-110340.png

DI_T2(CT-10k_MT-SRI-50/50). The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

Metrics

Inventory Single Record - Default Update Instance

Inventory Single Record - Default Create Instance

Average

1,2 second

1,5 second

Max 

1,65 second

1,8 second

Min

4,1 second

1 second

75th percentile

1,6 second

1,67 second

90th percentile

1,6 second

1,71 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”. As we can see from the graph, DI took less time for the first few tenants. This is probably because they were the first ones to start the data import process, followed by the others, like on the previous tests.

image-20240229-112256.png

Resource utilization

This section presents a comprehensive collection of key graphs and metrics for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.

Service CPU Utilization

Service 

DI_T1(CT-2k_MT-SRI-50/50)

DI_T2(CT-10k_MT-SRI-50/50)

mod-consortia-b

179%

189%

mod-quick-marc-b

152%

80%

mod-inventory-b

99%

73%

mod-di-converter-storage-b

95%

105%

mod-source-record-storage-b

49%

48%

mod-inventory-storage-b

36%

32%

okapi-b

28%

37%

mod-source-record-manager-b

26%

29%

image-20240229-113621.png

Memory Utilization

During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns

Service 

MU before the T1, T2

MU after the T1, T2

mod-data-import-b

36

50

mod-search-b

24

31

mod-inventory-b

15

28

mod-source-record-manager-b

14

22

mod-inventory-storage-b

13

23

mod-source-record-storage-b

12

16

image-20240229-114546.png

DB CPU Utilization

CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.

image-20240229-115251.png

DB Connections

The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.

image-20240229-123308.png

DB Load

The graph on Database Load provides an illustration of the resource utilization database over a period of time when 2 tests were running. The key SQL queries that were sent are located in the right section of the figure

image-20240229-123837.png

SQL queries

The SQL Queries section provides detailed information about the SQL queries executed on mcpt database. It includes metrics like the number of times each query was executed, the average time taken to execute each query, and the utilization of resources by each query.

image-20240229-124048.png

Methodology/Approach

All tests were run using the JMeter script for single record import, large data import on the central tenant was started manually.

Appendix

DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Tenant

Duration

Central Tenant: DI_Import MARC BIB create 2K records

1 minute

SDIU_(Update) cs00000001_0022

12 sec

SDIU_(Update) cs00000001_0016

13 sec

SDIU_(Update) cs00000001_0013

13,9 sec

SDIU_(Update) cs00000001_0010

13,3 sec

SDIU_(Update) cs00000001_0008

11 sec

SDIU_(Update) cs00000001_0006

11 sec

SDIU_(Update) cs00000001_0005

11 sec

SDIU_(Update) cs00000001_0002

11,5 sec

SDIC_(Create) cs00000001_0021

12 sec

SDIC_(Create) cs00000001_0020

12 sec

SDIC_(Create) cs00000001_0019

11 sec

SDIC_(Create) cs00000001_0018

12 sec

SDIC_(Create) cs00000001_0017

12 sec

SDIC_(Create) cs00000001_0015

12 sec

SDIC_(Create) cs00000001_0014

12,7 sec

SDIC_(Create) cs00000001_0012

12 sec

SDIC_(Create) cs00000001_0011

12 sec

SDIC_(Create) cs00000001_0009

10 sec

SDIC_(Create) cs00000001_0007

5 sec

SDIC_(Create) cs00000001_0004

5,3 sec

SDIC_(Create) cs00000001_0003

5,8 sec

DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Tenant

Duration

Central Tenant: DI_Importing MARC records workflow Transaction

5 minutes

SDIC_Single Record Import (Create) cs00000001_0002

5 sec

SDIC_Single Record Import (Create) cs00000001_0003

5 sec

SDIC_Single Record Import (Create) cs00000001_0006

5,6 sec

SDIC_Single Record Import (Create) cs00000001_0007

4,2 sec

SDIC_Single Record Import (Create) cs00000001_0008

4,1 sec

SDIC_Single Record Import (Create) cs00000001_0010

10 sec

SDIC_Single Record Import (Create) cs00000001_0012

6 sec

SDIC_Single Record Import (Create) cs00000001_0015

10 sec

SDIC_Single Record Import (Create) cs00000001_0017

11,3 sec

SDIC_Single Record Import (Create) cs00000001_0018

11,6 sec

SDIC_Single Record Import (Create) cs00000001_0021

12 sec

SDIC_Single Record Import (Create) cs00000001_0022

11 sec

SDIC_Single Record Import (Create) cs00000001_0023

16 sec

SDIC_Single Record Import (Create) cs00000001_0024

16 sec

SDIC_Single Record Import (Create) cs00000001_0025

16,5 sec

SDIC_Single Record Import (Create) cs00000001_0026

16,1 sec

SDIC_Single Record Import (Create) cs00000001_0030

15,6 sec

SDIC_Single Record Import (Create) cs00000001_0031

15,7 sec

SDIC_Single Record Import (Create) cs00000001_0032

15,7 sec

SDIC_Single Record Import (Create) cs00000001_0033

15,1 sec

SDIC_Single Record Import (Create) cs00000001_0034

15,5 sec

SDIC_Single Record Import (Create) cs00000001_0036

14,8 sec

SDIC_Single Record Import (Create) cs00000001_0039

15,6 sec

SDIC_Single Record Import (Create) cs00000001_0041

15,8 sec

SDIC_Single Record Import (Create) cs00000001_0047

15,1 sec

SDIC_Single Record Import (Create) cs00000001_0048

14,9 sec

SDIC_Single Record Import (Create) cs00000001_0049

15,4 sec

SDIC_Single Record Import (Create) cs00000001_0052

15,8 sec

SDIC_Single Record Import (Create) cs00000001_0055

15,3 sec

SDIC_Single Record Import (Create) cs00000001_0061

15,7 sec

SDIC_Single Record Import (Create) cs00000001_0062

15,8 sec

SDIU_Single Record Import (Update) cs00000001_0004

11,6 sec

SDIU_Single Record Import (Update) cs00000001_0005

10,3 sec

SDIU_Single Record Import (Update) cs00000001_0009

11,7 sec

SDIU_Single Record Import (Update) cs00000001_0011

11,8 sec

SDIU_Single Record Import (Update) cs00000001_0013

11,4 sec

SDIU_Single Record Import (Update) cs00000001_0014

11,6 sec

SDIU_Single Record Import (Update) cs00000001_0016

12,1 sec

SDIU_Single Record Import (Update) cs00000001_0019

11,3 sec

SDIU_Single Record Import (Update) cs00000001_0020

12,1 sec

SDIU_Single Record Import (Update) cs00000001_0027

16,1 sec

SDIU_Single Record Import (Update) cs00000001_0028

16,9 sec

SDIU_Single Record Import (Update) cs00000001_0029

17,1 sec

SDIU_Single Record Import (Update) cs00000001_0035

16,9 sec

SDIU_Single Record Import (Update) cs00000001_0037

16,5 sec

SDIU_Single Record Import (Update) cs00000001_0038

16 sec

SDIU_Single Record Import (Update) cs00000001_0040

18 sec

SDIU_Single Record Import (Update) cs00000001_0042

16,8 sec

SDIU_Single Record Import (Update) cs00000001_0043

17 sec

SDIU_Single Record Import (Update) cs00000001_0044

17,1 sec

SDIU_Single Record Import (Update) cs00000001_0045

16,5 sec

SDIU_Single Record Import (Update) cs00000001_0046

17,1 sec

SDIU_Single Record Import (Update) cs00000001_0050

16,5 sec

SDIU_Single Record Import (Update) cs00000001_0051

16,5 sec

SDIU_Single Record Import (Update) cs00000001_0053

16 sec

SDIU_Single Record Import (Update) cs00000001_0054

16,3 sec

SDIU_Single Record Import (Update) cs00000001_0056

15,5 sec

SDIU_Single Record Import (Update) cs00000001_0057

16,3 sec

SDIU_Single Record Import (Update) cs00000001_0058

16,6 sec

SDIU_Single Record Import (Update) cs00000001_0059

16,2 sec

SDIU_Single Record Import (Update) cs00000001_0060

15,9 sec

Infrastructure

PTF -environment mcpt

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database instance, writer

    Name Memory GIBv CPUsmax_connections

    db.r6g.4xlarge 128 GiB; 16 vCPU

  • MSK tenant

    • 2 m5.2xlarge brokers in 1 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

The table contains modules, memory, and CPU parameters

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mcpt-pvt 

mod-consortia

11

mod-consortia:1.0.3

2

2048

1802

128

768

88

128

FALSE

mod-organizations

6

mod-organizations:1.8.0

2

1024

896

128

700

88

128

FALSE

mod-data-import

7

mod-data-import:3.0.7

1

2048

1844

256

1292

384

512

FALSE

mod-search

12

mod-search:3.0.5

2

2592

2480

2048

1440

512

1024

FALSE

mod-configuration

6

mod-configuration:5.9.2

2

1024

896

128

768

88

128

FALSE

mod-permissions

14

mod-permissions:6.4.0

2

1684

1544

512

1024

384

512

FALSE

mod-entities-links

10

mod-entities-links:2.0.4

2

2592

2480

400

1440

0

1024

FALSE

mod-inventory-storage

10

mod-inventory-storage:27.0.4

2

4096

3690

2048

3076

384

512

FALSE

mod-data-export-worker

9

mod-data-export-worker:3.1.2

2

3072

2800

1024

2048

384

512

FALSE

mod-source-record-storage

10

mod-source-record-storage:5.7.5

2

5600

5000

2048

3500

384

512

FALSE

mod-inventory

10

mod-inventory:20.1.6

2

2880

2592

1024

1814

384

512

FALSE

mod-di-converter-storage

9

mod-di-converter-storage:2.1.5

2

1024

896

128

768

88

128

FALSE

mod-users

11

mod-users:19.2.2

2

1024

896

128

768

88

128

FALSE

mod-source-record-manager

10

mod-source-record-manager:3.7.7

2

5600

5000

2048

3500

384

512

FALSE

mod-quick-marc

10

mod-quick-marc:5.0.1

1

2288

2176

128

1664

384

512

FALSE

nginx-okapi

2

nginx-okapi:2023.06.14

2

1024

896

512

0

0

0

FALSE

okapi-b

1

okapi:5.1.2

3

1684

1440

1024

922

384

512

FALSE

mod-data-export

10

mod-data-export:4.8.7

1

1024

896

1024

768

88

128

FALSE

  • No labels