Skip to end of banner
Go to start of banner

Data Import (batch imports) on 62 tenants-MCPT

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Overview

This document contains the results of the testing Data Import (batch imports) on 62 tenants with the following scenarios on the MCPT cluster PERF-778 - Getting issue details... STATUS :

  1. Central tenant: Large update imports - 2K; 30% Member tenants: single record imports (50% create, 50% update)

  2. Central tenant: Large update imports - 10K; 100% Member tenants: single record imports (50% create, 50% update)

Summary

  1. The average duration of single record imports (50% create, 50% update) during :

  • DI_T1(CT-2k_MT-SRI-50/50). was 1,2 second for Create profile and 1 second for Update;

  • DI_T2(CT-10k_MT-SRI-50/50). was 1,5 second for Create profile and 1,2 second for Update;

  1. All Data-import finished with the status “Completed“

  2. Most CPU consuming services were mod-consortia-b , mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b, mod-source-record-storage-b, mod-inventory-storage-b, okapi-b, mod-source-record-manager-b it`s values are in Service CPU Utilization section

  3. During 2 tests few services showed changes in memory utilization: mod-data-import-b, mod-search-b, mod-inventory-b, mod-source-record-manager-b, mod-inventory-storage-b,mod-source-record-storage-b its values are in Service Memory Utilization section. No memory leaks.

  4. DB CPU usage during DI_T1(CT-2k_MT-SRI-50/50) has increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) increased up to 95%.

Recommendations & Jiras

Add more CPU for 4 first services that have high CPU(>100%) utilization mod-consortia-b, mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b

Test Results

DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Tenant

Duration

DI_Importing MARC 2K records

1 minute

SDIU_(Update) cs00000001_0022

12275.0

SDIU_(Update) cs00000001_0016

13087.0

SDIU_(Update) cs00000001_0013

13900.0

SDIU_(Update) cs00000001_0010

13340.0

SDIU_(Update) cs00000001_0008

10973.0

SDIU_(Update) cs00000001_0006

11200.0

SDIU_(Update) cs00000001_0005

10832.0

SDIU_(Update) cs00000001_0002

11448.0

SDIC_(Create) cs00000001_0021

12036.0

SDIC_(Create) cs00000001_0020

11906.0

SDIC_(Create) cs00000001_0019

11090.0

SDIC_(Create) cs00000001_0018

12009.0

SDIC_(Create) cs00000001_0017

12395.0

SDIC_(Create) cs00000001_0015

12139.0

SDIC_(Create) cs00000001_0014

12710.0

SDIC_(Create) cs00000001_0012

12025.0

SDIC_(Create) cs00000001_0011

12063.0

SDIC_(Create) cs00000001_0009

10323.0

SDIC_(Create) cs00000001_0007

5057.0

SDIC_(Create) cs00000001_0004

5398.0

SDIC_(Create) cs00000001_0003

5854.0

The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

Metrics

Inventory Single Record - Default Update Instance

Inventory Single Record - Default Create Instance

Avarage

1,2 second

1,0 second

Max 

1,4 second

1,4 second

Min

1 second

500 ms

75 th percentile

1,3 second

1,2 second

90 th percentile

1,35 second

1, 3 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”

image-20240229-110340.png

DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).

Tenant

Duration

DI_Importing MARC records workflow Transaction

5 minutes

SDIC_Single Record Import (Create) cs00000001_0002

5163.0

SDIC_Single Record Import (Create) cs00000001_0003

5156.0

SDIC_Single Record Import (Create) cs00000001_0006

5672.0

SDIC_Single Record Import (Create) cs00000001_0007

4240.0

SDIC_Single Record Import (Create) cs00000001_0008

4108.0

SDIC_Single Record Import (Create) cs00000001_0010

9930.0

SDIC_Single Record Import (Create) cs00000001_0012

6010.0

SDIC_Single Record Import (Create) cs00000001_0015

10533.0

SDIC_Single Record Import (Create) cs00000001_0017

11315.0

SDIC_Single Record Import (Create) cs00000001_0018

11665.0

SDIC_Single Record Import (Create) cs00000001_0021

12022.0

SDIC_Single Record Import (Create) cs00000001_0022

10825.0

SDIC_Single Record Import (Create) cs00000001_0023

16624.0

SDIC_Single Record Import (Create) cs00000001_0024

16185.0

SDIC_Single Record Import (Create) cs00000001_0025

16557.0

SDIC_Single Record Import (Create) cs00000001_0026

16172.0

SDIC_Single Record Import (Create) cs00000001_0030

15672.0

SDIC_Single Record Import (Create) cs00000001_0031

15733.0

SDIC_Single Record Import (Create) cs00000001_0032

15748.0

SDIC_Single Record Import (Create) cs00000001_0033

15163.0

SDIC_Single Record Import (Create) cs00000001_0034

15585.0

SDIC_Single Record Import (Create) cs00000001_0036

14865.0

SDIC_Single Record Import (Create) cs00000001_0039

15693.0

SDIC_Single Record Import (Create) cs00000001_0041

15893.0

SDIC_Single Record Import (Create) cs00000001_0047

15149.0

SDIC_Single Record Import (Create) cs00000001_0048

14920.0

SDIC_Single Record Import (Create) cs00000001_0049

15412.0

SDIC_Single Record Import (Create) cs00000001_0052

15820.0

SDIC_Single Record Import (Create) cs00000001_0055

15334.0

SDIC_Single Record Import (Create) cs00000001_0061

15792.0

SDIC_Single Record Import (Create) cs00000001_0062

15885.0

SDIU_Single Record Import (Update) cs00000001_0004

11596.0

SDIU_Single Record Import (Update) cs00000001_0005

10354.0

SDIU_Single Record Import (Update) cs00000001_0009

11572.0

SDIU_Single Record Import (Update) cs00000001_0011

11884.0

SDIU_Single Record Import (Update) cs00000001_0013

11458.0

SDIU_Single Record Import (Update) cs00000001_0014

11672.0

SDIU_Single Record Import (Update) cs00000001_0016

12129.0

SDIU_Single Record Import (Update) cs00000001_0019

11302.0

SDIU_Single Record Import (Update) cs00000001_0020

12091.0

SDIU_Single Record Import (Update) cs00000001_0027

16105.0

SDIU_Single Record Import (Update) cs00000001_0028

16928.0

SDIU_Single Record Import (Update) cs00000001_0029

17151.0

SDIU_Single Record Import (Update) cs00000001_0035

16954.0

SDIU_Single Record Import (Update) cs00000001_0037

16585.0

SDIU_Single Record Import (Update) cs00000001_0038

16069.0

SDIU_Single Record Import (Update) cs00000001_0040

17904.0

SDIU_Single Record Import (Update) cs00000001_0042

16827.0

SDIU_Single Record Import (Update) cs00000001_0043

17005.0

SDIU_Single Record Import (Update) cs00000001_0044

17192.0

SDIU_Single Record Import (Update) cs00000001_0045

16593.0

SDIU_Single Record Import (Update) cs00000001_0046

17086.0

SDIU_Single Record Import (Update) cs00000001_0050

16416.0

SDIU_Single Record Import (Update) cs00000001_0051

16409.0

SDIU_Single Record Import (Update) cs00000001_0053

15979.0

SDIU_Single Record Import (Update) cs00000001_0054

16308.0

SDIU_Single Record Import (Update) cs00000001_0056

15553.0

SDIU_Single Record Import (Update) cs00000001_0057

16380.0

SDIU_Single Record Import (Update) cs00000001_0058

16634.0

SDIU_Single Record Import (Update) cs00000001_0059

16209.0

SDIU_Single Record Import (Update) cs00000001_0060

15943.0

The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.

Metrics

Inventory Single Record - Default Update Instance

Inventory Single Record - Default Create Instance

Avarage

1,2 second

1,5 second

Max 

1,65 second

1.,8 second

Min

4,1 second

1 second

75 th percentile

1,6 second

1,67 second

90 th percentile

1, 6 second

1,71 second

Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”

image-20240229-112256.png

Resource utilization

This section presents a comprehensive collection of key graphs and metrics necessary for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.

Service CPU Utilization

Service 

DI_T1(CT-2k_MT-SRI-50/50).

DI_T2(CT-10k_MT-SRI-50/50).

mod-consortia-b

179%

189%

mod-quick-marc-b

152%

80%

mod-inventory-b

99%

73%

mod-di-converter-storage-b

95%

105%

mod-source-record-storage-b

49%

48%

mod-inventory-storage-b

36%

32%

okapi-b

28%

37%

mod-source-record-manager-b

26%

29%

image-20240229-113621.png

Memory Utilization

During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns

Service 

MU before the T1, T2

MU after the T1, T2

mod-data-import-b

36

50

mod-search-b

24

31

mod-inventory-b

15

28

mod-source-record-manager-b

14

22

mod-inventory-storage-b

13

23

mod-source-record-storage-b

12

16

image-20240229-114546.png

DB CPU Utilization

CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.

image-20240229-115251.png

DB Connections

The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.

image-20240229-123308.png

DB Load

image-20240229-123837.png

SQL queries

image-20240229-124048.png

Methodology/Approach

All tests were run using the JMeter script.

Infrastructure

PTF -environment mcpt

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database instance, writer

    Name Memory GIBv CPUsmax_connections

    db.r6g.4xlarge 128 GiB; 16 vCPU

  • MSK tenant

    • 2 m5.2xlarge brokers in 1 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

The table contains modules, memory, and CPU parameters

Module

Task Def. Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mcpt-pvt 

mod-consortia

11

mod-consortia:1.0.3

2

2048

1802

128

768

88

128

FALSE

mod-organizations

6

mod-organizations:1.8.0

2

1024

896

128

700

88

128

FALSE

mod-data-import

7

mod-data-import:3.0.7

1

2048

1844

256

1292

384

512

FALSE

mod-search

12

mod-search:3.0.5

2

2592

2480

2048

1440

512

1024

FALSE

mod-configuration

6

mod-configuration:5.9.2

2

1024

896

128

768

88

128

FALSE

mod-permissions

14

mod-permissions:6.4.0

2

1684

1544

512

1024

384

512

FALSE

mod-entities-links

10

mod-entities-links:2.0.4

2

2592

2480

400

1440

0

1024

FALSE

mod-inventory-storage

10

mod-inventory-storage:27.0.4

2

4096

3690

2048

3076

384

512

FALSE

mod-data-export-worker

9

mod-data-export-worker:3.1.2

2

3072

2800

1024

2048

384

512

FALSE

mod-source-record-storage

10

mod-source-record-storage:5.7.5

2

5600

5000

2048

3500

384

512

FALSE

mod-inventory

10

mod-inventory:20.1.6

2

2880

2592

1024

1814

384

512

FALSE

mod-di-converter-storage

9

mod-di-converter-storage:2.1.5

2

1024

896

128

768

88

128

FALSE

mod-users

11

mod-users:19.2.2

2

1024

896

128

768

88

128

FALSE

mod-source-record-manager

10

mod-source-record-manager:3.7.7

2

5600

5000

2048

3500

384

512

FALSE

mod-quick-marc

10

mod-quick-marc:5.0.1

1

2288

2176

128

1664

384

512

FALSE

nginx-okapi

2

nginx-okapi:2023.06.14

2

1024

896

512

0

0

0

FALSE

okapi-b

1

okapi:5.1.2

3

1684

1440

1024

922

384

512

FALSE

mod-data-export

10

mod-data-export:4.8.7

1

1024

896

1024

768

88

128

FALSE

  • No labels