Overview
This document contains the results of the testing Data Import (batch imports) on 62 tenants with the following scenarios on the MCPT cluster PERF-778 - Getting issue details... STATUS :
Summary
The average duration of single record imports on 61 member tenants (50% member tenants create, 50% member tenants update) during :
DI_T1(CT-2k_MT-SRI-50/50). was 1,2 second for Create profile and 1 second for Update;
DI_T2(CT-10k_MT-SRI-50/50). was 1,5 second for Create profile and 1,2 second for Update;
All Data-import finished with the status “Completed“
Most CPU consuming services were mod-consortia-b , mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b, mod-source-record-storage-b, mod-inventory-storage-b, okapi-b, mod-source-record-manager-b their values are in Service CPU Utilization section
During 2 tests no memory leaks.
DB CPU usage during DI_T1(CT-2k_MT-SRI-50/50) has increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) increased up to 95%.
Recommendations & Jiras
Add more CPU for 4 first services that have high CPU(>100%) utilization mod-consortia-b, mod-quick-marc-b, mod-inventory-b, mod-di-converter-storage-b
Test Runs
DI_T1(CT-2k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 2K MARC BIB records; 30% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.
DI_T2(CT-10k_MT-SRI-50/50). Central tenant(CT) performs a large update batch import with 10K MARC BIB records; 100% Member tenants(MT) does single record imports(SRI) where 50% of them doing Create and 50% doing Update of single records.
Test Results
DI_T1(CT-2k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 2K records and single record imports on 21 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).
Tenant | Duration |
Central Tenant: DI_Import MARC BIB create 2K records | 1 minute |
SDIU_(Update) cs00000001_0022 | 12 sec |
SDIU_(Update) cs00000001_0016 | 13 sec |
SDIU_(Update) cs00000001_0013 | 13,9 sec |
SDIU_(Update) cs00000001_0010 | 13,3 sec |
SDIU_(Update) cs00000001_0008 | 11 sec |
SDIU_(Update) cs00000001_0006 | 11 sec |
SDIU_(Update) cs00000001_0005 | 11 sec |
SDIU_(Update) cs00000001_0002 | 11,5 sec |
SDIC_(Create) cs00000001_0021 | 12 sec |
SDIC_(Create) cs00000001_0020 | 12 sec |
SDIC_(Create) cs00000001_0019 | 11 sec |
SDIC_(Create) cs00000001_0018 | 12 sec |
SDIC_(Create) cs00000001_0017 | 12 sec |
SDIC_(Create) cs00000001_0015 | 12 sec |
SDIC_(Create) cs00000001_0014 | 12,7 sec |
SDIC_(Create) cs00000001_0012 | 12 sec |
SDIC_(Create) cs00000001_0011 | 12 sec |
SDIC_(Create) cs00000001_0009 | 10 sec |
SDIC_(Create) cs00000001_0007 | 5 sec |
SDIC_(Create) cs00000001_0004 | 5,3 sec |
SDIC_(Create) cs00000001_0003 | 5,8 sec |
The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.
Metrics | Inventory Single Record - Default Update Instance | Inventory Single Record - Default Create Instance |
Average | 1,2 second | 1,0 second |
Max | 1,4 second | 1,4 second |
Min | 1 second | 500 ms |
75th percentile | 1,3 second | 1,2 second |
90th percentile | 1,35 second | 1, 3 second |
Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”
DI_T2(CT-10k_MT-SRI-50/50). This section contains durations for Data-import for jobs with “PTF-Create-3” on Central tenant: with 10K records and single record imports on 62 member tenants (50% with “Inventory Single Record - Default Update Instance“ profile, 50% with “Inventory Single Record - Default Create Instance“ profile).
Tenant | Duration |
Central Tenant: DI_Importing MARC records workflow Transaction | 5 minutes |
SDIC_Single Record Import (Create) cs00000001_0002 | 5 sec |
SDIC_Single Record Import (Create) cs00000001_0003 | 5 sec |
SDIC_Single Record Import (Create) cs00000001_0006 | 5,6 sec |
SDIC_Single Record Import (Create) cs00000001_0007 | 4,2 sec |
SDIC_Single Record Import (Create) cs00000001_0008 | 4,1 sec |
SDIC_Single Record Import (Create) cs00000001_0010 | 10 sec |
SDIC_Single Record Import (Create) cs00000001_0012 | 6 sec |
SDIC_Single Record Import (Create) cs00000001_0015 | 10 sec |
SDIC_Single Record Import (Create) cs00000001_0017 | 11,3 sec |
SDIC_Single Record Import (Create) cs00000001_0018 | 11,6 sec |
SDIC_Single Record Import (Create) cs00000001_0021 | 12 sec |
SDIC_Single Record Import (Create) cs00000001_0022 | 11 sec |
SDIC_Single Record Import (Create) cs00000001_0023 | 16 sec |
SDIC_Single Record Import (Create) cs00000001_0024 | 16 sec |
SDIC_Single Record Import (Create) cs00000001_0025 | 16,5 sec |
SDIC_Single Record Import (Create) cs00000001_0026 | 16,1 sec |
SDIC_Single Record Import (Create) cs00000001_0030 | 15,6 sec |
SDIC_Single Record Import (Create) cs00000001_0031 | 15,7 sec |
SDIC_Single Record Import (Create) cs00000001_0032 | 15,7 sec |
SDIC_Single Record Import (Create) cs00000001_0033 | 15,1 sec |
SDIC_Single Record Import (Create) cs00000001_0034 | 15,5 sec |
SDIC_Single Record Import (Create) cs00000001_0036 | 14,8 sec |
SDIC_Single Record Import (Create) cs00000001_0039 | 15,6 sec |
SDIC_Single Record Import (Create) cs00000001_0041 | 15,8 sec |
SDIC_Single Record Import (Create) cs00000001_0047 | 15,1 sec |
SDIC_Single Record Import (Create) cs00000001_0048 | 14,9 sec |
SDIC_Single Record Import (Create) cs00000001_0049 | 15,4 sec |
SDIC_Single Record Import (Create) cs00000001_0052 | 15,8 sec |
SDIC_Single Record Import (Create) cs00000001_0055 | 15,3 sec |
SDIC_Single Record Import (Create) cs00000001_0061 | 15,7 sec |
SDIC_Single Record Import (Create) cs00000001_0062 | 15,8 sec |
SDIU_Single Record Import (Update) cs00000001_0004 | 11,6 sec |
SDIU_Single Record Import (Update) cs00000001_0005 | 10,3 sec |
SDIU_Single Record Import (Update) cs00000001_0009 | 11,7 sec |
SDIU_Single Record Import (Update) cs00000001_0011 | 11,8 sec |
SDIU_Single Record Import (Update) cs00000001_0013 | 11,4 sec |
SDIU_Single Record Import (Update) cs00000001_0014 | 11,6 sec |
SDIU_Single Record Import (Update) cs00000001_0016 | 12,1 sec |
SDIU_Single Record Import (Update) cs00000001_0019 | 11,3 sec |
SDIU_Single Record Import (Update) cs00000001_0020 | 12,1 sec |
SDIU_Single Record Import (Update) cs00000001_0027 | 16,1 sec |
SDIU_Single Record Import (Update) cs00000001_0028 | 16,9 sec |
SDIU_Single Record Import (Update) cs00000001_0029 | 17,1 sec |
SDIU_Single Record Import (Update) cs00000001_0035 | 16,9 sec |
SDIU_Single Record Import (Update) cs00000001_0037 | 16,5 sec |
SDIU_Single Record Import (Update) cs00000001_0038 | 16 sec |
SDIU_Single Record Import (Update) cs00000001_0040 | 18 sec |
SDIU_Single Record Import (Update) cs00000001_0042 | 16,8 sec |
SDIU_Single Record Import (Update) cs00000001_0043 | 17 sec |
SDIU_Single Record Import (Update) cs00000001_0044 | 17,1 sec |
SDIU_Single Record Import (Update) cs00000001_0045 | 16,5 sec |
SDIU_Single Record Import (Update) cs00000001_0046 | 17,1 sec |
SDIU_Single Record Import (Update) cs00000001_0050 | 16,5 sec |
SDIU_Single Record Import (Update) cs00000001_0051 | 16,5 sec |
SDIU_Single Record Import (Update) cs00000001_0053 | 16 sec |
SDIU_Single Record Import (Update) cs00000001_0054 | 16,3 sec |
SDIU_Single Record Import (Update) cs00000001_0056 | 15,5 sec |
SDIU_Single Record Import (Update) cs00000001_0057 | 16,3 sec |
SDIU_Single Record Import (Update) cs00000001_0058 | 16,6 sec |
SDIU_Single Record Import (Update) cs00000001_0059 | 16,2 sec |
SDIU_Single Record Import (Update) cs00000001_0060 | 15,9 sec |
The table below contains metrics for single record Data-import with “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance” profiles.
Metrics | Inventory Single Record - Default Update Instance | Inventory Single Record - Default Create Instance |
Average | 1,2 second | 1,5 second |
Max | 1,65 second | 1,8 second |
Min | 4,1 second | 1 second |
75th percentile | 1,6 second | 1,67 second |
90th percentile | 1,6 second | 1,71 second |
Dependency graph illustrating the change in data import duration across tenants for two profiles “Inventory Single Record - Default Update Instance” and ”Inventory Single Record - Default Create Instance”
Resource utilization
This section presents a comprehensive collection of key graphs and metrics necessary for understanding system performance during tests. This area serves as a portal for visualizing and decoding how the various resources have been used and impacted during different stages of performance testing.
Service CPU Utilization
Service | DI_T1(CT-2k_MT-SRI-50/50) | DI_T2(CT-10k_MT-SRI-50/50) |
mod-consortia-b | 179% | 189% |
mod-quick-marc-b | 152% | 80% |
mod-inventory-b | 99% | 73% |
mod-di-converter-storage-b | 95% | 105% |
mod-source-record-storage-b | 49% | 48% |
mod-inventory-storage-b | 36% | 32% |
okapi-b | 28% | 37% |
mod-source-record-manager-b | 26% | 29% |
Memory Utilization
During testing, the memory usage increases only on the services that are in the table below, also Memory usage values before test 1 and test2 and after the tests are in the corresponding columns
Service | MU before the T1, T2 | MU after the T1, T2 |
mod-data-import-b | 36 | 50 |
mod-search-b | 24 | 31 |
mod-inventory-b | 15 | 28 |
mod-source-record-manager-b | 14 | 22 |
mod-inventory-storage-b | 13 | 23 |
mod-source-record-storage-b | 12 | 16 |
DB CPU Utilization
CPU usage before tets was about 3%, during DI_T1(CT-2k_MT-SRI-50/50) CPU usage increased up to 80% and during DI_T2(CT-10k_MT-SRI-50/50) CPU usage increased up to 95%.
DB Connections
The number of DB connections before the tests were about 750 during DI_T1(CT-2k_MT-SRI-50/50) maximal number was about 1750 and during DI_T2(CT-10k_MT-SRI-50/50) maximal number was about 2300.
DB Load
SQL queries
Methodology/Approach
All tests were run using the JMeter script.
Infrastructure
PTF -environment mcpt
11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIBv CPUsmax_connections
db.r6g.4xlarge 128 GiB; 16 vCPU
MSK tenant
2 m5.2xlarge brokers in 1 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
The table contains modules, memory, and CPU parameters
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
mcpt-pvt | ||||||||||
mod-consortia | 11 | 2 | 2048 | 1802 | 128 | 768 | 88 | 128 | FALSE | |
mod-organizations | 6 | 2 | 1024 | 896 | 128 | 700 | 88 | 128 | FALSE | |
mod-data-import | 7 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | FALSE | |
mod-search | 12 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | FALSE | |
mod-configuration | 6 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-permissions | 14 | 2 | 1684 | 1544 | 512 | 1024 | 384 | 512 | FALSE | |
mod-entities-links | 10 | 2 | 2592 | 2480 | 400 | 1440 | 0 | 1024 | FALSE | |
mod-inventory-storage | 10 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | FALSE | |
mod-data-export-worker | 9 | 2 | 3072 | 2800 | 1024 | 2048 | 384 | 512 | FALSE | |
mod-source-record-storage | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-inventory | 10 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | FALSE | |
mod-di-converter-storage | 9 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-users | 11 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | FALSE | |
mod-source-record-manager | 10 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | FALSE | |
mod-quick-marc | 10 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | FALSE | |
nginx-okapi | 2 | 2 | 1024 | 896 | 512 | 0 | 0 | 0 | FALSE | |
okapi-b | 1 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | FALSE | |
mod-data-export | 10 | 1 | 1024 | 896 | 1024 | 768 | 88 | 128 | FALSE |