Overview
- This document contains the results of testing MOBIUS master-script tests on the MCPT environment with different environment configurations.
PERF-942 - Getting issue details... STATUS
Summary
- During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
- "memory": 1024 → 2304
- "memoryReservation": 896 → 2048
- -XX:MaxMetaspaceSize=128m → 256m
- -Xmx768m → 1536m
- After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.
No memory leaks, memory consumption was stable during all of the tests.
- Database CPU usage varied up to a maximum of 26%
Test Runs and Results
This table contains durations for all Workflows.
| Workflow | Test 1 Baseline Env configuration | Test 2 CPU=0 | Test 3 x2gd.xlarge CPU=0 instances=6 | Test 4 x2gd.xlarge CPU=0 instances=8 | Test 5 x2gd.large CPU=0 instances=10 | Test 6 r6g.xlarge CPU=2 instances=12(14) | Test 7 r6g.xlarge CPU=2 instances=14 placement strategy (one task per host) | Test 8 x2gd.large CPU=2 instances=14 placement strategy (one task per host) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | Average (milliseconds) | Errors | |
| AIE_TC: Create Invoices | 7682 | 100% | 8106 | 100% | 50468 | 100% | 15554 | 100% | 92805 | 100% | 10191 | 100% | 9716 | 100% | 91611 | 100% |
| AIE_TC: Invoices Approve | 2818 | 100% | 3016 | 100% | 24781 | 100% | 7720 | 100% | 51554 | 100% | 4512 | 100% | 4210 | 100% | 51626 | 100% |
| AIE_TC: Paying Invoices | 2882 | 100% | 3164 | 100% | 32994 | 100% | 9788 | 100% | 64642 | 100% | 5430 | 100% | 4902 | 100% | 65029 | 100% |
| CICO_TC_Check-In Controller | 2000 | 0% | 2214 | 0% | 19075 | 0% | 6038 | 0% | 32909 | 0% | 3659 | 0% | 3194 | 0% | 30831 | 0% |
| CICO_TC_Check-Out Controller | 3548 | 0% | 3799 | 0% | 29224 | 1% | 10560 | 0% | 57347 | 0% | 6439 | 0% | 5595 | 0% | 55503 | 0% |
| CSI_TC:Share local instance | 13050 | 19% | 13073 | 19% | 14265 | 16% | 16192 | 0% | 17361 | 2% | 13782 | 14% | 13339 | 17% | 17460 | 1% |
| DE_Exporting MARC Bib records custom workflow | 84707 | 0% | 59099 | 0% | 382491 | 56% | 116869 | 0% | 567389 | 83% | 127885 | 0% | 88541 | 0% | 500759 | 71% |
| DE_Exporting MARC Bib records workflow | 74126 | 0% | 43093 | 0% | 417982 | 57% | 97656 | 4% | 623610 | 91% | 82126 | 0% | 83046 | 0% | 457212 | 58% |
| EVA_TC: View Account | 519 | 0% | 565 | 0% | 16034 | 19% | 1736 | 0% | 40619 | 2% | 1138 | 1% | 972 | 1% | 37669 | 1% |
| ILR_TC: Create ILR | 1422 | 0% | 1499 | 0% | 11607 | 1% | 3882 | 0% | 23676 | 0% | 2414 | 0% | 1997 | 0% | 22335 | 0% |
| MSF_TC: mod search by auth query | 755 | 2% | 668 | 0% | 1888 | 0% | 1500 | 0% | 6180 | 0% | 1080 | 0% | 910 | 0% | 5894 | 0% |
| MSF_TC: mod search by boolean query | 205 | 1% | 159 | 0% | 594 | 0% | 427 | 0% | 2022 | 0% | 258 | 0% | 238 | 0% | 1871 | 0% |
| MSF_TC: mod search by contributors | 440 | 1% | 394 | 0% | 856 | 0% | 839 | 0% | 3454 | 0% | 616 | 0% | 530 | 0% | 3300 | 0% |
| MSF_TC: mod search by filter query | 302 | 0% | 284 | 0% | 512 | 0% | 539 | 0% | 2018 | 0% | 416 | 0% | 362 | 0% | 1924 | 0% |
| MSF_TC: mod search by keyword query | 310 | 0% | 280 | 0% | 521 | 0% | 537 | 0% | 2013 | 0% | 416 | 0% | 361 | 0% | 1912 | 0% |
| MSF_TC: mod search by subject query | 448 | 0% | 406 | 0% | 797 | 0% | 778 | 0% | 3076 | 0% | 620 | 0% | 519 | 0% | 2917 | 0% |
| MSF_TC: mod search by title query | 1090 | 1% | 1025 | 0% | 1435 | 0% | 1387 | 0% | 3687 | 0% | 1432 | 0% | 1149 | 0% | 3543 | 0% |
| OPIH_/oai/records | 5330 | 0% | 5404 | 0% | 9330 | 0% | 7677 | 0% | 6881 | 0% | 3327 | 0% | 6990 | 0% | 8335 | 0% |
| POO_TC: Add Order Lines | 52142 | 0% | 54193 | 0% | 282192 | 0% | 79206 | 0% | 412004 | 0% | 57735 | 0% | 57749 | 0% | 399917 | 0% |
| POO_TC: Approve Order | 40656 | 0% | 42523 | 0% | 211747 | 0% | 56446 | 0% | 265167 | 0% | 43930 | 0% | 43834 | 0% | 275935 | 0% |
| POO_TC Create Order | 30734 | 0% | 31652 | 0% | 107318 | 0% | 42940 | 0% | 49234 | 0% | 32121 | 0% | 43834 | 0% | 175643 | 0% |
| RTAC_TC: edge-rtac | 3735 | 0% | 16295 | 0% | 1387 | 0% | 57195 | 0% | 4205 | 0% | 3950 | 0% | 55595 | 0% | ||
| SDIC_Single Record Import (Create) | 13279 | 19% | 13894 | 19% | 45024 | 16% | 17305 | 0% | 79053 | 2% | 14650 | 14% | 14531 | 17% | 74201 | 1% |
| SDIU_Single Record Import (Update) | 18466 | 0% | 19432 | 0% | 218270 | 100% | 28207 | 0% | 118399 | 0% | 21736 | 0% | 20965 | 0% | 115777 | 0% |
| TC: Receiving-an-Order-Line | 43765 | 100% | 46104 | 100% | 218270 | 100% | 65242 | 100% | 325230 | 100% | 49024 | 100% | 48538 | 100% | 322267 | 100% |
| Serials-Receiving-Workflow | 45694 | 100% | 47336 | 100% | 198116 | 100% | 68545 | 100% | 302203 | 100% | 49873 | 100% | 50028 | 100% | 295508 | 100% |
| Unreceiving-a-Piece | 7823 | 100% | 7757 | 100% | 40059 | 100% | 13335 | 100% | 64155 | 100% | 9018 | 100% | 8717 | 100% | 60295 | 100% |
| ULR_TC: Users loan Renewal Transaction | 2810 | 0% | 3078 | 0% | 22602 | 1% | 7829 | 0% | 38383 | 0% | 4673 | 0% | 4030 | 0% | 36143 | 0% |
This table contains Data Export durations for Test Set №2.
This table contains Data Export durations for Test Set №3.
This table contains Data Export durations for Test Set №4.
This table contains Data Export durations for Test Set №5.
This table contains Data Export durations for Test Set №6.
Resource utilization for Test Set №1
Service CPU Utilization
Here we can see that mod-data-export used 575% CPU in spike.
Service Memory Utilization
Here we can see that data-export module used 88% memory.
Kafka metrics
DB CPU Utilization
DB CPU was MAX 20-22% for all tests.
DB Connections
Max number of DB connections was 1500 in avarage.
DB load
Top SQL-queries
Resource utilization for Test Set №2
Service CPU Utilization
Here we can see that mod-data-export used 570% CPU in spike.
Service Memory Utilization
Here we can see that data-export module used 110% memory.
Kafka metrics
DB CPU Utilization
DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.
DB Connections
Max number of DB connections was 2230.
DB load
Top SQL-queries
Resource utilization for Test Set №3
Service CPU Utilization
Here we can see that mod-data-export used 230% CPU.
Service Memory Utilization
Here we can see that mod-data-export used 108% memory .
Kafka metrics
DB CPU Utilization
DB CPU was 20%.
DB Connections
Max number of DB connections was 2040.
DB load
Top SQL-queries
Resource utilization for Test Set №4
Service CPU Utilization
Here we can see that mod-data-export used 325% CPU.
Service Memory Utilization
Here we can see that mod-data-export used 73% memory .
Kafka metrics
DB CPU Utilization
DB CPU was 22%.
DB Connections
Max number of DB connections was 2630.
DB load
Top SQL-queries
Resource utilization for Test Set №5
Service CPU Utilization
Here we can see that mod-data-export used 130% CPU.
Service Memory Utilization
Here we can see that mod-data-export used 82% memory .
Kafka metrics
DB CPU Utilization
DB CPU was 25%.
DB Connections
Max number of DB connections was 1850.
DB load
Top SQL-queries
Resource utilization for Test Set №6
Service CPU Utilization
Here we can see that mod-data-export used 590% CPU.
Service Memory Utilization
Here we can see that mod-data-export used 82% memory .
Kafka metrics
DB CPU Utilization
DB CPU was 25%.
DB Connections
Max number of DB connections was 3050.
DB load
Top SQL-queries
Appendix
Infrastructure
PTF - Baseline MCPT environment configuration
- 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - Number of records in DB:
- cs00000001_0042
- instances - 637243
- cs00000001
- authorities - 55000
- cs00000001_0003
- holdings - 146704
- instances - 148713
- cs00000001_0042
- Open Search ptf-test
- Data nodes
- Instance type - r6g.2xlarge.search
- Number of nodes - 4
- Version: OpenSearch_2_7_R20240502
- Dedicated master nodes
- Instance type - r6g.large.search
- Number of nodes - 3
- Data nodes
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Methodology/Approach
MOBIUS Tests: scenarios were started by JMeter script from load generator. We had 100% error count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.
- Test 1: Baseline MCPT Environment configuration according to tunning environment from previous report: task count: 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, for mod-users and mod-authtoken task count 6. Parameter srs.marcIndexers.delete.interval.seconds=86400 for mod-source-record-storage. Instances count: 14. Database r6g.4xlarge, Amazon OpenSearch Service ptf-test: r6g.2хlarge.search (4 nodes).
- Test 2: The Baseline MCPT Environment configuration was applied, and CPU=0 was set for all modules.
- Test 3: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 6, and CPU=0 was set for all services.
- Test 4: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.xlarge, the number of instances was changed to 8, and CPU=0 was set for all services.
- Test 5: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 10, and CPU=0 was set for all services.
- Test 6: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14 but 12 were used, and CPU=2 was set for all services.
- Test 7: The Baseline MCPT Environment configuration was applied, the instance type was changed to r6g.xlarge, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.
- Test 8: The Baseline MCPT Environment configuration was applied, the instance type was changed to x2gd.large, the number of instances was changed to 14, placement strategy was updated to "one task per host", and CPU=2 was set for all services.