PTF - Performance Instance Resources Optimization - MCPT
Overview
- The primary objective of testing was to evaluate the performance of the Baseline MCPT Environment configuration while attempting to optimize costs by adjusting instance types and reducing the number of instances. The tests were designed to compare the performance outcomes across different configurations, including variations in instance types and counts within multiple Auto Scaling Groups (ASGs). By systematically modifying these variables, the goal was to maintain or improve the performance observed in the baseline configuration while achieving cost efficiency.
- PERF-961Getting issue details... STATUS
Summary
- Through a series of experiments involving different placement strategies, instance types, and total instance counts, we found that the performance remained consistent when using these two configuration:
- six
c7g.large
instances (three dedicated to theokapi
service and three allocated tomod-courses
,mod-sender
,mod-tasks-list
,mod-gobi
,edge-dematic
,mod-erm-usage
,mod-eusage-reports
,mod-notify
, andmod-data-import
services) alongside tenr7g.xlarge
instances for all other services, with the CPU parameter set to 2 for all services. three
c7g.large
instances dedicated to theokapi
service alongside fiver7g.2xlarge
instances for all other services, with the CPU parameter set to 2 for all services.
- six
- Notably, both environment configuration offers a 40% cost reduction compared to the existing setup, making it a more economical option without compromising on performance.
- Configurations with three c7g.large instances for the okapi service and five r7g.2xlarge instances for all other services show the best performance across all experiments. So it will be more tests in this ticket.
- Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.
Test Runs
Test # | Description | Status |
---|---|---|
Test 1 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: r7g.xlarge for others services. | Completed |
Test 2 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others modules. | Completed |
Test 3 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others services (Repeat Test 2). | Completed |
Test 4 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. | Completed |
Test 5 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules (Repeat Test 4). | Completed |
Test 6 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows. | Completed |
Test 7 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows (Repeat Test 6). | Completed |
Test 8 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. | Completed |
Test 9 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Test was run without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow workflows (Repeat Test 7 after Terminate Instances r7g.xlarge). | Completed |
Test 10 | CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules (Repeat Test 8 after Terminate Instances r7g.xlarge). | Completed |
Test 11 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. | Completed |
Test 12 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11). | Completed |
Test 13 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11). | Completed |
Test 14 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services (Repeat Test 11 after terminate 11 instances). | Completed |
Test 15 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi and mod-authtoken services and 11 Instance Type: r7g.xlarge for others services. | Completed |
Test 16 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: r7g.2xlarge for others services. | Completed |
Test 17 | CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 5 Instance Type: r7g.2xlarge for others services (Repeat Test 16). | Completed |
Test Results
This table contains durations for all Workflows.
Workflows | Test 1 | Test 2 | Test 3 (repeat 2) | Test 4 | Test 5 (repeat 4) | Test 6 | Test 7 (repeat 6 ) | Test 8 | Test 9 (repeat 7 after Terminate) | Test 10 (repeat 8 after Terminate) | Test 11 | Test 12 | Test 13 | Test 14 (Terminate) | Test 15 | Test 16 | Test 17 | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | Average response time (milliseconds) | Errors | |
AIE_TC: Create Invoices | 10227 | 100% | 7425 | 100% | 7645 | 100.00% | 8563 | 100.00% | 8132 | 100.00% | 7383 | 100.00% | 7575 | 100.00% | 7359 | 100.00% | 10061 | 100.00% | 11269 | 100.00% | 8607 | 100% | 11352 | 100% | 8293 | 100% | 9397 | 100% | 10687 | 100% | 8138 | 100% | 7063 | 100% |
AIE_TC: Invoices Approve | 4649 | 100% | 3281 | 100% | 3252 | 100.00% | 3442 | 100.00% | 3175 | 100.00% | 3081 | 100.00% | 2966 | 100.00% | 3100 | 100.00% | 5464 | 100.00% | 7132 | 100.00% | 3438 | 100% | 4792 | 100% | 3506 | 100% | 4052 | 100% | 5297 | 100% | 3054 | 100% | 2873 | 100% |
AIE_TC: Paying Invoices | 5686 | 100% | 3375 | 100% | 3481 | 100.00% | 3439 | 100.00% | 3250 | 100.00% | 3079 | 100.00% | 2948 | 100.00% | 3115 | 100.00% | 4504 | 100.00% | 4245 | 100.00% | 3580 | 100% | 4635 | 100% | 3548 | 100% | 4387 | 100% | 6265 | 100% | 3093 | 100% | 2811 | 100% |
CICO_TC_Check-In Controller | 3828 | 0% | 2350 | 0% | 2446 | 0.00% | 2325 | 0.00% | 2259 | 0.00% | 2092 | 0.00% | 2046 | 0.00% | 2122 | 0.00% | 3113 | 0.00% | 2965 | 0.00% | 2508 | 0% | 3162 | 0% | 2396 | 0% | 2896 | 0% | 3817 | 0% | 2154 | 0% | 1902 | 0% |
CICO_TC_Check-Out Controller | 5960 | 0% | 4085 | 0% | 4273 | 0.00% | 4129 | 0.00% | 3905 | 0.00% | 3769 | 0.00% | 3549 | 0.00% | 3816 | 0.00% | 5167 | 0.00% | 5224 | 0.00% | 4307 | 0% | 5954 | 1% | 4406 | 0% | 5100 | 0% | 6788 | 0% | 3721 | 0% | 3436 | 0% |
CSI_TC:Share local instance | 13045 | 19% | 16081 | 0% | 16091 | 0.00% | 13105 | 18.63% | 13046 | 18.98% | 16087 | 0.00% | 13119 | 18.66% | 16071 | 0.00% | 16076 | 0.00% | 16058 | 0.00% | 4307 | 0% | 13261 | 18% | 16054 | 0% | 13123 | 19% | 16071 | 0% | 13076 | 19% | 16005 | 0% |
DE_Exporting MARC Bib records custom workflow | 53484 | 0% | 101324 | 0% | 46860 | 0.00% | 97169 | 0.00% | 160022 | 0.00% | 44292 | 0.00% | 72372 | 0.00% | 49515 | 0% | 179136 | 38% | 64827 | 0% | 127373 | 0% | 63206 | 0% | 84900 | 0% | 69446 | 0% | ||||||
DE_Exporting MARC Bib records workflow | 42737 | 0% | 92213 | 0% | 64897 | 0.00% | 77747 | 0.00% | 265651 | 0.00% | 37622 | 0.00% | 70557 | 0.00% | 45640 | 0% | 187909 | 41% | 63085 | 0% | 159453 | 0% | 84220 | 0% | 72805 | 0% | 69905 | 0% | ||||||
EVA_TC: View Account | 822 | 0% | 578 | 0% | 582 | 0.00% | 578 | 0.00% | 553 | 0.00% | 523 | 0.00% | 506 | 0.00% | 534 | 0.00% | 3539 | 0.00% | 2709 | 0.00% | 602 | 0% | 1513 | 2% | 527 | 0% | 670 | 0% | 967 | 0% | 503 | 0% | 480 | 0% |
ILR_TC: Create ILR | 1979 | 0% | 1605 | 0% | 1645 | 0.00% | 1595 | 0.00% | 1523 | 0.00% | 1436 | 0.00% | 1394 | 0.00% | 1528 | 0.00% | 3147 | 0.00% | 3122 | 0.00% | 1576 | 0% | 2124 | 0% | 1708 | 0% | 1928 | 0% | 2643 | 0% | 1496 | 0% | 1298 | 0% |
MSF_TC: mod search by auth query | 2227 | 26% | 1672 | 0% | 824 | 0.00% | 933 | 2.15% | 686 | 0.00% | 686 | 0.00% | 680 | 0.00% | 692 | 0.00% | 693 | 0.00% | 622 | 0.00% | 770 | 0% | 1224 | 0% | 680 | 0% | 810 | 0% | 1079 | 0% | 666 | 0% | 669 | 0% |
MSF_TC: mod search by boolean query | 448 | 2% | 406 | 0% | 201 | 0.00% | 257 | 0.41% | 172 | 0.00% | 167 | 0.00% | 167 | 0.00% | 168 | 0.00% | 167 | 0.00% | 156 | 0.00% | 193 | 0% | 254 | 0% | 163 | 0% | 206 | 0% | 285 | 0% | 159 | 0% | 163 | 0% |
MSF_TC: mod search by contributors | 882 | 0% | 982 | 0% | 495 | 0.00% | 566 | 1.25% | 407 | 0.00% | 415 | 0.00% | 402 | 0.00% | 406 | 0.00% | 394 | 0.00% | 377 | 0.00% | 441 | 0% | 510 | 0% | 399 | 0% | 480 | 0% | 605 | 0% | 393 | 0% | 406 | 0% |
MSF_TC: mod search by filter query | 649 | 2% | 603 | 0% | 344 | 0.00% | 358 | 0.23% | 285 | 0.00% | 286 | 0.00% | 285 | 0.00% | 291 | 0.00% | 274 | 0.00% | 267 | 0.00% | 315 | 0% | 343 | 0% | 288 | 0% | 331 | 0% | 407 | 0% | 285 | 0% | 285 | 0% |
MSF_TC: mod search by keyword query | 604 | 0% | 596 | 0% | 345 | 0.00% | 374 | 0.41% | 290 | 0.00% | 288 | 0.00% | 286 | 0.00% | 292 | 0.00% | 274 | 0.00% | 263 | 0.00% | 316 | 0% | 343 | 0% | 283 | 0% | 329 | 0% | 402 | 0% | 284 | 0% | 287 | 0% |
MSF_TC: mod search by subject query | 930 | 1% | 932 | 0% | 515 | 0.00% | 598 | 0.80% | 420 | 0.00% | 430 | 0.00% | 419 | 0.00% | 425 | 0.00% | 408 | 0.00% | 390 | 0.00% | 456 | 0% | 563 | 0% | 416 | 0% | 485 | 0% | 588 | 0% | 413 | 0% | 417 | 0% |
MSF_TC: mod search by title query | 2381 | 0% | 1965 | 0% | 1260 | 0.00% | 1362 | 0.49% | 1059 | 0.00% | 1061 | 0.00% | 1054 | 0.00% | 1064 | 0.00% | 1026 | 0.00% | 1012 | 0.00% | 1093 | 0% | 1157 | 0% | 1062 | 0% | 1135 | 0% | 1248 | 0% | 1064 | 0% | 1081 | 0% |
OPIH_/oai/records | 5086 | 0% | 3248 | 0% | 3254 | 0.00% | 5193 | 0.00% | 3069 | 0.00% | 4113 | 0.00% | 2451 | 0.00% | 6267 | 0% | 4885 | 0% | 882 | 0% | 3026 | 0% | 943 | 0% | 2980 | 0% | 940 | 0% | ||||||
POO_TC: Add Order Lines | 60193 | 1% | 53341 | 0% | 54507 | 0.00% | 55008 | 0.00% | 54646 | 0.00% | 52806 | 0.00% | 52115 | 0.00% | 54083 | 0.00% | 49475 | 0.00% | 44615 | 0.00% | 56555 | 0% | 65436 | 2% | 59207 | 0% | 59732 | 0% | 62116 | 0% | 55661 | 0% | 53052 | 0% |
POO_TC: Approve Order | 45294 | 0% | 41750 | 0% | 41935 | 0.00% | 42445 | 0.00% | 42486 | 0.00% | 41569 | 0.00% | 41194 | 0.00% | 42034 | 0.00% | 37310 | 0.00% | 33982 | 0.00% | 44255 | 0% | 46133 | 0% | 46189 | 0% | 45238 | 0% | 47387 | 0% | 42564 | 0% | 41507 | 0% |
POO_TC Create Order | 34305 | 0% | 30583 | 0% | 30965 | 0.00% | 31853 | 0.00% | 31440 | 0.00% | 30970 | 0.00% | 30392 | 0.00% | 30575 | 0.00% | 28081 | 0.00% | 25451 | 0.00% | 32601 | 0% | 42267 | 1% | 33058 | 0% | 33996 | 0% | 22828 | 0% | 31745 | 0% | 30479 | 0% |
RTAC_TC: edge-rtac | 4050 | 0% | 2228 | 0% | 2184 | 0.00% | 3665 | 0.00% | 3787 | 0.00% | 1978 | 0.00% | 3888 | 0.00% | 1980 | 0.00% | 9297 | 0.00% | 10041 | 0.00% | 3868 | 0% | 8348 | 0% | 2115 | 0% | 3854 | 0% | 2236 | 0% | 3729 | 0% | 2005 | 0% |
SDIC_Single Record Import (Create) | 14783 | 19% | 11882 | 0% | 11441 | 0.00% | 13868 | 18.39% | 13345 | 18.82% | 10964 | 0.00% | 12949 | 18.23% | 11001 | 0.00% | 12015 | 0.00% | 11556 | 0.00% | 13848 | 18% | 16236 | 18% | 12538 | 0% | 14482 | 18% | 33721 | 0% | 13357 | 18% | 10702 | 0% |
SDIU_Single Record Import (Update) | 22347 | 4% | 19685 | 0% | 19298 | 0.00% | 19504 | 0.00% | 3787 | 0.00% | 18869 | 0.00% | 18706 | 0.00% | 18748 | 0.00% | 21745 | 0.00% | 20829 | 0.00% | 19741 | 0% | 23376 | 3% | 20741 | 0% | 20933 | 0% | 22828 | 0% | 19420 | 0% | 18330 | 0% |
TC: Receiving-an-Order-Line | 50466 | 100% | 44954 | 100% | 46684 | 100.00% | 46396 | 100.00% | 45548 | 100.00% | 45629 | 100.00% | 44356 | 100.00% | 45106 | 100.00% | 40982 | 100.00% | 36920 | 100.00% | 47570 | 100% | 53302 | 100% | 50146 | 100% | 49516 | 100% | 52266 | 100% | 46404 | 100% | 44880 | 100% |
Serials-Receiving-Workflow | 51572 | 100% | 46047 | 100% | 47460 | 100.00% | 48142 | 100.00% | 47661 | 100.00% | 45645 | 100.00% | 45404 | 100.00% | 46299 | 100.00% | 42105 | 100.00% | 38241 | 100.00% | 49262 | 100% | 61545 | 100% | 50285 | 100% | 50943 | 100% | 52493 | 100% | 48123 | 100% | 45827 | 100% |
Unreceiving-a-Piece | 9422 | 100% | 8028 | 100% | 8303 | 100.00% | 8247 | 100.00% | 7865 | 100.00% | 7897 | 100.00% | 7748 | 100.00% | 7887 | 100.00% | 7468 | 100.00% | 6812 | 100.00% | 8438 | 100% | 9945 | 100% | 8544 | 100% | 8829 | 100% | 9368 | 100% | 8058 | 100% | 7581 | 100% |
ULR_TC: Users loan Renewal Transaction | 4027 | 0% | 3198 | 0% | 3275 | 0.00% | 3060 | 0.00% | 2957 | 0.00% | 2814 | 0.00% | 2810 | 0.00% | 2969 | 0.00% | 44632 | 0.00% | 88602 | 0.00% | 3197 | 0% | 5080 | 1% | 3406 | 0% | 3583 | 0% | 4955 | 0% | 2865 | 0% | 2626 | 0% |
Comparison
This graph shows the durations of all workflows compared between the best test result from Test №8, MCPT Basile configuration with 14 instances m6g.2xlarge and result from mcpt tunning report.
Average Case workflows: Part 1.
Average Case workflows: Part 2.
Average Case workflows: Part 3.
Test №1
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of test was to evaluate the performance of the MCPT environment by applying the baseline configuration, while adjusting the CPU allocation for all services to 2 vCPUs. Additionally, the test aimed to assess the impact of using two distinct autoscaling groups with different instance types: c7g.large for the Okapi service and r7g.xlarge for all other services. By running the Fixed Load (average case) MOBIUS test, the goal was to determine if this modified configuration could achieve similar or improved performance compared to the baseline, while potentially optimizing resource allocation and cost.
Results: Almost all workflows shows significant performance degradation.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi modules used 42k% CPU power parameter CPU=2 for module.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% average with ERW: Exporting Receiving Information.
DB Connections
Max number of DB connections was 5600 in maximum.
Test №2
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others modules. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to maintain the use of m6g.2xlarge
instances for the main modules, as in the Baseline configuration, while introducing a second autoscaling group with c7g.large
instances specifically for the Okapi service. This setup aims to explore the performance and cost-effectiveness of separating the Okapi service onto a smaller instance type while keeping the main services on the larger m6g.2xlarge
instances.
Results: We observed nearly identical performance results in over half of the workflows compared to the Baseline configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 47k% of the CPU power parameter CPU=2 for module.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% average with ERW: Exporting Receiving Information
DB Connections
Max number of DB connections was 5710.
Test №3
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 10 Instance Type: m6g.2xlarge for others services. Fixed Load (average case) MOBIUS test was run (Repeat Test 2).
Objective: The objective of this test was to validate the consistency of performance observed in Test 2 by repeating the same configuration.
Results: We observed nearly identical performance results almost for all the workflows compared to the Baseline configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of the parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99% maximum.
DB Connections
Max number of DB connections was 5500.
Test №4
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to add more services with lower activity levels to the first autoscaling group, increasing the number of c7g.large
instances from 3 to 6. For the main services, r7g.xlarge
instances were used with a total of 10 instances, which are more cost-effective compared to the Baseline configuration with r6g.4xlarge.
Results: We observed a better performance compared to previous test.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5810.
DB load
Top SQL-queries
Test №5
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 4).
Objective: The objective of this test was to validate the consistency of performance observed in Test 4 by repeating the same configuration.
Results: We confirmed performance improvements with this configuration, also a several workflows shoes better performance than in Test 4.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5500.
DB load
Top SQL-queries
Test №6
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run.
Objective: The goal of this test was to exclude several unstable workflows and observe whether their removal would have a noticeable impact on the performance of the remaining workflows. Specifically, by omitting the DE_Exporting MARC Bib records
custom workflow, DE_Exporting MARC Bib records
workflow, and OPIH_/oai/records
workflow, the test aimed to assess if the stability and performance of the other workflows would improve under the same configuration.
Results: We observed minor performance improvements across almost all workflows. However, these improvements were not substantial. It appears that the observed effects are relatively small and can be attributed to a reduced overall load in the environment.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 46k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 6000.
DB load
Top SQL-queries
Test №7 - 8
Introduction:
- Test 7: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 6).
- Test 8: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of these tests was to validate the consistency of performance observed in Test 5 and Test 6. This was achieved by repeating the same configuration.
Results: We see performance improvements for Test 8.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 6010.
DB load
Top SQL-queries
Test №9 - 10
Introduction:
- Test 9: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case without DE_Exporting MARC Bib records custom workflow, DE_Exporting MARC Bib records workflow, and OPIH_/oai/records workflow) MOBIUS test was run (Repeat Test 7 after Terminate Instances r7g.xlarge).
- Test 10: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all modules, used two autoscaling groups, 1st with 6 Instance Type: c7g.large, 3 of them for okapi service and other 3 for mod-cources, mod-sender, mod-tasks-list, mod-gobi, edge-dematic, mod-erm-usage, mod-eusage-reports, mod-notify, mod-data-import services and 10 Instance Type: r7g.xlarge for others modules. Fixed Load (average case) MOBIUS test was run (Repeat Test 8 after Terminate Instances r7g.xlarge).
Objective: The objective of these tests was to validate the consistency of performance observed in Test 7 and Test 8. This was achieved by repeating the same configuration and applying new random task locations per instance after terminating the r7g.xlarge
instances.
Results: Performance result were worse on 50% for several workflows after terminating the r7g.xlarge instances and applying new random task locations per instance.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 48k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5150.
DB load
Top SQL-queries
Test №11
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The goal of this test was to replace three c7g.large
instances with one additional r7g.xlarge
instance for all services, while retaining three c7g.large
instances specifically for the okapi
service.
Results: The performance was worse compared to when we used the additional three c7g.large
instances for services with smaller workloads.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 45k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 5900.
DB load
Top SQL-queries
Test №12 - 13
Introduction: The Baseline MCPT Environment configuration was applied, and CPU=2 was set for all services, used two autoscaling groups, 1st with 3 Instance Type: c7g.large for okapi service and 11 Instance Type: r7g.xlarge for others services. Fixed Load (average case) MOBIUS test was run.
Objective: The objective of this test was to validate the degradation of performance observed in Test 11 by repeating the same configuration.
Results: We confirmed performance degradation with this configuration.
Instance CPU Utilization
Service CPU Utilization
Here we can see that okapi used 43k% of the CPU power of parameter CPU=2.
Service Memory Utilization
Here we can't see any sign of memory leaks on every module. Memory shows stable trend.
Kafka metrics
DB CPU Utilization
DB CPU was 99%.
DB Connections
Max number of DB connections was 6200.