PTF - Performance testing of Consolidating EC2 Instances (mcpt)
- 1 Overview
- 2 Summary
- 3 Test Runs and Results
- 4 Comparison
- 5 Test №1
- 6 Test №2
- 7 Test №3
- 8 Test №4
- 9 Test №5
- 10 Test №6
- 10.1 Service CPU Utilization
- 10.2 Service Memory Utilization
- 10.3 Instance CPU Utilization
- 10.4 Kafka metrics
- 10.5 DB CPU Utilization
- 10.6 DB Connections
- 10.7 DB load
- 10.8 Top SQL-queries
- 11 Test №7
- 11.1 Service CPU Utilization
- 11.2 Service Memory Utilization
- 11.3 Instance CPU Utilization
- 11.4 Kafka metrics
- 11.5 DB CPU Utilization
- 11.6 DB Connections
- 11.7 DB load
- 11.8 Top SQL-queries
- 12 Test №8
- 12.1 Service CPU Utilization
- 12.2 Service Memory Utilization
- 12.3 Instance CPU Utilization
- 12.4 Kafka metrics
- 12.5 DB CPU Utilization
- 12.6 DB Connections
- 12.7 DB load
- 12.8 Top SQL-queries
- 13 Appendix
- 13.1 Infrastructure
- 14 Methodology/Approach
Overview
Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances. We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0. To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments.
Summary
Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.
No memory leaks, memory consumption was stable during all of the tests.
Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.
Test Runs and Results
This table contains durations for all Workflows.
Workflow | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 | Test 8 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Average | Errors | Average | Errors | Average | Errors | Average | Errors | Average | Errors | Average | Errors | Average | Errors | Average | Errors | |
AIE_TC: Create Invoices | 7682 | 100% | 8106 | 100% | 50468 | 100% | 15554 | 100% | 92805 | 100% | 10191 | 100% | 9716 | 100% | 91611 | 100% |
AIE_TC: Invoices Approve | 2818 | 100% | 3016 | 100% | 24781 | 100% | 7720 | 100% | 51554 | 100% | 4512 | 100% | 4210 | 100% | 51626 | 100% |
AIE_TC: Paying Invoices | 2882 | 100% | 3164 | 100% | 32994 | 100% | 9788 | 100% | 64642 | 100% | 5430 | 100% | 4902 | 100% | 65029 | 100% |
CICO_TC_Check-In Controller | 2000 | 0% | 2214 | 0% | 19075 | 0% | 6038 | 0% | 32909 | 0% | 3659 | 0% | 3194 | 0% | 30831 | 0% |
CICO_TC_Check-Out Controller | 3548 | 0% | 3799 | 0% | 29224 | 1% | 10560 | 0% | 57347 | 0% | 6439 | 0% | 5595 | 0% | 55503 | 0% |
CSI_TC:Share local instance | 13050 | 19% | 13073 | 19% | 14265 | 16% | 16192 | 0% | 17361 | 2% | 13782 | 14% | 13339 | 17% | 17460 | 1% |
DE_Exporting MARC Bib records custom workflow | 84707 | 0% | 59099 | 0% | 382491 | 56% | 116869 | 0% | 567389 | 83% | 127885 | 0% | 88541 | 0% | 500759 | 71% |
DE_Exporting MARC Bib records workflow | 74126 | 0% | 43093 | 0% | 417982 | 57% | 97656 | 4% | 623610 | 91% | 82126 | 0% | 83046 | 0% | 457212 | 58% |
EVA_TC: View Account | 519 | 0% | 565 | 0% | 16034 | 19% | 1736 | 0% | 40619 | 2% | 1138 | 1% | 972 | 1% | 37669 | 1% |
ILR_TC: Create ILR | 1422 | 0% | 1499 | 0% | 11607 | 1% | 3882 | 0% | 23676 | 0% | 2414 | 0% | 1997 | 0% | 22335 | 0% |
MSF_TC: mod search by auth query | 755 | 2% | 668 | 0% | 1888 | 0% | 1500 | 0% | 6180 | 0% | 1080 | 0% | 910 | 0% | 5894 | 0% |
MSF_TC: mod search by boolean query | 205 | 1% | 159 | 0% | 594 | 0% | 427 | 0% | 2022 | 0% | 258 | 0% | 238 | 0% | 1871 | 0% |
MSF_TC: mod search by contributors | 440 | 1% | 394 | 0% | 856 | 0% | 839 | 0% | 3454 | 0% | 616 | 0% | 530 | 0% | 3300 | 0% |
MSF_TC: mod search by filter query | 302 | 0% | 284 | 0% | 512 | 0% | 539 | 0% | 2018 | 0% | 416 | 0% | 362 | 0% | 1924 | 0% |
MSF_TC: mod search by keyword query | 310 | 0% | 280 | 0% | 521 | 0% | 537 | 0% | 2013 | 0% | 416 | 0% | 361 | 0% | 1912 | 0% |
MSF_TC: mod search by subject query | 448 | 0% | 406 | 0% | 797 | 0% | 778 | 0% | 3076 | |||||||