/
PTF - Performance testing of Consolidating EC2 Instances (mcpt)

PTF - Performance testing of Consolidating EC2 Instances (mcpt)

Overview

  • Currently FSE's hosting of FOLIO consists of running ECS tasks (module docker containers) over 10-14 ec2 m6g.2xlarge instances.  We'd like to experiment with a different types of ec2 instance, x2gd, and with different numbers of ec2 instances to see if the performance is comparable to the baseline. Also, we'd like to understand the true CPU usage of the modules by setting setting CPU (units) in the modules' task definitions to 0.  To do this we ran the MOBIUS (multi-workflows, multi-tenants) tests on the mcpt environment with different environment configurations and compare the test results to the baseline's report. This report contains the test results of these experiments. 

PERF-942 - Getting issue details... STATUS  

Summary

  • Much performance degradation when we use less instance count for environment or instance type: .x2gd.large with less resources.
  • During the tests Parameter CPU=0 improved performance for a several workflows, so this point we are going to investigate more in this ticket.
  • No memory leaks, memory consumption was stable during all of the tests.

  • Tests had 100% errors count for AIE_TC: Create Invoices, AIE_TC: Invoices Approve, AIE_TC: Paying Invoices, TC: Receiving-an-Order-Line, Unreceiving-a-Piece and Unreceiving-a-Piece Workflows because data was not regenerated.  

Test Runs and Results

This table contains durations for all Workflows. 

WorkflowTest 1
Baseline Env
configuration
Test 2
CPU=0
Test 3
x2gd.xlarge
CPU=0
instances=6
Test 4
x2gd.xlarge
CPU=0
instances=8
Test 5
x2gd.large
CPU=0
instances=10
Test 6
r6g.xlarge
CPU=2
instances=12(14) 
Test 7
r6g.xlarge
CPU=2
instances=14
placement strategy
(one task per host)
Test 8
x2gd.large
CPU=2
instances=14
placement strategy
(one task per host)
Average
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
ErrorsAverage
(milliseconds)
Errors
AIE_TC: Create Invoices7682100%8106100%50468100%15554100%92805100%10191100%9716100%91611100%
AIE_TC: Invoices Approve2818100%3016100%24781100%7720100%51554100%4512100%4210100%51626100%
AIE_TC: Paying Invoices2882100%3164100%32994100%9788100%64642100%5430100%4902100%65029100%
CICO_TC_Check-In Controller20000%22140%190750%60380%329090%36590%31940%308310%
CICO_TC_Check-Out Controller35480%37990%292241%105600%573470%64390%55950%555030%
CSI_TC:Share local instance1305019%1307319%1426516%161920%173612%1378214%1333917%174601%
DE_Exporting MARC Bib records custom workflow847070%590990%38249156%1168690%56738983%1278850%885410%50075971%
DE_Exporting MARC Bib records workflow741260%430930%41798257%976564%62361091%821260%830460%45721258%
EVA_TC: View Account5190%5650%1603419%17360%406192%11381%9721%376691%
ILR_TC: Create ILR14220%14990%116071%38820%236760%24140%19970%223350%
MSF_TC: mod search by auth query7552%6680%18880%15000%61800%10800%9100%58940%
MSF_TC: mod search by boolean query2051%1590%5940%4270%20220%2580%2380%18710%
MSF_TC: mod search by contributors4401%3940%8560%8390%34540%6160%5300%33000%
MSF_TC: mod search by filter query3020%2840%5120%5390%20180%4160%3620%19240%
MSF_TC: mod search by keyword query3100%2800%5210%5370%20130%4160%3610%19120%
MSF_TC: mod search by subject query4480%4060%7970%7780%30760%6200%5190%29170%
MSF_TC: mod search by title query10901%10250%14350%13870%36870%14320%11490%35430%
OPIH_/oai/records53300%54040%93300%76770%68810%33270%69900%83350%
POO_TC: Add Order Lines521420%541930%2821920%792060%4120040%577350%577490%3999170%
POO_TC: Approve Order406560%425230%2117470%564460%2651670%439300%438340%2759350%
POO_TC Create Order307340%316520%1073180%429400%492340%321210%438340%1756430%