Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Test Details & Resource Utilization

1 user - non-Eureka

...

Image Removed

1 user - Eureka

...

Image Removed

20 users - non-Eureka

...

Image Removed

20 users - Eureka

...

Image Removed

OpenSearch metrics

Image Removed

Image Removed

Image Removed

...

Table of Contents

Overview

  1. In this workflow, we load test mod-search for the Quesnelia release -
    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-993

We tested with 1, 20 virtual users on non-Eureka and Eureka environments. 

Summary

  • For a single user, performance is better in the Eureka environment. However, when scaled to 20 users, there is a performance degradation of approximately 7.8%

  • Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 users - non-Eureka. 
  • Service CPU utilization was up to 60% for mod-search and 5% for mod-inventory-storage, for all other services CPU did not exceed 4%.

  • Memory utilization was stable and no memory leaks or OOM issues were observed.

  • RDS CPU utilization was about up to 6%.

Recommendations & Jiras

Errors during 20 users - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

  • Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
  • For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
    (8 VCPUs * 3) / 2 + 1 = 13 operations
  • For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
    4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs /Results

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.

...

55.2 ops/s

...

Table of Contents

Overview

  1. In this workflow, we perform a load test on mod-search for the Quesnelia release.
  2. Run these tests in both Quesnelia (qcp1) for baseline numbers and Eureka Quesnelia (qep1) environments and compare results 
  3. Run test with 1, 20 virtual users on non-Eureka and Eureka environments. 
  4. Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-993

Summary

  • For a single user, performance is better in the Eureka environment. However, when scaled to 20 vusers, there is a performance degradation of approximately 7.8%

  • Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 vusers - non-Eureka. 
  • Service CPU utilization was up to 14% for mod-search, for all other services CPU did not exceed 6%.

  • Memory utilization was stable and no memory leaks or OOM issues were observed.

  • RDS CPU utilization was about up to 10% for all tests.

  • Non-Eureka environment uses 50 fewer database connections than Eureka.
  • Open search CPU utilization was up to 100% with 20 user tests.

Recommendations & Jiras

Errors during 20 vusers - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

  • Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
  • For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
    (8 VCPUs * 3) / 2 + 1 = 13 operations
  • For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
    4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

60 minutes tests

Test#Test nameresultsComments
11 vuser - non-EurekaSuccess0 errors
220 vusers - non-EurekaCompleted with errors3224 errors out of 199179 API calls
ERROR: "Too Many Requests"
320 vusers - non-Eureka retestCompleted with errors41894 errors out of 252249 API calls
ERROR: "Too Many Requests"
41 vuser - EurekaSuccess0 errors
520 vusers - EurekaSuccess0 errors

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.


1 vuser - non-Eureka(avg)1 vuser - Eureka (avg)%20 vusers - non-Eureka(avg)20 vusers - Eureka (avg)%
Total requests processed in 60 minutes test run19657208115.9%199179 (3224 errors)183864-7.6%
Average throughput5.44 ops/s5.77 ops/s6%
55.2 ops/s
50.9 ops/s-7.8%
Average response time194185-4.6%3653824.7%



1 vuser - non-Eureka1 vuser - Eureka
20 vusers - non-Eureka20 vusers - Eureka
transactionNumber Of SamplesAverageNumber Of SamplesAverage%Number Of SamplesAverageNumber Of SamplesAverage%
MSF_GET /search/authorities keyword <> random sentence393207416200-3.3816439435403682477-11.6667
MSF_GET /search/authorities keyword <> randomword393202417194-3.960441045163684462-10.4651
MSF_GET /search/authorities keyword = *random sentence*39312244161142-6.699353933212036821545-27.1226
MSF_GET /search/authorities keyword = random sentence3933041623-23.33333937953682245157.8947
MSF_GET /search/authorities keyword = randomword3934441732-27.27274246127368423484.25197
MSF_GET /search/authorities keyword == random sentence3932941622-24.13793949923683225144.5652
MSF_GET /search/authorities keyword == randomword3934341731-27.9074041138368324476.81159
MSF_GET /search/authorities keyword all *randomword*393891417821-7.856343997162636831191-26.7528
MSF_GET /search/authorities keyword all random sentence3933241725-21.87539791043683243133.6538
MSF_GET /search/authorities keyword all randomword3946741754-19.4035481141368626285.8156
MSF_GET /search/authorities keyword any random sentence3939941686-13.13133958226368327220.35398
MSF_GET /search/authorities keyword any randomword3945541743-21.81824618129368523279.84496
MSF_GET /search/instances: contributors <> 2random words393158416152-3.79747387737936794087.651715
MSF_GET /search/instances: contributors = *randomword393198416192-3.0303388942936804412.797203
MSF_GET /search/instances: contributors = randomword*3935741646-19.298238921013680227124.7525
MSF_GET /search/instances: contributors == 2random words3932341617-26.0873880783679217178.2051
MSF_GET /search/instances: contributors all 2random words3932341616-30.43483876853677231171.7647
MSF_GET /search/instances: contributors all randomword3935841647-18.965538961093680232112.844
MSF_GET /search/instances: contributors any 2random words3936541651-21.538538851153680244112.1739
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang7864083235-12.57747987354233137.7551
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")786119832113-5.042027743273735034626.73993
MSF_GET /search/instances: languages == lang786110832104-5.454557743260734934432.30769
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"786123832114-7.317077744287735035322.99652
MSF_GET /search/instances: subjects <> 1randomword3937041659-15.714338671263670338168.254
MSF_GET /search/instances: subjects = (randomword OR randomword)3936741652-22.38813905129368224791.47287
MSF_GET /search/instances: subjects = *1randomword3933144163181.27388538686453672601-6.82171
MSF_GET /search/instances: subjects = 1randomword*3937041658-17.142938691133672235107.9646
MSF_GET /search/instances: subjects == 1randomword3934041632-203868123367023288.61789
MSF_GET /search/instances: subjects all 1randomword3934741637-21.27663869124367224194.35484
MSF_GET /search/instances: subjects all randomword AND title all randomword3933641627-253930983682237141.8367
MSF_GET /search/instances: subjects all randomword NOT title all randomword3938141667-17.2843900125368123588
MSF_GET /search/instances: subjects all randomword OR title all randomword3939641683-13.54173913165368226158.18182
MSF_GET /search/instances: subjects any 2random words3936341652-17.46033868133367025591.72932
MSF_GET /search/instances: title <> 3 word sentence3931674161670386340336704152.977667
MSF_GET /search/instances: title == 3 word sentence3933641629-19.44443866933670237154.8387
MSF_GET /search/instances: title all *1randomword393340341634651.8219223859547136703984-27.1797
MSF_GET /search/instances: title all 1randomword*3939441678-17.02133860144367026483.33333
MSF_GET /search/instances: title all 3 word sentence3934041638-538671013670236133.6634
MSF_GET /search/instances: title any 3 word sentence393408416395-3.1862738677353670653-11.1565
Total16901194.108717895185.1522-4.61418170397365.1304158131382.15224.661824

Test Details

1 vuser - non-Eureka

transactionNumber Of SamplesAverageMedianpct90pct95pct99MinMax
MSF_GET /search/authorities keyword <> random sentence393207206241252266171266
MSF_GET /search/authorities keyword <> randomword393202202241250311165311
MSF_GET /search/authorities keyword = *random sentence*393122412201468148115099211509
MSF_GET /search/authorities keyword = random sentence3933030395311522116
MSF_GET /search/authorities keyword = randomword39344449113420119201
MSF_GET /search/authorities keyword == random sentence39329284046642165
MSF_GET /search/authorities keyword == randomword393434411213617519175
MSF_GET /search/authorities keyword all *randomword*3938918951331134313825001382
MSF_GET /search/authorities keyword all random sentence3933230436519623196
MSF_GET /search/authorities keyword all randomword394676417333238319383
MSF_GET /search/authorities keyword any random sentence393998321830531023310
MSF_GET /search/authorities keyword any randomword394555514015224418244
MSF_GET /search/instances: contributors <> 2random words393158158183188225131225
MSF_GET /search/instances: contributors = *randomword393198189394646668157668
MSF_GET /search/instances: contributors = randomword*393574821022126317264
MSF_GET /search/instances: contributors == 2random words3932323345114416145
MSF_GET /search/instances: contributors all 2random words3932323324410317103
MSF_GET /search/instances: contributors all randomword393585815922539617398
MSF_GET /search/instances: contributors any 2random words393656613418743418437
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang786402913125729422294
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")78611911926427131941319
MSF_GET /search/instances: languages == lang78611011024825229440294
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"78612312126227033141331
MSF_GET /search/instances: subjects <> 1randomword393706911515922818229
MSF_GET /search/instances: subjects = (randomword OR randomword)393676711819256519565
MSF_GET /search/instances: subjects = *1randomword393314309475543546259546
MSF_GET /search/instances: subjects = 1randomword*393706320830932217322
MSF_GET /search/instances: subjects == 1randomword3934040739425816259
MSF_GET /search/instances: subjects all 1randomword39347469112444218444
MSF_GET /search/instances: subjects all randomword AND title all randomword39336337612413619136
MSF_GET /search/instances: subjects all randomword NOT title all randomword393817716230240719409
MSF_GET /search/instances: subjects all randomword OR title all randomword393969619929867030673
MSF_GET /search/instances: subjects any 2random words393636012522838019381
MSF_GET /search/instances: title <> 3 word sentence393167167197207218138218
MSF_GET /search/instances: title == 3 word sentence393362913420021821218
MSF_GET /search/instances: title all *1randomword3933403340146027838927322769273
MSF_GET /search/instances: title all 1randomword*393948723027072222727
MSF_GET /search/instances: title all 3 word sentence393403113515318620186
MSF_GET /search/instances: title any 3 word sentence393408384149816111820391823
MSF_TC: mod search by auth query fs090000003932923289535523735377820403778
MSF_TC: mod search by boolean query fs0900000039328027455061610121121014
MSF_TC: mod search by contributors query fs09000000393583567998114916873921696
MSF_TC: mod search by filter query fs09000000393396377886106610801501080
MSF_TC: mod search by keyword query fs09000000393394378864974990151990
MSF_TC: mod search by subject query fs0900000039360358790394415783651583
MSF_TC: mod search by title query fs09000000393414940845985893010027261710027

Image Added

1 vuser - Eureka

transactionNumber Of SamplesAverageMedianpct90pct95pct99MinMax
MSF_GET /search/authorities keyword <> random sentence416200199233240276167276
MSF_GET /search/authorities keyword <> randomword417194192227268293161293
MSF_GET /search/authorities keyword = *random sentence*416114211441352138114129131412
MSF_GET /search/authorities keyword = random sentence41623222684981898
MSF_GET /search/authorities keyword = randomword41732317910412714127
MSF_GET /search/authorities keyword == random sentence41622212679921892
MSF_GET /search/authorities keyword == randomword4173129669017414174
MSF_GET /search/authorities keyword all *randomword*4178218241216123613454921346
MSF_GET /search/authorities keyword all random sentence41725233115118119181
MSF_GET /search/authorities keyword all randomword417544815223454415548
MSF_GET /search/authorities keyword any random sentence416866718256358118581
MSF_GET /search/authorities keyword any randomword417434113422924914249
MSF_GET /search/instances: contributors <> 2random words416152151173184195129195
MSF_GET /search/instances: contributors = *randomword416192181271670676158676
MSF_GET /search/instances: contributors = randomword*416463620324537912379
MSF_GET /search/instances: contributors == 2random words4161716345822412225
MSF_GET /search/instances: contributors all 2random words41616151923411341
MSF_GET /search/instances: contributors all randomword416474413222733612338
MSF_GET /search/instances: contributors any 2random words416515111017740713410
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang832352217837251017510
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")83211311125937337538375
MSF_GET /search/instances: languages == lang83210410324224725536255
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"83211411325225931437314
MSF_GET /search/instances: subjects <> 1randomword416595512115118213183
MSF_GET /search/instances: subjects = (randomword OR randomword)41652519812130914310
MSF_GET /search/instances: subjects = *1randomword416318306481843870264870
MSF_GET /search/instances: subjects = 1randomword*416584723926953812538
MSF_GET /search/instances: subjects == 1randomword4163232567224312245
MSF_GET /search/instances: subjects all 1randomword41637357811423913240
MSF_GET /search/instances: subjects all randomword AND title all randomword41627228710814114141
MSF_GET /search/instances: subjects all randomword NOT title all randomword416675716426127715278
MSF_GET /search/instances: subjects all randomword OR title all randomword416837619739968622690
MSF_GET /search/instances: subjects any 2random words416524710218236114362
MSF_GET /search/instances: title <> 3 word sentence416167163204410448136448
MSF_GET /search/instances: title == 3 word sentence41629228824125816258
MSF_GET /search/instances: title all *1randomword4163465340246967599765022617650
MSF_GET /search/instances: title all 1randomword*416786627236370116707
MSF_GET /search/instances: title all 3 word sentence416382315525126317263
MSF_GET /search/instances: title any 3 word sentence416395373153416841792321795
MSF_TC: mod search by auth query fs090000004162673263331793948413720214137
MSF_TC: mod search by boolean query fs090000004162292074957691002891005
MSF_TC: mod search by contributors query fs09000000416521497807131115663611575
MSF_TC: mod search by filter query fs09000000416375348854130313571331357
MSF_TC: mod search by keyword query fs09000000416369348820128913361331336
MSF_TC: mod search by subject query fs09000000416555525933138314273531427
MSF_TC: mod search by title query fs090000004164172403463799095920925589209

Image Added

20 vusers - non-Eureka

transactionNumberOf SamplesAverageMedianpct90pct95pct99MinMaxError Count
MSF_GET /search/authorities keyword <> random sentence3943540553982131728532828685
MSF_GET /search/authorities keyword <> randomword41045165329341283317027317761
MSF_GET /search/authorities keyword = *random sentence*3933212021662965324945373745392
MSF_GET /search/authorities keyword = random sentence3937959628351314992015104
MSF_GET /search/authorities keyword = randomword42461271304226662186172199142
MSF_GET /search/authorities keyword == random sentence3949929528548514422114526
MSF_GET /search/authorities keyword == randomword4041138140439844204418204943
MSF_GET /search/authorities keyword all *randomword*39971626166825932876413524414817
MSF_GET /search/authorities keyword all random sentence3979104106338565216722217321
MSF_GET /search/authorities keyword all randomword54811411464667602589162606863
MSF_GET /search/authorities keyword any random sentence395822623362095326032226188
MSF_GET /search/authorities keyword any randomword46181291324267582308182324372
MSF_GET /search/instances: contributors <> 2random words387737938967795320987521081
MSF_GET /search/instances: contributors = *randomword3889429439760109920512520621
MSF_GET /search/instances: contributors = randomword*389210110434366618411518523
MSF_GET /search/instances: contributors == 2random words3880787926847515801515922
MSF_GET /search/instances: contributors all 2random words3876858728951118061618162
MSF_GET /search/instances: contributors all randomword389610911137661224661524814
MSF_GET /search/instances: contributors any 2random words388511511737863519421719514
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang77479810030553023572023743
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")77432732786999632421402424
MSF_GET /search/instances: languages == lang774326026567791921864021901
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"77442872927389862456412470
MSF_GET /search/instances: subjects <> 1randomword38671261213887322354172358
MSF_GET /search/instances: subjects = (randomword OR randomword)390512912942266326171826315
MSF_GET /search/instances: subjects = *1randomword38686456591014140531262613133
MSF_GET /search/instances: subjects = 1randomword*386911311436164221841621921
MSF_GET /search/instances: subjects == 1randomword386812312441070721891621951
MSF_GET /search/instances: subjects all 1randomword38691241273926072507162519
MSF_GET /search/instances: subjects all randomword AND title all randomword39309899326590219517221117
MSF_GET /search/instances: subjects all randomword NOT title all randomword390012512737461519711819804
MSF_GET /search/instances: subjects all randomword OR title all randomword391316516946576125902726048
MSF_GET /search/instances: subjects any 2random words38681331383996632313192325
MSF_GET /search/instances: title <> 3 word sentence3863403411737102225823025963
MSF_GET /search/instances: title == 3 word sentence3866939330154116971817053
MSF_GET /search/instances: title all *1randomword385954715549814586991024026102414
MSF_GET /search/instances: title all 1randomword*386014414741065422622022781
MSF_GET /search/instances: title all 3 word sentence38671011023365351924191932
MSF_GET /search/instances: title any 3 word sentence3867735741271330934191344203
MSF_TC: mod search by auth query fs09000000547557246037762582081074524107721544
MSF_TC: mod search by boolean query fs09000000393051553611991765357227358734
MSF_TC: mod search by contributors query fs0900000038911295133921942721473530475017
MSF_TC: mod search by filter query fs09000000387491794421082543452214745401
MSF_TC: mod search by keyword query fs0900000038729139372115248545163545303
MSF_TC: mod search by subject query fs0900000038691262129121772853562737956402
MSF_TC: mod search by title query fs090000003866693970721020111284141603551417111

Image Added

20 vusers - Eureka

transactionNumber Of SamplesAverageMedianpct90pct95pct99MinMax
MSF_GET /search/authorities keyword <> random sentence3682477487862114822141752400
MSF_GET /search/authorities keyword <> randomword3684462472850110824231662848
MSF_GET /search/authorities keyword = *random sentence*3682154515552153254843329414350
MSF_GET /search/authorities keyword = random sentence36822452526539932623192635
MSF_GET /search/authorities keyword = randomword36842342366089733428163451
MSF_GET /search/authorities keyword == random sentence36832252335918851783201792
MSF_GET /search/authorities keyword == randomword36832442526259653221163239
MSF_GET /search/authorities keyword all *randomword*3683119112061816211741185184139
MSF_GET /search/authorities keyword all random sentence36832432516319363322213341
MSF_GET /search/authorities keyword all randomword368626226666010352643172652
MSF_GET /search/authorities keyword any random sentence36832722796039022910202920
MSF_GET /search/authorities keyword any randomword36852322375929473309163332
MSF_GET /search/instances: contributors <> 2random words3679408418786107326851332702
MSF_GET /search/instances: contributors = *randomword3680441451844106429581622969
MSF_GET /search/instances: contributors = randomword*36802272345618823076143094
MSF_GET /search/instances: contributors == 2random words36792172265949022062142073
MSF_GET /search/instances: contributors all 2random words36772312396139523320143338
MSF_GET /search/instances: contributors all randomword36802322395929733142143157
MSF_GET /search/instances: contributors any 2random words36802442526229842020152030
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang73542332376149013484193511
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")735034635677210843477393501
MSF_GET /search/instances: languages == lang734934435079711143435383458
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"735035336077510853426393449
MSF_GET /search/instances: subjects <> 1randomword367033825592611772866142880
MSF_GET /search/instances: subjects = (randomword OR randomword)36822472566099182879162896
MSF_GET /search/instances: subjects = *1randomword36726016101032136529392682940
MSF_GET /search/instances: subjects = 1randomword*36722352415898773489143509
MSF_GET /search/instances: subjects == 1randomword367023223859510042564152573
MSF_GET /search/instances: subjects all 1randomword36722412486358983062163082
MSF_GET /search/instances: subjects all randomword AND title all randomword36822372436209762919132935
MSF_GET /search/instances: subjects all randomword NOT title all randomword36812352425808532493162508
MSF_GET /search/instances: subjects all randomword OR title all randomword36822612706379982581262582
MSF_GET /search/instances: subjects any 2random words367025526464010312757152759
MSF_GET /search/instances: title <> 3 word sentence3670415424770108722841402295
MSF_GET /search/instances: title == 3 word sentence367023724463410732361182367
MSF_GET /search/instances: title all *1randomword36703984398857546226745122607453
MSF_GET /search/instances: title all 1randomword*36702642736789772829172839
MSF_GET /search/instances: title all 3 word sentence36702362436288913082183091
MSF_GET /search/instances: title any 3 word sentence3670653642197424835288365311
MSF_TC: mod search by auth query fs090000003682562757507399820610533221710551
MSF_TC: mod search by boolean query fs09000000368198010151870234142711074272
MSF_TC: mod search by contributors query fs090000003677199820693160375668814236898
MSF_TC: mod search by filter query fs090000003677126713102360281250261445043
MSF_TC: mod search by keyword query fs090000003672127313152334278564161426444
MSF_TC: mod search by subject query fs090000003670190018813226363759163915920
MSF_TC: mod search by title query fs090000003670578758518198911611512262111530

Image Added

Resource Utilization

Memory Utilization

1 vuser - non-Eureka

Image Added

1 vuser - Eureka

Image Added

20 vusers - non-Eureka

Image Added

20 vusers - Eureka

Image Added

Service CPU Utilization

1 vuser - non-Eureka

Image Added

1 vuser - Eureka

Image Added

20 vusers - non-Eureka

Image Added

20 vusers - Eureka

Image Added

RDS CPU Utilization

1 vuser - non-Eureka

Image Added

1 vuser - Eureka

Image Added

20 vusers - non-Eureka

Image Added

20 vusers - Eureka

Image Added

RDS Connections count

1 vuser - non-Eureka

Image Added

1 vuser - Eureka

Image Added

20 vusers - non-Eureka

Image Added

20 vusers - Eureka

Image Added

OpenSearch metrics

Image Added

Image Added


Image Added

Image Added

Appendix

Infrastructure

PTF environment qecp1
  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 1 instance of db.r6.xlarge database instance: Writer instance
  • MSK tenant [ kafka configurations]
    • 4 kafka.m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


DB table records size:

  • fs09000000
    • instances -            24493326
    • items -                  25662862
    • holding records - 24459683

Files

Modules

Expand
titleAll modules


ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage3mod-remote-storage:3.2.024920447203960512512
mod-remote-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-ncip3mod-ncip:1.14.521024896076888128
mod-ncip - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-finance-storage3mod-finance-storage:8.6.121024896070088128
mod-finance-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-agreements3mod-agreements:7.0.62159214880000
mod-agreements - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-ebsconet3mod-ebsconet:2.2.02124810240700128256
mod-ebsconet - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-organizations3mod-organizations:1.9.221024896076888128
mod-organizations - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-sip22edge-sip2:3.2.521024896076888128
mod-settings3mod-settings:1.0.321024896076888128
mod-settings - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-serials-management3mod-serials-management:1.0.322480231201792384512
mod-serials-management - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-search3mod-search:3.2.6225922480014405121024
mod-search - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-import3mod-data-import:3.1.112048184401292384512
mod-data-import - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-dematic2edge-dematic:2.2.311024896076888128
mod-tags3mod-tags:2.2.021024896076888128
mod-tags - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-courses2edge-courses:1.4.321024896076888128
mod-notify3mod-notify:3.2.021024896076888128
mod-notify - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-inventory-update3mod-inventory-update:3.3.121024896076888128
mod-inventory-update - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-configuration3mod-configuration:5.10.021024896076888128
mod-configuration - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-orders-storage3mod-orders-storage:13.7.421024896070088128
mod-orders-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
folio-keycloak3folio-keycloak:25.0.31240020480204800
edge-caiasoft2edge-caiasoft:2.2.421024896076888128
mod-licenses3mod-licenses:6.0.322480231201792384512
mod-licenses - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-gobi3mod-gobi:2.8.121024896076888128
mod-gobi - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-password-validator3mod-password-validator:3.2.12144012980768384512
mod-password-validator - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-fqm-manager3mod-fqm-manager:2.0.501024896076888128
mod-fqm-manager - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-bulk-operations3mod-bulk-operations:2.0.223072260001536384512
mod-bulk-operations - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-graphql3mod-graphql:1.12.121024896076888128
mod-graphql - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-finance3mod-finance:4.9.021024896076888128
mod-finance - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-roles-keycloak3mod-roles-keycloak:1.4.521024896051288256
mod-roles-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-batch-print3mod-batch-print:1.1.021024896076888128
mod-batch-print - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-copycat3mod-copycat:1.6.021024896076888128
mod-copycat - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-lists4mod-lists:2.0.621024896076888128
mod-lists - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-entities-links3mod-entities-links:3.0.22259224800144001024
mod-entities-links - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-permissions6mod-permissions:6.5.02168415445121024384512
mod-permissions - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
pub-edge1pub-edge:2023.06.1421024896076800
mod-orders3mod-orders:12.8.822048174001024384512
mod-orders - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
kong9kong:3.7.1161446144204876888128
edge-patron2edge-patron:5.1.22102489625676888128
mod-users-keycloak3mod-users-keycloak:1.5.321024896051288256
mod-users-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-ncip2edge-ncip:1.10.121024896076888128
mgr-applications2mgr-applications:1.3.31102489651251288256
mod-users-bl3mod-users-bl:7.8.0214401152092288128
mod-users-bl - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mgr-tenants2mgr-tenants:1.3.31102489651251288256
mod-invoice3mod-invoice:5.8.2214401152092288128
mod-invoice - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-inventory-storage4mod-inventory-storage:27.1.424096369003076384512
mod-inventory-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512025600
mod-user-import3mod-user-import:3.8.021024896076888128
mod-user-import - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-sender3mod-sender:1.12.021024896076888128
mod-sender - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-oai-pmh2edge-oai-pmh:2.9.221512136010241440384512
mod-login-keycloak8mod-login-keycloak:1.4.421024896051288256
mod-login-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mgr-tenant-entitlements2mgr-tenant-entitlements:1.2.41102489651251288256
mod-data-export-worker3mod-data-export-worker:3.2.423072280002048384512
mod-data-export-worker - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-rtac4mod-rtac:3.6.021024896076888128
mod-rtac - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512025600
mod-circulation-storage4mod-circulation-storage:17.2.322880259201814384512
mod-circulation-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512025600
mod-source-record-storage4mod-source-record-storage:5.8.625600500003500384512
mod-source-record-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-scheduler3mod-scheduler:1.2.321024896051288256
mod-scheduler - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-calendar4mod-calendar:3.1.0220481536076888128
mod-calendar - Sidecar 1N/Afolio-module-sidecar:1.0.0.317N/A665512012800
mod-event-config3mod-event-config:2.7.121024896076888128
mod-event-config - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-courses3mod-courses:1.4.1021024896076888128
mod-courses - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-inventory5mod-inventory:20.2.722881259201814384512
mod-inventory - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-email3mod-email:1.17.022800255001800384512
mod-email - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-di-converter-storage3mod-di-converter-storage:2.2.221024896076888128
mod-di-converter-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-pubsub3mod-pubsub:2.14.02153614400922384512
mod-pubsub - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-circulation4mod-circulation:24.2.522880259201814384512
mod-circulation - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512025600
edge-orders2edge-orders:3.0.321024896076888128
edge-rtac2edge-rtac:2.7.321024896076888128
mod-template-engine3mod-template-engine:1.20.021024896076888128
mod-template-engine - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-users3mod-users:19.3.121024896076888128
mod-users - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-patron-blocks3mod-patron-blocks:1.10.021024896076888128
mod-patron-blocks - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-audit3mod-audit:2.9.021024896076888128
mod-audit - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-fqm2edge-fqm:2.0.221024896076888128
mod-source-record-manager3mod-source-record-manager:3.8.625600500003500384512
mod-source-record-manager - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
nginx-edge1nginx-edge:2023.06.14110248960000
mod-quick-marc3mod-quick-marc:5.1.112288217601664384512
mod-quick-marc - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-feesfines3mod-feesfines:19.1.021024896076888128
mod-feesfines - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-invoice-storage3mod-invoice-storage:5.8.221872153601024384512
mod-invoice-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-service-interaction3mod-service-interaction:4.0.222048184401290384512
mod-service-interaction - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-export11mod-data-export:5.0.412592248001440881024
mod-data-export - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-patron3mod-patron:6.1.021024896076888128
mod-patron - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-oai-pmh3mod-oai-pmh:3.13.224096369003076384512
mod-oai-pmh - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-connexion2edge-connexion:1.3.121024896076888128
mod-kb-ebsco-java3mod-kb-ebsco-java:4.0.021024896076888128
mod-kb-ebsco-java - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-notes3mod-notes:5.2.0210248960952384512
mod-notes - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-organizations-storage3mod-organizations-storage:4.7.021024896076888128
mod-organizations-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-export-spring3mod-data-export-spring:3.2.212048184401536384512
mod-data-export-spring - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-erm2edge-erm:1.2.121024896076888128


PTF -  QCP1 environment configuration

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instance, writer


    NameMemory GIBvCPUs

    db.r6g.xlarge

    32 GB4 vCPUs


  • Open Search ptf-test 
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK fse-tenant
    • brokers, kafka.m7g.xlarge brokers in 2 zones
    • Apache Kafka version 3.7.x 

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Expand
titleqcp1 modules memory and CPU parameters

Cluster Resources - qcp1-pvt


ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage9mod-remote-storage:3.2.024920447210243960512512
mod-ncip9mod-ncip:1.14.421024896076888128
mod-finance-storage9mod-finance-storage:8.6.021024896102470088128
mod-sudoc3mod-sudoc:1.021024896076888128
mod-agreements9mod-agreements:7.0.02159214880000
mod-ebsconet9mod-ebsconet:2.2.0212481024128700128256
mod-organizations9mod-organizations:1.9.021024896076888128
mod-consortia7mod-consortia:1.1.025136477610244416384512
edge-sip27edge-sip2:3.2.0-SNAPSHOT.20921024896076888128
mod-serials-management9mod-serials-management:1.0.022480231201792384512
mod-settings9mod-settings:1.0.32102489620076888128
mod-data-import12mod-data-import:3.1.01204818442561292384512
edge-dematic9edge-dematic:2.2.011024896076888128
mod-search9mod-search:3.2.0225922480204814405121024
mod-inn-reach7mod-inn-reach:3.2.0-SNAPSHOT.86236003240102428805121024
mod-tags9mod-tags:2.2.021024896076888128
edge-courses9edge-courses:1.4.021024896076888128
mod-authtoken13mod-authtoken:2.15.121440115251292288128
mod-inventory-update9mod-inventory-update:3.3.021024896076888128
mod-notify9mod-notify:3.2.021024896076888128
mod-configuration9mod-configuration:5.10.021024896076888128
mod-orders-storage9mod-orders-storage:13.7.02102489651270088128
edge-caiasoft9edge-caiasoft:2.2.021024896076888128
mod-login-saml9mod-login-saml:2.8.021024896076888128
mod-erm-usage-harvester9mod-erm-usage-harvester:4.5.021024896076888128
mod-licenses9mod-licenses:6.0.02248023125121792384512
mod-gobi9mod-gobi:2.8.021024896076888128
mod-password-validator9mod-password-validator:3.2.02144012980768384512
mod-bulk-operations9mod-bulk-operations:2.0.023072260001536384512
mod-fqm-manager9mod-fqm-manager:2.0.121024896076888128
edge-dcb9edge-dcb:1.1.021024896076888128
mod-graphql10mod-graphql:1.12.121024896076888128
mod-finance9mod-finance:4.9.021024896076888128
mod-erm-usage9mod-erm-usage:4.7.022800255010241800384512
mod-batch-print10mod-batch-print:1.1.021024896076888128
mod-copycat9mod-copycat:1.6.021024896076888128
mod-lists9mod-lists:2.0.021024896076888128
mod-entities-links10mod-entities-links:3.0.0225922480400144001024
mod-permissions16mod-permissions:6.5.02168415445121024384512
pub-edge5pub-edge:2023.06.1421024896276800
mod-orders9mod-orders:12.8.022048174010241024384512
edge-patron10edge-patron:5.1.02102489625676888128
edge-ncip9edge-ncip:1.9.221024896076888128
mod-marc-migrations7mod-marc-migrations:1.0.0-SNAPSHOT.521024896076888128
edge-inn-reach7edge-inn-reach:3.1.1-SNAPSHOT.4521024896076888128
mod-users-bl9mod-users-bl:7.7.021440115251292288128
mod-oa8mod-oa:2.1.0-SNAPSHOT.6221024896076888128
mod-inventory-storage12mod-inventory-storage:27.1.024096369020483076384512
mod-invoice11mod-invoice:5.8.121440115251292288128
mod-user-import9mod-user-import:3.8.021024896076888128
mod-sender10mod-sender:1.12.021024896076888128
edge-oai-pmh9edge-oai-pmh:2.9.021512136010241440384512
mod-data-export-worker9mod-data-export-worker:3.2.123072280010242048384512
mod-rtac10mod-rtac:3.6.021024896076888128
mod-circulation-storage10mod-circulation-storage:17.2.022880259215361814384512
mod-calendar9mod-calendar:3.1.021024896076888128
mod-source-record-storage21mod-source-record-storage:5.8.725600500020483500384512
mod-event-config9mod-event-config:2.7.021024896076888128
mod-courses9mod-courses:1.4.1021024896076888128
mod-circulation-item9mod-circulation-item:1.0.021024896076888128
mod-inventory11mod-inventory:20.2.022880259210241814384512
mod-email9mod-email:1.17.02280025505121800384512
mod-pubsub9mod-pubsub:2.13.02153614401024922384512
mod-circulation10mod-circulation:24.2.022880259215361814384512
mod-di-converter-storage9mod-di-converter-storage:2.2.021024896076888128
edge-rtac9edge-rtac:2.7.121024896076888128
edge-orders9edge-orders:3.0.021024896076888128
mod-users10mod-users:19.3.121024896076888128
mod-template-engine9mod-template-engine:1.20.021024896076888128
mod-patron-blocks9mod-patron-blocks:1.10.021024896102476888128
mod-audit9mod-audit:2.9.021024896076888128
edge-fqm9edge-fqm:2.0.021024896076888128
mod-source-record-manager10mod-source-record-manager:3.9.0-SNAPSHOT.33025600500020483500384512
nginx-edge5nginx-edge:2023.06.14210248962000
mod-quick-marc11mod-quick-marc:5.1.012288217601664384512
nginx-okapi5nginx-okapi:2023.06.14210248962000
okapi-b8okapi:5.3.03168414402922384512
mod-feesfines9mod-feesfines:19.1.021024896076888128
mod-invoice-storage9mod-invoice-storage:5.8.021872153610241024384512
mod-dcb10mod-dcb:1.1.021024896076888128
mod-service-interaction9mod-service-interaction:4.0.12204818442561290384512
mod-data-export22mod-data-export:5.0.412592248010241440881024
mod-patron10mod-patron:6.1.021024896076888128
mod-oai-pmh9mod-oai-pmh:3.13.024096369020483076384512
edge-connexion9edge-connexion:1.2.021024896076888128
mod-kb-ebsco-java9mod-kb-ebsco-java:4.0.021024896076888128
mod-notes9mod-notes:5.2.0210248960952384512
mod-data-export-spring9mod-data-export-spring:3.2.01204818442561536384512
mod-organizations-storage9mod-organizations-storage:4.7.021024896076888128
mod-login9mod-login:7.11.02144012981024768384512
pub-okapi5pub-okapi:2023.06.1421024896276800
mod-eusage-reports9mod-eusage-reports:2.1.121024896076888128




Additional information

environments: qcp1-non-Eureka, qecp1-Eureka

Ran Run mod-search load testing for the following variant of queries:

...