PTF - [Quesnelia][Eureka] Searching via mod-search

Overview

  1. In this testing effort we perform load tests of mod-search on the Quesnelia release.
  2. These tests with 1, 20 virtual users were run in both Okapi Quesnelia (qcp1) environment for baseline numbers and Eureka Quesnelia (qecp1) environment with the goal of comparing Eureka's performance against Okapi FOLIO. 
  3. PERF-993 - Getting issue details... STATUS

Summary

  • For a single user, performance is better in the Eureka environment. However, when scaled to 20 vusers, there is a performance degradation of approximately 7.8%

  • It is impossible to reach an average throughput higher than 50.9 ops/s for the Eureka environment (for non-Eureka environment over 55.2 ops/s). A higher number of users up to 60 will only increase response time even if we have 4 tasks for mod-search. We observed an increase in response time from sidecar-mod-search from 282ms for 5 vusers and up to 1,2 seconds for 60 vusers. Additional information in this table.
  • Very high load for the OpenSearch cluster with more than 52 search operations per second. ERROR: "Too Many Requests" observed during the test with 20 vusers - non-Eureka. 
  • Open search CPU utilization was up to 100% with 20 user tests.
  • Service CPU utilization was up to 14% for mod-search, for all other services CPU did not exceed 6%.

  • Memory utilization was stable and no memory leaks or OOM issues were observed.

  • RDS CPU utilization was about up to 10% for all tests.

  • Non-Eureka environment uses 50 fewer database connections than Eureka. The number of connections for mod-search was the same for both environments
                    {
                        "name": "DB_MAXPOOLSIZE",
                        "value": "20"
                    },

Recommendations & Jiras

Errors during 20 vusers - non-Eureka test are due to a very high load for the OpenSearch cluster. For this test we have a throughput of about 55 ops/s and the maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error) according to our cluster configurations.

  • Use the following formula to calculate maximum active threads for search requests:int ((# of available_processors * 3) / 2) + 1.
  • For an r6g.2xlarge.search node, you can perform a maximum of 13 search operations:
    (8 VCPUs * 3) / 2 + 1 = 13 operations
  • For an OpenSearch Service cluster with four nodes, you can perform a maximum of 52 search operations:
    4 nodes * 13 = 52 operations

To avoid this error, either increase the number of OpenSearch data nodes to 5 or scale the size of the data nodes to a bigger one.

Test Runs

60 minutes tests

Test#Test nameresultsComments
11 vuser - non-EurekaSuccess0 errors
220 vusers - non-EurekaCompleted with errors3224 errors out of 199179 API calls
ERROR: "Too Many Requests"
320 vusers - non-Eureka retestCompleted with errors41894 errors out of 252249 API calls
ERROR: "Too Many Requests"
41 vuser - EurekaSuccess0 errors
520 vusers - EurekaSuccess0 errors

Results and Comparison Eureka and non-Eureka

mod-search performance is good, maximal Elastic Search throughput is 52 search operations (https://repost.aws/knowledge-center/opensearch-resolve-429-error).

Show total requests processed increase as we increase the number of users concurrently.


1 vuser - non-Eureka(avg)1 vuser - Eureka (avg)%20 vusers - non-Eureka(avg)20 vusers - Eureka (avg)%
Total requests processed in 60 minutes test run19657208115.9%199179 (3224 errors)183864-7.6%
Average throughput5.44 ops/s5.77 ops/s6%
55.2 ops/s
50.9 ops/s-7.8%
Average response time194185-4.6%3653824.7%

1 vuser - non-Eureka1 vuser - Eureka
20 vusers - non-Eureka20 vusers - Eureka
transactionNumber Of SamplesAverageNumber Of SamplesAverage%Number Of SamplesAverageNumber Of SamplesAverage%
MSF_GET /search/authorities keyword <> random sentence393207416200-3.3816439435403682477-11.6667
MSF_GET /search/authorities keyword <> randomword393202417194-3.960441045163684462-10.4651
MSF_GET /search/authorities keyword = *random sentence*39312244161142-6.699353933212036821545-27.1226
MSF_GET /search/authorities keyword = random sentence3933041623-23.33333937953682245157.8947
MSF_GET /search/authorities keyword = randomword3934441732-27.27274246127368423484.25197
MSF_GET /search/authorities keyword == random sentence3932941622-24.13793949923683225144.5652
MSF_GET /search/authorities keyword == randomword3934341731-27.9074041138368324476.81159
MSF_GET /search/authorities keyword all *randomword*393891417821-7.856343997162636831191-26.7528
MSF_GET /search/authorities keyword all random sentence3933241725-21.87539791043683243133.6538
MSF_GET /search/authorities keyword all randomword3946741754-19.4035481141368626285.8156
MSF_GET /search/authorities keyword any random sentence3939941686-13.13133958226368327220.35398
MSF_GET /search/authorities keyword any randomword3945541743-21.81824618129368523279.84496
MSF_GET /search/instances: contributors <> 2random words393158416152-3.79747387737936794087.651715
MSF_GET /search/instances: contributors = *randomword393198416192-3.0303388942936804412.797203
MSF_GET /search/instances: contributors = randomword*3935741646-19.298238921013680227124.7525
MSF_GET /search/instances: contributors == 2random words3932341617-26.0873880783679217178.2051
MSF_GET /search/instances: contributors all 2random words3932341616-30.43483876853677231171.7647
MSF_GET /search/instances: contributors all randomword3935841647-18.965538961093680232112.844
MSF_GET /search/instances: contributors any 2random words3936541651-21.538538851153680244112.1739
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang7864083235-12.57747987354233137.7551
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")786119832113-5.042027743273735034626.73993
MSF_GET /search/instances: languages == lang786110832104-5.454557743260734934432.30769
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"786123832114-7.317077744287735035322.99652
MSF_GET /search/instances: subjects <> 1randomword3937041659-15.714338671263670338168.254
MSF_GET /search/instances: subjects = (randomword OR randomword)3936741652-22.38813905129368224791.47287
MSF_GET /search/instances: subjects = *1randomword3933144163181.27388538686453672601-6.82171
MSF_GET /search/instances: subjects = 1randomword*3937041658-17.142938691133672235107.9646
MSF_GET /search/instances: subjects == 1randomword3934041632-203868123367023288.61789
MSF_GET /search/instances: subjects all 1randomword3934741637-21.27663869124367224194.35484
MSF_GET /search/instances: subjects all randomword AND title all randomword3933641627-253930983682237141.8367
MSF_GET /search/instances: subjects all randomword NOT title all randomword3938141667-17.2843900125368123588
MSF_GET /search/instances: subjects all randomword OR title all randomword3939641683-13.54173913165368226158.18182
MSF_GET /search/instances: subjects any 2random words3936341652-17.46033868133367025591.72932
MSF_GET /search/instances: title <> 3 word sentence3931674161670386340336704152.977667
MSF_GET /search/instances: title == 3 word sentence3933641629-19.44443866933670237154.8387
MSF_GET /search/instances: title all *1randomword393340341634651.8219223859547136703984-27.1797
MSF_GET /search/instances: title all 1randomword*3939441678-17.02133860144367026483.33333
MSF_GET /search/instances: title all 3 word sentence3934041638-538671013670236133.6634
MSF_GET /search/instances: title any 3 word sentence393408416395-3.1862738677353670653-11.1565
Total16901194.108717895185.1522-4.61418170397365.1304158131382.15224.661824

Test Details

1 vuser - non-Eureka

transactionNumber Of SamplesAverageMedianpct90pct95pct99MinMax
MSF_GET /search/authorities keyword <> random sentence393207206241252266171266
MSF_GET /search/authorities keyword <> randomword393202202241250311165311
MSF_GET /search/authorities keyword = *random sentence*393122412201468148115099211509
MSF_GET /search/authorities keyword = random sentence3933030395311522116
MSF_GET /search/authorities keyword = randomword39344449113420119201
MSF_GET /search/authorities keyword == random sentence39329284046642165
MSF_GET /search/authorities keyword == randomword393434411213617519175
MSF_GET /search/authorities keyword all *randomword*3938918951331134313825001382
MSF_GET /search/authorities keyword all random sentence3933230436519623196
MSF_GET /search/authorities keyword all randomword394676417333238319383
MSF_GET /search/authorities keyword any random sentence393998321830531023310
MSF_GET /search/authorities keyword any randomword394555514015224418244
MSF_GET /search/instances: contributors <> 2random words393158158183188225131225
MSF_GET /search/instances: contributors = *randomword393198189394646668157668
MSF_GET /search/instances: contributors = randomword*393574821022126317264
MSF_GET /search/instances: contributors == 2random words3932323345114416145
MSF_GET /search/instances: contributors all 2random words3932323324410317103
MSF_GET /search/instances: contributors all randomword393585815922539617398
MSF_GET /search/instances: contributors any 2random words393656613418743418437
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang786402913125729422294
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")78611911926427131941319
MSF_GET /search/instances: languages == lang78611011024825229440294
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"78612312126227033141331
MSF_GET /search/instances: subjects <> 1randomword393706911515922818229
MSF_GET /search/instances: subjects = (randomword OR randomword)393676711819256519565
MSF_GET /search/instances: subjects = *1randomword393314309475543546259546
MSF_GET /search/instances: subjects = 1randomword*393706320830932217322
MSF_GET /search/instances: subjects == 1randomword3934040739425816259
MSF_GET /search/instances: subjects all 1randomword39347469112444218444
MSF_GET /search/instances: subjects all randomword AND title all randomword39336337612413619136
MSF_GET /search/instances: subjects all randomword NOT title all randomword393817716230240719409
MSF_GET /search/instances: subjects all randomword OR title all randomword393969619929867030673
MSF_GET /search/instances: subjects any 2random words393636012522838019381
MSF_GET /search/instances: title <> 3 word sentence393167167197207218138218
MSF_GET /search/instances: title == 3 word sentence393362913420021821218
MSF_GET /search/instances: title all *1randomword3933403340146027838927322769273
MSF_GET /search/instances: title all 1randomword*393948723027072222727
MSF_GET /search/instances: title all 3 word sentence393403113515318620186
MSF_GET /search/instances: title any 3 word sentence393408384149816111820391823
MSF_TC: mod search by auth query fs090000003932923289535523735377820403778
MSF_TC: mod search by boolean query fs0900000039328027455061610121121014
MSF_TC: mod search by contributors query fs09000000393583567998114916873921696
MSF_TC: mod search by filter query fs09000000393396377886106610801501080
MSF_TC: mod search by keyword query fs09000000393394378864974990151990
MSF_TC: mod search by subject query fs0900000039360358790394415783651583
MSF_TC: mod search by title query fs09000000393414940845985893010027261710027

1 vuser - Eureka

transactionNumber Of SamplesAverageMedianpct90pct95pct99MinMax
MSF_GET /search/authorities keyword <> random sentence416200199233240276167276
MSF_GET /search/authorities keyword <> randomword417194192227268293161293
MSF_GET /search/authorities keyword = *random sentence*416114211441352138114129131412
MSF_GET /search/authorities keyword = random sentence41623222684981898
MSF_GET /search/authorities keyword = randomword41732317910412714127
MSF_GET /search/authorities keyword == random sentence41622212679921892
MSF_GET /search/authorities keyword == randomword4173129669017414174
MSF_GET /search/authorities keyword all *randomword*4178218241216123613454921346
MSF_GET /search/authorities keyword all random sentence41725233115118119181
MSF_GET /search/authorities keyword all randomword417544815223454415548
MSF_GET /search/authorities keyword any random sentence416866718256358118581
MSF_GET /search/authorities keyword any randomword417434113422924914249
MSF_GET /search/instances: contributors <> 2random words416152151173184195129195
MSF_GET /search/instances: contributors = *randomword416192181271670676158676
MSF_GET /search/instances: contributors = randomword*416463620324537912379
MSF_GET /search/instances: contributors == 2random words4161716345822412225
MSF_GET /search/instances: contributors all 2random words41616151923411341
MSF_GET /search/instances: contributors all randomword416474413222733612338
MSF_GET /search/instances: contributors any 2random words416515111017740713410
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang832352217837251017510
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")83211311125937337538375
MSF_GET /search/instances: languages == lang83210410324224725536255
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"83211411325225931437314
MSF_GET /search/instances: subjects <> 1randomword416595512115118213183
MSF_GET /search/instances: subjects = (randomword OR randomword)41652519812130914310
MSF_GET /search/instances: subjects = *1randomword416318306481843870264870
MSF_GET /search/instances: subjects = 1randomword*416584723926953812538
MSF_GET /search/instances: subjects == 1randomword4163232567224312245
MSF_GET /search/instances: subjects all 1randomword41637357811423913240
MSF_GET /search/instances: subjects all randomword AND title all randomword41627228710814114141
MSF_GET /search/instances: subjects all randomword NOT title all randomword416675716426127715278
MSF_GET /search/instances: subjects all randomword OR title all randomword416837619739968622690
MSF_GET /search/instances: subjects any 2random words416524710218236114362
MSF_GET /search/instances: title <> 3 word sentence416167163204410448136448
MSF_GET /search/instances: title == 3 word sentence41629228824125816258
MSF_GET /search/instances: title all *1randomword4163465340246967599765022617650
MSF_GET /search/instances: title all 1randomword*416786627236370116707
MSF_GET /search/instances: title all 3 word sentence416382315525126317263
MSF_GET /search/instances: title any 3 word sentence416395373153416841792321795
MSF_TC: mod search by auth query fs090000004162673263331793948413720214137
MSF_TC: mod search by boolean query fs090000004162292074957691002891005
MSF_TC: mod search by contributors query fs09000000416521497807131115663611575
MSF_TC: mod search by filter query fs09000000416375348854130313571331357
MSF_TC: mod search by keyword query fs09000000416369348820128913361331336
MSF_TC: mod search by subject query fs09000000416555525933138314273531427
MSF_TC: mod search by title query fs090000004164172403463799095920925589209

20 vusers - non-Eureka

transactionNumberOf SamplesAverageMedianpct90pct95pct99MinMaxError Count
MSF_GET /search/authorities keyword <> random sentence3943540553982131728532828685
MSF_GET /search/authorities keyword <> randomword41045165329341283317027317761
MSF_GET /search/authorities keyword = *random sentence*3933212021662965324945373745392
MSF_GET /search/authorities keyword = random sentence3937959628351314992015104
MSF_GET /search/authorities keyword = randomword42461271304226662186172199142
MSF_GET /search/authorities keyword == random sentence3949929528548514422114526
MSF_GET /search/authorities keyword == randomword4041138140439844204418204943
MSF_GET /search/authorities keyword all *randomword*39971626166825932876413524414817
MSF_GET /search/authorities keyword all random sentence3979104106338565216722217321
MSF_GET /search/authorities keyword all randomword54811411464667602589162606863
MSF_GET /search/authorities keyword any random sentence395822623362095326032226188
MSF_GET /search/authorities keyword any randomword46181291324267582308182324372
MSF_GET /search/instances: contributors <> 2random words387737938967795320987521081
MSF_GET /search/instances: contributors = *randomword3889429439760109920512520621
MSF_GET /search/instances: contributors = randomword*389210110434366618411518523
MSF_GET /search/instances: contributors == 2random words3880787926847515801515922
MSF_GET /search/instances: contributors all 2random words3876858728951118061618162
MSF_GET /search/instances: contributors all randomword389610911137661224661524814
MSF_GET /search/instances: contributors any 2random words388511511737863519421719514
MSF_GET /search/instances: keyword all random sentence AND languages == randomlang77479810030553023572023743
MSF_GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")77432732786999632421402424
MSF_GET /search/instances: languages == lang774326026567791921864021901
MSF_GET /search/instances: languages == lang AND items.status.name == "Available"77442872927389862456412470
MSF_GET /search/instances: subjects <> 1randomword38671261213887322354172358
MSF_GET /search/instances: subjects = (randomword OR randomword)390512912942266326171826315
MSF_GET /search/instances: subjects = *1randomword38686456591014140531262613133
MSF_GET /search/instances: subjects = 1randomword*386911311436164221841621921
MSF_GET /search/instances: subjects == 1randomword386812312441070721891621951
MSF_GET /search/instances: subjects all 1randomword38691241273926072507162519
MSF_GET /search/instances: subjects all randomword AND title all randomword39309899326590219517221117
MSF_GET /search/instances: subjects all randomword NOT title all randomword390012512737461519711819804
MSF_GET /search/instances: subjects all randomword OR title all randomword391316516946576125902726048
MSF_GET /search/instances: subjects any 2random words38681331383996632313192325
MSF_GET /search/instances: title <> 3 word sentence3863403411737102225823025963
MSF_GET /search/instances: title == 3 word sentence3866939330154116971817053
MSF_GET /search/instances: title all *1randomword385954715549814586991024026102414
MSF_GET /search/instances: title all 1randomword*386014414741065422622022781
MSF_GET /search/instances: title all 3 word sentence38671011023365351924191932
MSF_GET /search/instances: title any 3 word sentence3867735741271330934191344203
MSF_TC: mod search by auth query fs09000000547557246037762582081074524107721544
MSF_TC: mod search by boolean query fs09000000393051553611991765357227358734
MSF_TC: mod search by contributors query fs0900000038911295133921942721473530475017
MSF_TC: mod search by filter query fs09000000387491794421082543452214745401
MSF_TC: mod search by keyword query fs0900000038729139372115248545163545303
MSF_TC: mod search by subject query fs090000003869126212912177285