PTF - mod search (Lotus)


















Overview

  1. In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - PERF-225 - Getting issue details... STATUS

We tested with 1, 20 virtual users against 10 million Inventory records. 

       Test plan:

  1. Run reindex for tenant A.
  2. Run test with n users for tenant B


Backend:

  • mod-search-1.6.4
  • mod-inventory-18.1.4
  • mod-inventory-storage-23.0.2
  • okapi-4.13.0

Frontend:

  • folio_inventory-9.0.1

Environment:

  • 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
  • 80 FOLIO back-end modules deployed in 166 ECS services
  • 3 okapi ECS services
  • 7 m5.xlarge  EC2 instances
  • writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

  • authority key search
  • boolean search
  • contributors search
  • filter search
  • keyword search
  • subjects search
  • title search
  • wildcard search
  • wildcard prefix search


mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

Query1 user - Baseline (avg)1 user - Under the reindexing load (avg)20 user - Baseline (avg)20 user - Under the reindexing load (avg)
Regular336 ms448 ms728 ms0.850 ms
Wildcard1.42 seconds1.63 seconds2.32 seconds2.71 seconds (2 failed requests for  /search/instances: title all *1randomword)

*Note that "Regular" queries are queries that don't have the wild card character ( * ).

Show total requests processed increases as we increase the number of users concurrently.


1 user - Baseline (avg) ms1 user - Under the load (avg) ms20 user - Under the load (avg) ms20 user - Under the load (avg) ms
Total requests processed in 60 minutes test run1071580516924459420

Comparison with Kiwi


1 user1 user reindexing20 users20 users reindexing

Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)Kiwi(ms)Lotus(ms)
Regular64833656944831907287009850
Wildcard queries660268561159418569811160832
Wildcard prefix queries134411740115692657174243624220094212



1 user1 user reindexing20 users20 users reindexing

PercentageTimes better than KiwiPercentageTimes better than KiwiPercentageTimes better than KiwiPercentageTimes better than Kiwi
Regular Delta48.15%121.27%177.18%487.87%8
Wildcard queries Delta59.39%271.66%383.32%592.54%13
Wildcard prefix queries Delta87.05%777.03%479.20%480.86%5

CPU
MemoryNumber of Tasks

KiwiLotusKiwiLotusKiwiLotus
mod-search256256536144021
okapi25612853654433
mod-authtoken12812836032222
mod-permissions12812853653722
Elastic SearchNo differences

Test Runs

1 User without any background reindexing

  • runtime 60 minutes


RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence2330.0650.2500.2640.2980.5200.7210.9730.3020.520
fs09000000GET /search/authorities keyword <> randomword2330.0650.2460.2610.2890.5480.8101.7710.3120.548
fs09000000GET /search/authorities keyword = *random sentence*2330.0650.5670.7100.9621.6812.2402.5820.8741.681
fs09000000GET /search/authorities keyword = random sentence2330.0650.0170.0210.0230.0340.0480.0740.0230.034
fs09000000GET /search/authorities keyword = randomword2330.0650.0140.0240.0560.1350.2040.4490.0440.135
fs09000000GET /search/authorities keyword == random sentence2330.0650.0170.0200.0220.0350.0550.1010.0220.033
fs09000000GET /search/authorities keyword == randomword2330.0650.0150.0220.0340.0890.1640.2410.0340.089
fs09000000GET /search/authorities keyword all *randomword*2330.0650.3020.5540.8241.1981.6752.0130.6441.198
fs09000000GET /search/authorities keyword all random sentence2330.0650.0170.0210.0250.0400.0760.2300.0250.040
fs09000000GET /search/authorities keyword all randomword2330.0650.0250.0470.0890.1580.2700.3790.0670.158
fs09000000GET /search/authorities keyword any random sentence2330.0650.0170.0920.1290.2000.3150.3540.0980.199
fs09000000GET /search/authorities keyword any randomword2330.0650.0140.0280.0620.1450.1970.2280.0460.145
fs09000000GET /search/instances: contributors <> 2random words2330.0650.2550.2740.2960.4220.7521.5990.3020.420
fs09000000GET /search/instances: contributors = *randomword2330.0650.4490.4810.5700.9431.3571.7900.5620.942
fs09000000GET /search/instances: contributors = randomword*2330.0650.0130.0410.1600.2190.2420.2760.0840.218
fs09000000GET /search/instances: contributors == 2random words2330.0650.0130.0200.0260.0400.0690.0780.0230.039
fs09000000GET /search/instances: contributors all 2random words2330.0650.0140.0170.0240.0380.1360.2110.0230.038
fs09000000GET /search/instances: contributors all randomword2330.0650.0140.0890.1790.2400.3640.4410.1120.238
fs09000000GET /search/instances: contributors any 2random words2330.0650.0170.1520.1960.2400.3900.6220.1390.239
fs09000000GET /search/instances: keyword <> 1word2330.0650.2580.2820.3240.4890.7091.2190.3230.488
fs09000000GET /search/instances: keyword <> random 2words2330.0650.2590.2810.3300.5700.7250.8030.3260.569
fs09000000GET /search/instances: keyword = 1word*2330.0650.0180.1700.2070.2740.4560.5910.1590.273
fs09000000GET /search/instances: keyword = random 2words2330.0650.0160.0510.0820.1530.2990.3130.0630.153
fs09000000GET /search/instances: keyword == random 2words2330.0650.0160.0280.0410.0820.1220.3700.0370.082
fs09000000GET /search/instances: keyword all random 2words2330.0650.0180.0590.0960.1980.3970.4340.0800.197
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang2330.0650.0200.0750.1210.1940.2890.4620.0930.194
fs09000000GET /search/instances: keyword any random 2words2330.0650.0490.2030.2370.3430.5991.2010.2200.342
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")2330.0650.0430.0570.1720.2370.3770.4530.1050.236
fs09000000GET /search/instances: languages == lang2330.0650.0430.0540.1480.1850.3580.4610.0940.184
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"2330.0650.0470.0700.2130.2440.4940.6180.1270.243
fs09000000GET /search/instances: subjects <> 1randomword2330.0650.0150.1830.2190.2590.3570.6970.1690.258
fs09000000GET /search/instances: subjects = (randomword OR randomword)2330.0650.0230.1730.2160.2580.4070.4530.1670.257
fs09000000GET /search/instances: subjects = *1randomword2330.0650.6470.7430.8721.3272.3032.9160.8501.325
fs09000000GET /search/instances: subjects = 1randomword*2330.0650.0130.1160.1580.2100.2700.2890.1040.209
fs09000000GET /search/instances: subjects == 1randomword2330.0650.0140.0540.1090.2010.2450.3900.0810.199
fs09000000GET /search/instances: subjects all 1randomword2330.0650.0150.0720.1600.2290.3040.3530.0990.227
fs09000000GET /search/instances: subjects all randomword AND title all randomword2330.0650.0150.0490.0730.1300.2450.3750.0590.130
fs09000000GET /search/instances: subjects all randomword NOT title all randomword2330.0650.0150.1870.2360.3340.5140.7450.1820.333
fs09000000GET /search/instances: subjects all randomword OR title all randomword2330.0650.0260.2130.2560.3250.4610.5280.2140.324
fs09000000GET /search/instances: subjects any 2random words2330.0650.0160.1320.1760.2450.4590.8020.1410.243
fs09000000GET /search/instances: title <> 3 word sentence2320.0650.2600.2820.3140.4990.6920.7510.3200.498
fs09000000GET /search/instances: title == 3 word sentence2320.0650.0170.0310.0500.1100.1940.2300.0430.110
fs09000000GET /search/instances: title all *1randomword2320.0654.6025.8118.43811.23414.75015.8336.88611.234
fs09000000GET /search/instances: title all 1randomword*2320.0650.0170.0850.1740.2360.3800.6220.1170.234
fs09000000GET /search/instances: title all 3 word sentence2330.0650.0200.0660.1150.1880.2640.2980.0840.188
fs09000000GET /search/instances: title any 3 word sentence2330.0650.1160.3130.6901.7472.4292.8330.5851.745



Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.0840.0%
fs09000000GET /search/instances: keyword = 1word*0.1590.0%
fs09000000GET /search/instances: subjects = 1randomword*0.1040.0%
fs09000000GET /search/instances: title all 1randomword*0.1170.0%

0.2676
Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*0.8740.0%
fs09000000GET /search/authorities keyword all *randomword*0.6440.0%
fs09000000GET /search/instances: contributors = *randomword0.5620.0%
fs09000000GET /search/authorities keyword all *randomword*0.6440.0%
fs09000000GET /search/instances: subjects = *1randomword0.850.0%
fs09000000GET /search/instances: title all *1randomword6.8860.0%

1.74
  • Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

  • Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.


  • Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

CPU and Memory Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

mod-search memory is stable around 40% throughtout the test run


1 User with background reindexing

  • runtime 60 minutes


Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18% 

RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence1750.0490.0880.2210.3390.5570.6851.1040.2630.557
fs09000000GET /search/authorities keyword <> randomword1750.0490.0870.1580.2890.5050.5921.0690.2160.505
fs09000000GET /search/authorities keyword = *random sentence*1750.0490.5700.9171.2031.7082.3373.7681.0131.708
fs09000000GET /search/authorities keyword = random sentence1750.0490.0170.0210.0280.0460.0670.1060.0260.046
fs09000000GET /search/authorities keyword = randomword1750.0490.0150.0280.0540.1400.1860.3420.0460.140
fs09000000GET /search/authorities keyword == random sentence1750.0490.0170.0210.0290.0680.1670.2100.0290.068
fs09000000GET /search/authorities keyword == randomword1750.0490.0150.0260.0400.1500.2290.2710.0420.149
fs09000000GET /search/authorities keyword all *randomword*1750.0490.3110.6570.8391.2691.4903.5620.7201.269
fs09000000GET /search/authorities keyword all random sentence1750.0490.0170.0260.0350.0730.1370.1980.0330.073
fs09000000GET /search/authorities keyword all randomword1750.0490.0250.0470.1200.1950.2790.6520.0830.195
fs09000000GET /search/authorities keyword any random sentence1750.0490.0180.1450.1900.3420.4080.4250.1490.342
fs09000000GET /search/authorities keyword any randomword1750.0490.0140.0300.0900.2080.2440.3870.0620.208
fs09000000GET /search/instances: contributors <> 2random words1750.0490.1440.3130.3870.6680.7730.9690.3390.667
fs09000000GET /search/instances: contributors = *randomword1750.0490.4600.7390.9401.4452.0642.2450.8301.444
fs09000000GET /search/instances: contributors = randomword*1750.0490.0140.0770.2130.3130.3740.4070.1190.312
fs09000000GET /search/instances: contributors == 2random words1750.0490.0140.0270.0360.0610.1090.2080.0320.061
fs09000000GET /search/instances: contributors all 2random words1750.0490.0140.0200.0280.0480.0580.0720.0250.048
fs09000000GET /search/instances: contributors all randomword1750.0490.0150.1430.2570.3250.4210.6070.1590.325
fs09000000GET /search/instances: contributors any 2random words1750.0490.0190.2050.2720.3810.4370.5930.1980.379
fs09000000GET /search/instances: keyword <> 1word1750.0490.1440.3240.4290.6530.7691.0730.3520.652
fs09000000GET /search/instances: keyword <> random 2words1750.0490.1490.3160.4180.6901.0371.1770.3650.688
fs09000000GET /search/instances: keyword = 1word*1750.0490.0180.2280.2670.3770.5870.7990.2170.376
fs09000000GET /search/instances: keyword = random 2words1750.0490.0170.0720.1230.1980.2710.4210.0870.198
fs09000000GET /search/instances: keyword == random 2words1750.0490.0170.0320.0740.1780.3030.3450.0600.178
fs09000000GET /search/instances: keyword all random 2words1750.0490.0190.1060.1560.2710.3600.4350.1250.271
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang1750.0490.0480.1690.2260.3060.4920.8070.1810.306
fs09000000GET /search/instances: keyword any random 2words1750.0490.0540.2820.3720.5600.7961.0310.3240.559
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")1750.0490.0450.0810.1830.2840.4080.4410.1250.283
fs09000000GET /search/instances: languages == lang1750.0490.0440.0810.1640.2990.3790.5190.1200.298
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"1750.0490.0490.1080.2340.4230.6150.6760.1670.422
fs09000000GET /search/instances: subjects <> 1randomword1750.0490.0160.2490.2810.3800.4480.5550.2280.379
fs09000000GET /search/instances: subjects = (randomword OR randomword)1750.0490.0240.2480.2870.3630.4640.6820.2420.362
fs09000000GET /search/instances: subjects = *1randomword1750.0490.6931.0711.3051.8492.3682.8551.1561.848
fs09000000GET /search/instances: subjects = 1randomword*1750.0490.0140.1690.2180.2830.3080.5760.1440.281
fs09000000GET /search/instances: subjects == 1randomword1750.0490.0150.0760.1590.2880.3660.5650.1110.287
fs09000000GET /search/instances: subjects all 1randomword1750.0490.0140.1120.2120.2970.3971.2430.1420.296
fs09000000GET /search/instances: subjects all randomword AND title all randomword1750.0490.0230.0880.1220.2000.2970.5110.1000.200
fs09000000GET /search/instances: subjects all randomword NOT title all randomword1750.0490.0230.2790.3410.4450.5570.7090.2650.444
fs09000000GET /search/instances: subjects all randomword OR title all randomword1750.0490.0680.3100.3580.5210.7280.8360.3260.520
fs09000000GET /search/instances: subjects any 2random words1750.0490.0200.2060.2620.3450.4950.5970.2010.344
fs09000000GET /search/instances: title <> 3 word sentence1750.0490.1520.3470.4580.7711.1311.1800.3970.770
fs09000000GET /search/instances: title == 3 word sentence1750.0490.0170.0370.1200.2040.3590.4130.0790.204
fs09000000GET /search/instances: title all *1randomword1750.0495.0068.10311.27318.09322.11422.7489.56818.093
fs09000000GET /search/instances: title all 1randomword*1750.0490.0170.1130.2450.3910.5040.6390.1590.390
fs09000000GET /search/instances: title all 3 word sentence1750.0490.0390.1330.1780.2740.3170.7770.1480.274
fs09000000GET /search/instances: title any 3 word sentence1750.0490.1750.4730.9952.8223.7204.9220.8602.821
Wildcard queriesAvg (seconds)Error
fs09000000GET /search/instances: contributors = randomword*0.1190.0%
fs09000000GET /search/instances: keyword = 1word*0.2170.0%
fs09000000GET /search/instances: subjects = 1randomword*0.1440.0%
fs09000000GET /search/instances: title all 1randomword*0.1590.0%

0.159


Wildcard prefix queriesAvg (seconds)Error
fs09000000GET /search/authorities keyword = *random sentence*1.0130.0%
fs09000000GET /search/authorities keyword all *randomword*0.7200.0%
fs09000000GET /search/instances: contributors = *randomword0.8300.0%
fs09000000GET /search/instances: subjects = *1randomword1.1560.0%
fs09000000GET /search/instances: title all *1randomword9.5680.0%

2.657
  • Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.


  • Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing


  • Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing


http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850


CPU and Memory Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.


mod-search memory is stable at around 65% throughout the test run and also during background reindexing


20 Users without any background reindexing

  • runtime 60 minutes


RequestsTotalReq/sMin50pct75pct95pct99pctMaxAvgLatency
fs09000000GET /search/authorities keyword <> random sentence15080.4200.1490.7231.0421.7812.5896.3430.8501.781
fs09000000GET /search/authorities keyword <> randomword15110.4200.1560.7131.0301.8402.8875.7960.8511.836
fs09000000GET /search/authorities keyword = *random sentence*15080.4200.8831.6712.0482.8474.6878.3261.7882.847
fs09000000GET /search/authorities keyword = random sentence15080.4200.0310.4600.7171.3762.21910.2180.5961.376
fs09000000GET /search/authorities keyword = randomword15110.4200.0260.4490.7111.4062.5946.0430.5931.404
fs09000000GET /search/authorities keyword == random sentence15080.4200.0680.4470.6931.2782.1535.4110.5581.278
fs09000000GET /search/authorities keyword == randomword15100.4200.0160.4720.7341.5612.5777.3780.6141.561
fs09000000GET /search/authorities keyword all *randomword*15100.4190.5451.3591.7432.5453.7226.9971.4682.545
fs09000000GET /search/authorities keyword all random sentence15090.4190.0530.4710.7221.3612.4915.3850.5911.361
fs09000000GET /search/authorities keyword all randomword15120.4200.0210.4860.7711.6032.6485.1970.6321.603
fs09000000GET /search/authorities keyword any random sentence15080.4200.0510.5070.7671.4642.6145.4630.6361.464
fs09000000GET /search/authorities keyword any randomword15120.4200.0190.4850.7591.5462.5594.9320.6131.546
fs09000000GET /search/instances: contributors <> 2random words15060.4200.2290.8451.2022.0953.86610.6960.9982.094
fs09000000GET /search/instances: contributors = *randomword15060.4200.6021.3601.7282.5263.4114.8991.4722.526
fs09000000GET /search/instances: contributors = randomword*15060.4200.0640.5080.7781.5482.3196.5980.6421.547
fs09000000GET /search/instances: contributors == 2random words15060.4200.0650.4590.7021.4012.3736.6190.5881.401
fs09000000GET /search/instances: contributors all 2random words15060.4200.0580.4720.7491.4242.6185.9750.6071.424
fs09000000GET /search/instances: contributors all randomword15080.4200.0420.5500.8131.5122.4606.0070.6701.512
fs09000000GET /search/instances: contributors any 2random words15060.4200.0900.5900.8601.5582.39511.6320.7261.556
fs09000000GET /search/instances: keyword <> 1word15030.4190.2190.8901.2192.0472.91810.9881.0122.046
fs09000000GET /search/instances: keyword <> random 2words15040.4190.2150.8521.1901.9242.8716.3200.9751.922
fs09000000GET /search/instances: keyword = 1word*15030.4190.0650.5980.8671.5582.5874.8830.7281.557
fs09000000GET /search/instances: keyword = random 2words15040.4190.0770.4720.7141.4362.5534.6210.6011.434
fs09000000GET /search/instances: keyword == random 2words15040.4190.0790.4890.7711.4172.2546.3590.6141.417
fs09000000GET /search/instances: keyword all random 2words15050.4190.0570.5260.7701.4532.7309.9470.6491.453
fs09000000GET /search/instances: keyword all random sentence AND languages == randomlang15060.4200.0660.4920.7401.4972.1714.6920.6191.484
fs09000000GET /search/instances: keyword any random 2words15040.4190.1880.6610.9411.5952.5207.0170.7901.594
fs09000000GET /search/instances: languages == (lang) AND items.status.name == ("Available" OR "In transit")15050.4190.1030.6120.9031.6852.5728.2750.7461.681
fs09000000GET /search/instances: languages == lang15050.4190.1060.5840.8761.5302.4307.0040.7121.528
fs09000000GET /search/instances: languages == lang AND items.status.name == "Available"15060.4190.0990.6380.9291.7272.7917.6070.7781.720
fs09000000GET /search/instances: subjects <> 1randomword14980.4180.0820.5890.8671.6653.14714.8820.744