/
PTF - mod search (Lotus)

PTF - mod search (Lotus)

Jun 28, 2023

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Overview

  1. In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - https://folio-org.atlassian.net/browse/PERF-225

We tested with 1, 20 virtual users against 10 million Inventory records. 

       Test plan:

  1. Run reindex for tenant A.

  2. Run test with n users for tenant B

 

Backend:

  • mod-search-1.6.4

  • mod-inventory-18.1.4

  • mod-inventory-storage-23.0.2

  • okapi-4.13.0

Frontend:

  • folio_inventory-9.0.1

Environment:

  • 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)

  • 80 FOLIO back-end modules deployed in 166 ECS services

  • 3 okapi ECS services

  • 7 m5.xlarge  EC2 instances

  • 1  writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

  • authority key search

  • boolean search

  • contributors search

  • filter search

  • keyword search

  • subjects search

  • title search

  • wildcard search

  • wildcard prefix search

 

mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

Query

1 user - Baseline (avg)

1 user - Under the reindexing load (avg)

20 user - Baseline (avg)

20 user - Under the reindexing load (avg)

Regular

336 ms

448 ms

728 ms

0.850 ms

Wildcard

1.42 seconds

1.63 seconds

2.32 seconds

2.71 seconds (2 failed requests for  /search/instances: title all *1randomword)

*Note that "Regular" queries are queries that don't have the wild card character ( * ).

Show total requests processed increases as we increase the number of users concurrently.

 

1 user - Baseline (avg) ms

1 user - Under the load (avg) ms

20 user - Under the load (avg) ms

20 user - Under the load (avg) ms

Total requests processed in 60 minutes test run

10715

8051

69244

59420

Comparison with Kiwi

 

1 user

1 user reindexing

20 users

20 users reindexing

 

Kiwi(ms)

Lotus(ms)

Kiwi(ms)

Lotus(ms)

Kiwi(ms)

Lotus(ms)

Kiwi(ms)

Lotus(ms)

Regular

648

336

569

448

3190

728

7009

850

Wildcard queries

660

268

561

159

4185

698

11160

832

Wildcard prefix queries

13441

1740

11569

2657

17424

3624

22009

4212

 

 

1 user

1 user reindexing

20 users

20 users reindexing

 

Percentage

Times better than Kiwi

Percentage

Times better than Kiwi

Percentage

Times better than Kiwi

Percentage

Times better than Kiwi

Regular Delta

48.15%

1

21.27%

1

77.18%

4

87.87%

8

Wildcard queries Delta

59.39%

2

71.66%

3

83.32%

5

92.54%

13

Wildcard prefix queries Delta

87.05%

7

77.03%

4

79.20%

4

80.86%

5

 

CPU

 

Memory

Number of Tasks

 

Kiwi

Lotus

Kiwi

Lotus

Kiwi

Lotus

mod-search

256

256

536

1440

2

1

okapi

256

128

536

544

3

3

mod-authtoken

128

128

360

322

2

2

mod-permissions

128

128

536

537

2

2

Elastic Search

No differences

Test Runs

1 User without any background reindexing

  • runtime 60 minutes

 

Requests

Total

Req/s

Min

50pct

75pct

95pct

99pct

Max

Avg

Latency

fs09000000GET /search/authorities keyword <> random sentence

233

0.065

0.250

0.264

0.298

0.520

0.721

0.973

0.302

0.520

fs09000000GET /search/authorities keyword <> randomword

233

0.065

0.246

0.261

0.289

0.548

0.810

1.771

0.312

0.548

fs09000000GET /search/authorities keyword = *random sentence*

233

0.065

0.567

0.710

0.962

1.681

2.240

2.582

0.874

1.681

fs09000000GET /search/authorities keyword = random sentence

233

0.065

0.017

0.021

0.023

0.034

0.048

0.074

0.023

0.034

fs09000000GET /search/authorities keyword = randomword

233

0.065

0.014

0.024

0.056

0.135

0.204

0.449

0.044

0.135

fs09000000GET /search/authorities keyword == random sentence

233

0.065

0.017

0.020

0.022

0.035

0.055

0.101

0.022

0.033

fs09000000GET /search/authorities keyword == randomword

233

0.065

0.015

0.022

0.034

0.089

0.164

0.241

0.034

0.089

fs09000000GET /search/authorities keyword all *randomword*

233

0.065

0.302

0.554

0.824

1.198

1.675

2.013

0.644

1.198

fs09000000GET /search/authorities keyword all random sentence

233

0.065

0.017

0.021

0.025

0.040

0.076

0.230

0.025

0.040

fs09000000GET /search/authorities keyword all randomword

233

0.065

0.025

0.047

0.089

0.158

0.270

0.379

0.067

0.158

fs09000000GET /search/authorities keyword any random sentence

233