Overview

1. In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - PERF-225 - Getting issue details... STATUS

We tested with 1, 20 virtual users against 10 million Inventory records. 

       Test plan:

1. Run reindex for tenant A.
2. Run test with n users for tenant B

Backend:

• mod-search-1.6.4
• mod-inventory-18.1.4
• mod-inventory-storage-23.0.2
• okapi-4.13.0

Frontend:

• folio_inventory-9.0.1

Environment:

• 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
• 80 FOLIO back-end modules deployed in 166 ECS services
• 3 okapi ECS services
• 7 m5.xlarge  EC2 instances
• 1  writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

• authority key search

• boolean search

• contributors search

• filter search

• keyword search

• subjects search
• title search
• wildcard search
• wildcard prefix search

mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

*Note that "Regular" queries are queries that don't have the wild card character ( * ).

Show total requests processed increases as we increase the number of users concurrently.

Comparison with Kiwi

Test Runs

1 User without any background reindexing

• runtime 60 minutes

• Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

• Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.

• Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

• Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649439816854&to=1649444031072

CPU and Memory Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

mod-search memory is stable around 40% throughtout the test run

1 User with background reindexing

• runtime 60 minutes

Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18% 

• Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.

• Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing

• Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing

http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850

CPU and Memory Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.

mod-search memory is stable at around 65% throughout the test run and also during background reindexing

20 Users without any background reindexing

• runtime 60 minutes