Overview

1. In this workflow, we load test mod-search without and with ES background reindexing across 2 tenants for the Lotus release - PERF-225 - Getting issue details... STATUS

We tested with 1, 20 virtual users against 10 million Inventory records. 

       Test plan:

1. Run reindex for tenant A.
2. Run test with n users for tenant B

Backend:

• mod-search-1.6.4
• mod-inventory-18.1.4
• mod-inventory-storage-23.0.2
• okapi-4.13.0

Frontend:

• folio_inventory-9.0.1

Environment:

• 7.2 million inventory records (7.3 Million instances, 7.8 Million holdings record, 8.9 Million items)
• 80 FOLIO back-end modules deployed in 166 ECS services
• 3 okapi ECS services
• 7 m5.xlarge  EC2 instances
• 1  writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance

High-Level Summary

Ran mod-search load testing for the following variant of queries:

• authority key search

• boolean search

• contributors search

• filter search

• keyword search

• subjects search
• title search
• wildcard search
• wildcard prefix search

mod-search performance is good without any background load but degrades gradually under background Elastic Search reindexing of Inventory instances. The background load is on the second tenant.

*Note that "Regular" queries are queries that don't have the wild card character ( * ).

Show total requests processed increases as we increase the number of users concurrently.

Comparison with Kiwi

Test Runs

1 User without any background reindexing

• runtime 60 minutes

• Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~15 seconds For all other queries it was around 2.5 seconds or less.

• Throughput

Throughput for wild card queries was slowest. Otherwise, on average for all other queries, the Request per second was good around 5 RPS.

• Response time distribution

Out of ~10K request, 686 requests had response time > 700 millisecond All these 686 requests were wildcard queries

• Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649439816854&to=1649444031072

CPU and Memory Utilization

mod-search's CPU was reasonable at around 10% throughout the test run

mod-search memory is stable around 40% throughtout the test run

1 User with background reindexing

• runtime 60 minutes

Requests per second(RPS) dropped to 4.1 with background indexing. Without any indexing, it was 5.0 which means with background indexing performance dropped by 18% 

• Response Time Over Time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword spiked to ~22 seconds For all other queries it was closer to 5 seconds or less.

• Throughput

Throughput dropped to an average of 4.1 Requests per second which means throughput dropped by 18% compared to without background indexing

• Response Time distribution

Out of ~8K request, 765 requests had response time > 700 millisecond All these 765 requests were wildcard queries. This means performance dropped with background indexing

http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649782951255&to=1649787290850

CPU and Memory Utilization

When we start reindexing mod-search linearly spikes to 90% max. Now with background reindexing running, when we start the JMeter script, the CPU stays stable at 90% throughout the test run. This means CPU performance dropped by 80% with background indexing. CPU drops back to normal after reindexing is complete.

mod-search memory is stable at around 65% throughout the test run and also during background reindexing

20 Users without any background reindexing

• runtime 60 minutes

• Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~28 seconds For all other queries it is closer to 10 seconds or less.

• Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

• Response time distribution

Out of ~69K requests, 31K requests had a response time > 700 milliseconds All these 31K requests were wildcard queries

• CPU and Memory Utilization

mod-search's CPU was reasonable at around 30% throughout the test run which is 20% more than 1 user. 

Memory is stable at around 65% throughout the test run

• Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1649944107574&to=1649948967673

20 Users with background reindexing

• runtime 60 minutes

• Response time over time

Max response time was for query: fs09000000GET /search/instances: title all *1randomword - ~30 seconds For all other queries it is closer to 10 seconds or less.

• Throughput

Throughput for wild card queries was the slowest. Otherwise, on average for all other queries, the Request per second was good around 19 RPS. It is higher because this test is run for 20 concurrent users.

• Response time distribution

Out of ~69K requests, 32K requests had a response time > 700 milliseconds All these 32K requests were wildcard queries

• CPU and Memory Utilization

CPU spikes linearly 40% as we start background reindexing. When we start the JMeter script run, the CPU continues to move upwards and reaches a max of 70%. Okapi also consumes resources but it is less than 40% which reasonable.

Memory is stable at around 65% throughout the test run and also during background reindexing 

• Reference to Grafana dashboard(link will not work after 30 days) - http://carrier-io.int.folio.ebsco.com/grafana/d/q69rYQlik/jmeter-performance?orgId=1&var-percentile=95&var-test_type=baseline&var-test=mod-search&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=700&var-db_name=jmeter&var-sampler_type=All&from=1650032755868&to=1650036953158

Longevity 24 hours test

Looking at CPU, Memory and Graphana dashboard, the longevity test for mod-search looks promising. mod-search is stable. Http request for wildcard prefix failed which is expected fs09000000GET /search/instances: title all *1randomword_GET_500

Also, not related to mod-search, there was an issue with the carrier io environment but other than the test looks good.

CPU and Memory Utilization

mod-search is around 40% which is relatively stable. CPU spiked to 40% from 30% after starting background ES reindexing.

Memory also looks stable at around 61%

JIRA tickets created as part of this investigation

1. MSEARCH-331 - Getting issue details... STATUS  
2. MSEARCH-337 - Getting issue details... STATUS

Additional information

• Different permutations of random words and sentences were generated from https://www.mit.edu/~ecprice/wordlist.10000 

• Artifact stored http://carrier-io.int.folio.ebsco.com/artifacts?q=mod-search - mod-search.zip

• Kafka topics related to mod-search

{EnvID}.{tenant}.inventory.instance
{EnvID}.{tenant}.inventory.item
{EnvID}.{tenant}.inventory.holding-record
{EnvID}.{tenant}.inventory.bound-with (in new versions of mod-search\mod-inventory)

• https://confluence.epnet.com/pages/viewpage.action?spaceKey=GOPAC&title=mod-search+performance+under+the+background+load