Overview

This document contains the results of testing Lists App refreshing multiple concurrent lists with different queries MODLISTS-58 on a single tenant with R/W split enabled (Poppy release). The goal of testing is to assess the performance of the Lists App modules (mod-lists, mod-fqm-manager) with load from across multiple concurrent users.

Ticket: https://folio-org.atlassian.net/browse/PERF-811

Summary

• All tests were finished successfully. List for query (item_effective_call_number_type_name == Library of Congress classification) AND (item_effective_location_name not in [acq admin, Administration Office]) AND (item_status == Available) AND (item_material_type == book) AND (item_updated_date > 2019-07-01) was not created because of long creation duration(> 5 hours)

• Performance testing of Lists App refreshing was performed for 1, 5, and 10 parallel threads (virtual users). With the increase in the number of threads, the duration increased in some cases by 4 times.

• For all 3 tests, the services CPU utilization has about the same value. It reached a maximum of 40% for mod-fqm-manager and 35% for mod-lists. No memory leak is suspected for all the modules.

• Database CPU utilization on LA_test_5 and LA_test_10 was a maximum of 95% on the reader instance and 60% on the writer instance. Reader instance is a bottleneck

Test runs

LA_test_1 - was executed to collect baseline metrics. 30 lists were updated one after another, from first to last. Number of loops =3;

LA_test_5 - load test for 5 concurrent threads;

LA_test_10 - load test for 10 concurrent threads.

Results

CPU Utilization 

LA_test_1

During the LA_test_1 test CPU utilization reached a maximum of 33% for mod-fqm-manager and 30% for mod-lists.

Open

LA_test_5

During the test CPU utilization reached a maximum of 40% for mod-fqm-manager and 30% for mod-lists.

Open

LA_test_10

During the test CPU utilization reached a maximum of 40% for mod-fqm-manager and 37% for mod-lists.

Memory Utilization

No memory leak is suspected for all the modules during all tests.

LA_test_1.

The mod-fqm-manager service increased the use of CPU from 64% to 84% and mod-lists from 62% to 68%

LA_test_5

The mod-fqm-manager service increased the use of CPU from 86% to 92% and mod-lists from 74% to 76%

LA_test_10

The mod-fqm-manager service increased the use of CPU from 87% to 92% and mod-lists from 76% to 77%

DB CPU Utilization

Maximum DB CPU utilization reached 57% (writer instance) and 82% (reader instance) during the LA_test_1 test. 

Maximum DB CPU utilization reached 60% (writer instance) and 95% (reader instance) during the LA_test_5 test. 

Maximum DB CPU utilization reached 73% (writer instance) and 94% (reader instance) during the LA_test_10 test. 

DB Connections

During all tests, the number of connections to the reader instance = 20; the number of connections to the writer instance ~400;

LA_test_1

LA_test_5

LA_test_10

DB Load

LA_test_1

Writer instance

Reader instance

LA_test_5

Writer instance

Reader instance

LA_test_10

Writer instance

Reader instance

Appendix

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• DB Cluster with 2 instances (reader and writer)

  NameAPI NameMemory GIBvCPUsmax_connectionsR6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731

• MSK tenant
  4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3