/
[Poppy] List App with multiple workflows and R/W split enabled test report

[Poppy] List App with multiple workflows and R/W split enabled test report

Nov 09, 2023

Overview

This document contains the results of testing List App refreshing of 200k records with multiple parallel workflows (Check-in/Check-out and Data import) and R/W split enabled in the Poppy release. The goal of testing is to assess the performance of mod-lists along with other workflows (CICO and DI) and R/W split enabled  and to measure mod-lists' impact on other modules

Ticket:

PERF-708: [List App] Create test to measure the performance of refreshing a list with many records (with multiple workflows) with DB R/W split enabledClosed

Summary

  • Load tests showed that average List App refresh duration in multiple-workflow test with R/W split enabled is 11.4 min.

  • ListApp duration while testing separately is 3.9 minutes less (7.5 min).

  • ListApp duration while testing with other workflows but with R/W split disabled is 6.3 minutes more (17.7 min).

  • According to the results, maximum CPU utilization for DI+CICO test is about 126%, while with ListApp scenario added CPU reaches 150%.

  • CPU utilization of mod-lists and mod-fqm-manager for ListApp+DI+CICO test is up to 140% with R/W split enabled, while it reaches only 105% with R/W split disabled. 

  • RDS CPU utilization (writer instance) is about 96%  for DI+CICO test and it doesn't increase with ListApp scenario added. RDS CPU utilization for reader instance reaches 95% during ListApp+DI+CICO test.

  • RDS CPU utilization for ListApp+DI+CICO (writer instance) didn't change in comparison with the same test but with R/W split disabled for mod-fqm-manager.

  • No memory leak is suspected of all the modules.

Test runs

Query used in lists - "Item status != Available". List refresh result is 200K records.

Scenario

Data quantity

List App + Check-in/Check-out + Data Import Create*

ListApp - 10 users

CICO - 20 users

DI - 25k

 Check-in/Check-out + Data import*

CICO - 20 users

DI - 25k

List App

ListApp - 10 users

*PTF Create 2 job was used for DI scenario

Results

Transaction

Duration

 R/W split disabled*

 R/W split enabled**

Testing separately

(ListApp and DI+CICO)

 

 

Testing in parallel

 

Testing separately

(ListApp and DI+CICO)

Testing in parallel

ListApp refresh, avg

8.5 min

17.7 min

7.5 min (-1 min)

11.4 min (-6.3 min)

Check-in, avg

1.173 sec

1.290 sec

0.992 (-0.181 sec)

1.087 (-0.203 sec)

Check-out, avg

1.648 sec

1.948 sec

1.634 (-0.014 sec)

1.765 (-0.183 sec)

DI Create 25k 

18 min

22 min

13 min (-5 min)

15 min (-7 min)

* Query used in lists - "Item status == Checked out". List refresh result is 200K records. Results are taken from previous test report: [Poppy] List App with multiple workflows and R/W split disabled test report

**Query used in lists - "Item status != Available". List refresh result is about 200K records.

Grafana results

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

According to the results, maximum CPU utilization for DI+CICO test is about 126%, while with ListApp scenario added CPU reaches 150%.

CPU utilization of mod-lists and mod-fqm-manager for ListApp+DI+CICO test is up to 140% with R/W split enabled, while it reaches only 105% with R/W split disabled. 

ListApp

DI + CICO

ListApp + DI + CICO

 

ListApp + DI + CICO with R/W split disabled*

*Results are taken from the previous report: [Poppy] List App with multiple workflows and R/W split disabled test report.

Memory Utilization

No memory leaks were found during the tests.

ListApp

DI + CICO

ListApp + DI + CICO


DB CPU Utilization

RDS CPU utilization (writer instance) is about 96% for DI+CICO test and it doesn't increase with ListApp scenario added. RDS CPU utilization for reader instance reaches 95% during ListApp+DI+CICO test.

RDS CPU utilization for ListApp+DI+CICO (writer instance) didn't change in comparison with the same test but with R/W split disabled for mod-fqm-manager.

ListApp

DI + CICO

ListApp + DI + CICO

 

ListApp + DI + CICO with R/W split disabled*

*Results are taken from the previous report: [Poppy] List App with multiple workflows and R/W split disabled test report.


DB Connections

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

Reader instance

Writer instance

DI + CICO

Writer instance

ListApp + DI + CICO

Reader instance

Writer instance

 

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

  • MSK tenant

    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

 

Module
pcp1-pvt

Task Def. Revision

Task Count

Mem Hard Limit

Mem Soft limit

CPU units

Xmx

MetaspaceSize

MaxMetaspaceSize

R/W split enabled

mod-inventory-storage:27.0.0

10

2

4096

3690

2048

3076

384

512

false

mod-users:19.2.0

19

2

1024

896

128

768

88

128

false

nginx-okapi:2023.06.14

8

2

1024

896

128

0

0

0

false

mod-circulation-storage:17.1.0

10

2

2880

2592

1536

1814

384

512

false

okapi:5.1.1

9

3

1684

1440

1024

922

384

512

false

mod-inventory:20.1.0

9

2

2880

2592

1024

1814

384

512

false

mod-circulation:24.0.0

10

2

2880

2592

1536

1814

384

512

false

pub-okapi:2023.06.14

8

2

1024

896

128

768

0

0

false

mod-fqm-manager:1.0.0

5

2

1024

896

128

768

88

128

true

mod-lists:1.0.0

5

2

3000

2600

128

2048

384

512

false

 

Additional info

Grafana graphs:

List App + Check-in/Check-out + Data Import Create

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1699353475555&to=1699354950119

Check-in/Check-out + Data import

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All&from=1699290758840&to=1699291480510

 

Methodology

  1. Enable R/W split for mod-fqm-manager.

  2. Create 10 lists with the query "Item status != Available" on each of three tenants to be able to run a test for up to 30 concurrent lists.

  3. Prepare 200K item records for the query to return. Details can be found at the link: Steps for testing process#ListApp

  4. Conduct test with ListApp scenario

  5. Conduct test with ListApp scenario + CICO + DI

  6. Conduct test withCICO + DI scenarios

  7. Compare the results