[Poppy] List App with multiple workflows and R/W split disabled test report

Owned by Hanna Tuhuzbayeva
Last updated: Nov 09, 2023
4 min read

Overview

This document contains the results of testing List App refreshing of 200k records with multiple parallel workflows (Check-in/Check-out and Data import) in the Poppy release. The goal of testing is to assess the performance of mod-lists along with other workflows (CICO and DI)  and to measure mod-lists's impact on other modules

Ticket:

PERF-668 - Getting issue details... STATUS

PERF-691 - Getting issue details... STATUS

Summary

  • ListApp refreshing duration is about 8.5 min when testing separately from other workflows. With Data Import and Check-in/Check-out scenarios added ListApp duration has +108% increase and reaches 17.7 min. It's above the expected value of 10 minutes.
  • Data Import and Check-in/Check-out response times have increase about 10-20% with ListApp scenario added.
  • According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.
  • RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

Test runs

Query used in lists - "Item status == Checked out". List refresh result is 200K records.

ScenarioData quantity
List App + Check-in/Check-out + Data Import Create*

ListApp - 10 users

CICO - 20 users

DI - 25k

 Check-in/Check-out + Data import*

CICO - 20 users

DI - 25k

List AppListApp - 10 users

*PTF Create 2 job was used for DI scneario

Results

TransactionDuration

Degradation

[Orchid]

Testing separately

[Poppy]

Testing separately

(ListApp and DI+CICO)

[Poppy]

Testing in parallel

ListApp refresh, avg

from 9 min 40 sec to

10 min 40 sec

8.5 min17.7 min+108%
Check-in, avg
1.173 sec1.290 sec+9.9%
Check-out, avg
1.648 sec1.948 sec+18.2%
DI Create 25k 
18 min22 min+22%

Grafana results

DI + CICO

ListApp + DI + CICO


Instance CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.

ListApp

DI + CICO

ListApp + DI + CICO

Memory Utilization

No memory leaks were found during the tests.

ListApp

DI + CICO

ListApp + DI + CICO

DB CPU Utilization

RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

DI + CICO

ListApp + DI + CICO

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


ModuleTask Def. RevisionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-remote-storage:3.0.0-SNAPSHOT.133924920447210243960512512false
mod-finance-storage:8.5.0-SNAPSHOT.150821024896102470088128false
mod-agreements:6.0.0-SNAPSHOT.1618215921488128968384512false
mod-ebsconet:2.1.0-SNAPSHOT.648212481024128700128256false
mod-organizations:1.8.0-SNAPSHOT.6682102489612870088128false
mod-settings:1.1.0-SNAPSHOT.49821024896200000false
edge-dematic:3.0.0-SNAPSHOT.114811024896128000false
mod-data-import:2.8.0-SNAPSHOT.14891204818442561292384512false
mod-search:3.0.0-SNAPSHOT.15116225922480204814405121024false
mod-authtoken:2.14.0-SNAPSHOT.263821440115251292288128false
mod-configuration:5.9.2-SNAPSHOT.33682102489612876888128false
mod-orders-storage:13.6.0-SNAPSHOT.20482102489651270088128false
mod-fqm-manager:1.0.0-SNAPSHOT.92932300026001282048384512true
mod-bulk-operations:2.0.0-SNAPSHOT.53623072260010241536384512false
mod-graphql:1.11.0.364122102489612876888128false
mod-finance:4.8.0-SNAPSHOT.14182102489612870088128false
mod-erm-usage:4.6.0-SNAPSHOT.93112102489612876888128false
mod-copycat:1.5.0-SNAPSHOT.15982102489612876888128false
mod-lists:1.0.0-SNAPSHOT.119932300026001282048384512false
mod-permissions:6.4.0-SNAPSHOT.347282168415445121024384512false
mod-orders:12.7.0-SNAPSHOT.315822048144010241024384512false
mod-users-bl:7.6.0-SNAPSHOT.260821440115251292288128false
mod-inventory-storage:26.1.0-SNAPSHOT.696924096369020483076384512false
mod-invoice:5.7.0-SNAPSHOT.164821440115251292288128false
mod-user-import:3.8.0-SNAPSHOT.18382102489612876888128false
edge-oai-pmh:2.7.0-SNAPSHOT.141621512136010241440384512false
mod-data-export-worker:3.1.0-SNAPSHOT.174823072280010242048384512false
mod-circulation-storage:16.1.0-SNAPSHOT.356922880259215361814384512false
mod-source-record-storage:5.7.0-SNAPSHOT.308925600500020483500384512false
mod-inventory:20.1.0-SNAPSHOT.446822880259210241814384512false
mod-email:1.15.4-SNAPSHOT.11382102489612876888128false
mod-di-converter-storage:2.1.0-SNAPSHOT.64122102489612876888128false
mod-circulation:24.0.0-SNAPSHOT.6019228802592153676888128false
mod-pubsub:2.10.0-SNAPSHOT.16782153614401024922384512false
edge-orders:2.9.0-SNAPSHOT.11082102489612876888128false
mod-users:19.2.0-SNAPSHOT.638182102489612876888128false
folio/mod-patron-blocks:1.9.0-SNAPSHOT.119821024896102476888128false
folio/mod-audit:2.7.1-SNAPSHOT.18982102489612876888128false
folio/mod-source-record-manager:3.7.0-SNAPSHOT.240825600500020483500384512false
folio/nginx-edge:2023.06.14621024896128000false
folio/mod-quick-marc:5.0.0-SNAPSHOT.11481228821761281664384512false
folio/nginx-okapi:2023.06.14621024896128000false
folio/okapi:5.0.173168414401024922384512false
folio/mod-feesfines:18.3.0-SNAPSHOT.17292102489612876888128false
folio/mod-invoice-storage:5.7.0-SNAPSHOT.97821872153610241024384512false
folio/mod-service-interaction:3.0.0-SNAPSHOT.8782204818442561290384512false
folio/mod-data-export:4.8.0-SNAPSHOT.181911024896102476888128false
folio/mod-patron:6.0.0-SNAPSHOT.13882102489612876888128false
folio/mod-oai-pmh:3.12.0-SNAPSHOT.362324096369020483076384512false
mod-organizations-storage:4.6.0-SNAPSHOT.10682102489612870088128false
mod-data-export-spring:3.0.0-SNAPSHOT.129101204818442561536384512false
folio/mod-login:7.10.0-SNAPSHOT.30482144012981024768384512false
folio/pub-okapi:2023.06.1462102489612876800false

Additional info

Grafana graphs:

List App + Check-in/Check-out + Data Import Create

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&from=1696864147333&to=1696867911025&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Check-in/Check-out + Data import

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&from=1696927212822&to=1696931260640&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Methodology

1.Prepare data for ListApp: check-out 200K items with CICO JMeter script.

2. Disable timers before running the tests:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers

3. Conduct test with ListApp scenario

4. Conduct test with ListApp scenario + CICO + DI

5. Conduct test withCICO + DI scenarios

6. Compare the results

7. Enable timers:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"30"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"35"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"20"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"60"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers