Skip to end of banner
Go to start of banner

[Poppy] List App with multiple workflows test report and R/W split disabled

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 36 Next »

Overview

This document contains the results of testing List App refreshing of 200k records with multiple parallel workflows (Check-in/Check-out and Data import) in the Poppy release. The goal of testing is to assess the performance of mod-lists along with other workflows (CICO and DI)  and to measure mod-lists's impact on other modules

Ticket:

PERF-668 - Getting issue details... STATUS

PERF-691 - Getting issue details... STATUS

Summary

  • ListApp refreshing duration is about 8.5 min when testing separately from other workflows. With Data Import and Check-in/Check-out scenarios added ListApp duration has +108% increase and reaches 17.7 min. It's above the expected value of 10 minutes.
  • Data Import and Check-in/Check-out response times have increase about 10-20% with ListApp scenario added.
  • According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.
  • RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

Test runs

Query used in lists - "Item status == Checked out". List refresh result is 200K records.

ScenarioData quantity
List App + Check-in/Check-out + Data Import Create*

ListApp - 10 users

CICO - 20 users

DI - 25k

 Check-in/Check-out + Data import*

CICO - 20 users

DI - 25k

List AppListApp - 10 users

*PTF Create 2 job was used for DI scneario

Results

TransactionDuration

Degradation

[Orchid]

Testing separately

[Poppy]

Testing separately

(ListApp and DI+CICO)

[Poppy]

Testing in parallel

ListApp refresh, avg

from 9 min 40 sec to

10 min 40 sec

8.5 min17.7 min+108%
Check-in, avg
1.173 sec1.290 sec+9.9%
Check-out, avg
1.648 sec1.948 sec+18.2%
DI Create 25k 
18 min22 min+22%

Grafana results

DI + CICO

ListApp + DI + CICO


Instance CPU Utilization

ListApp

DI + CICO

ListApp + DI + CICO

Service CPU Utilization

According to the results, maximum CPU utilization for DI + CICO test is about 85%, while with ListApp scenario added CPU reaches 160%.

ListApp

DI + CICO

ListApp + DI + CICO

Memory Utilization

No memory leaks were found during the tests.

ListApp

DI + CICO

ListApp + DI + CICO

DB CPU Utilization

RDS CPU utilizatoin is about 95% for DI+CICO test and it doesn't increase with ListApp scenario added.

ListApp

DI + CICO

ListApp + DI + CICO

DB Load

ListApp

DI + CICO

ListApp + DI + CICO

Appendix

Infrastructure

PTF -environment pcp1

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database  instance, writer

    NameAPI NameMemory GIBvCPUsmax_connections
    R6G Extra Largedb.r6g.xlarge32 GiB4 vCPUs2731
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


ModuleTask Def. RevisionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
mod-remote-storage:3.0.0-SNAPSHOT.133924920447210243960512512false
mod-finance-storage:8.5.0-SNAPSHOT.150821024896102470088128false
mod-agreements:6.0.0-SNAPSHOT.1618215921488128968384512false
mod-ebsconet:2.1.0-SNAPSHOT.648212481024128700128256false
mod-organizations:1.8.0-SNAPSHOT.6682102489612870088128false
mod-settings:1.1.0-SNAPSHOT.49821024896200000false
edge-dematic:3.0.0-SNAPSHOT.114811024896128000false
mod-data-import:2.8.0-SNAPSHOT.14891204818442561292384512false
mod-search:3.0.0-SNAPSHOT.15116225922480204814405121024false
mod-authtoken:2.14.0-SNAPSHOT.263821440115251292288128false
mod-configuration:5.9.2-SNAPSHOT.33682102489612876888128false
mod-orders-storage:13.6.0-SNAPSHOT.20482102489651270088128false
mod-fqm-manager:1.0.0-SNAPSHOT.92932300026001282048384512true
mod-bulk-operations:2.0.0-SNAPSHOT.53623072260010241536384512false
mod-graphql:1.11.0.364122102489612876888128false
mod-finance:4.8.0-SNAPSHOT.14182102489612870088128false
mod-erm-usage:4.6.0-SNAPSHOT.93112102489612876888128false
mod-copycat:1.5.0-SNAPSHOT.15982102489612876888128false
mod-lists:1.0.0-SNAPSHOT.119932300026001282048384512false
mod-permissions:6.4.0-SNAPSHOT.347282168415445121024384512false
mod-orders:12.7.0-SNAPSHOT.315822048144010241024384512false
mod-users-bl:7.6.0-SNAPSHOT.260821440115251292288128false
mod-inventory-storage:26.1.0-SNAPSHOT.696924096369020483076384512false
mod-invoice:5.7.0-SNAPSHOT.164821440115251292288128false
mod-user-import:3.8.0-SNAPSHOT.18382102489612876888128false
edge-oai-pmh:2.7.0-SNAPSHOT.141621512136010241440384512false
mod-data-export-worker:3.1.0-SNAPSHOT.174823072280010242048384512false
mod-circulation-storage:16.1.0-SNAPSHOT.356922880259215361814384512false
mod-source-record-storage:5.7.0-SNAPSHOT.308925600500020483500384512false
mod-inventory:20.1.0-SNAPSHOT.446822880259210241814384512false
mod-email:1.15.4-SNAPSHOT.11382102489612876888128false
mod-di-converter-storage:2.1.0-SNAPSHOT.64122102489612876888128false
mod-circulation:24.0.0-SNAPSHOT.6019228802592153676888128false
mod-pubsub:2.10.0-SNAPSHOT.16782153614401024922384512false
edge-orders:2.9.0-SNAPSHOT.11082102489612876888128false
mod-users:19.2.0-SNAPSHOT.638182102489612876888128false
folio/mod-patron-blocks:1.9.0-SNAPSHOT.119821024896102476888128false
folio/mod-audit:2.7.1-SNAPSHOT.18982102489612876888128false
folio/mod-source-record-manager:3.7.0-SNAPSHOT.240825600500020483500384512false
folio/nginx-edge:2023.06.14621024896128000false
folio/mod-quick-marc:5.0.0-SNAPSHOT.11481228821761281664384512false
folio/nginx-okapi:2023.06.14621024896128000false
folio/okapi:5.0.173168414401024922384512false
folio/mod-feesfines:18.3.0-SNAPSHOT.17292102489612876888128false
folio/mod-invoice-storage:5.7.0-SNAPSHOT.97821872153610241024384512false
folio/mod-service-interaction:3.0.0-SNAPSHOT.8782204818442561290384512false
folio/mod-data-export:4.8.0-SNAPSHOT.181911024896102476888128false
folio/mod-patron:6.0.0-SNAPSHOT.13882102489612876888128false
folio/mod-oai-pmh:3.12.0-SNAPSHOT.362324096369020483076384512false
mod-organizations-storage:4.6.0-SNAPSHOT.10682102489612870088128false
mod-data-export-spring:3.0.0-SNAPSHOT.129101204818442561536384512false
folio/mod-login:7.10.0-SNAPSHOT.30482144012981024768384512false
folio/pub-okapi:2023.06.1462102489612876800false

Additional info

Grafana graphs:

List App + Check-in/Check-out + Data Import Create

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&from=1696864147333&to=1696867911025&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Check-in/Check-out + Data import

http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-with-average-latency?orgId=1&from=1696927212822&to=1696931260640&var-percentile=95&var-test_type=baseline&var-test=circulation_checkInCheckOut_Poppy_2&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All&var-Request=All

Methodology

1.Disable timers before running the tests:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"0"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers

2. Conduct test with ListApp scenario

3. Conduct test with ListApp scenario + CICO + DI

4. Conduct test withCICO + DI scenarios

5. Compare the results

6. Enable timers:

curl -XPATCH -d'{"id":"mod-circulation-storage_1","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_4","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_1","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation-storage_0","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_3","routingEntry":{"delay":"5"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_8","routingEntry":{"delay":"30"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_9","routingEntry":{"delay":"35"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_0","routingEntry":{"delay":"3"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_2","routingEntry":{"delay":"20"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_5","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_6","routingEntry":{"delay":"2"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers
curl -XPATCH -d'{"id":"mod-circulation_7","routingEntry":{"delay":"60"}}' https://okapi-ptf-pcp1-00.int.aws.folio.org/_/proxy/tenants/fs09000000/timers


  • No labels