Versions Compared

Old Version 38

changes.mady.by.user Olga Kondratenko

Saved on

compared with

New Version Current

changes.mady.by.user Martin Tran

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • All tests were run successfully and without errors on the Mobius-performance testing cluster (mcpt). Approximately over 100 tests with various environment configurations were conducted. The tests and configurations that showed the best results for two database modifications, such as db.r6g.4xlarge and db.r6g.8xlarge, will be demonstrated.
  • During testing, an insufficient amount of resources for Amazon OpenSearch Service was detected, so data node instance type was increased to r6g.2xlarge.search.

  • To improve performance, the AWS task count was adjusted for certain services, which increased the memory load on the database. To avoid OOM (Out of Memory) errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. Specifically, the shared_buffers parameter was modified to SUM({DBInstanceClassMemory/24076},-50003).

  • To avoid "remaining connection slots are reserved for non-replication superuser and rds_superuser connections" errors, configurations were changed in both the DB instance parameter group and the DB cluster parameter group. max_connections was modified to LEAST({DBInstanceClassMemory/9531392},9000)
  •  For mod-authtoken CPU was changed from 512 to 1024 and task count was adjusted to 6 tasks (2 default), okapi and nginx-okapi CPU  was changed from 1024 to1536.

  • No sign of memory leaks on every module during master-script tests. Memory shows a stable trend. For tests with a high load of single record create and update (62 virtual users without pauses) OOM issue of mod-inventory was reported 

    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMODINV-1023

  • No modules are showing a growing CPU trend or any spikes. 

  • Average response time differs and depends on environment configurations.

  • The "Receiving: downloading" workflow. Due to its design, it's processed by the frontend and sends frequent requests to the database in a very large quantity. When under a load of 70% of tenants simultaneously, this causes significant degradation in the performance of all other workflows. Therefore, results are provided in tables with additional tests conducted with this workflow and without it. The CPU usage by the database increases by more than 25% with this workflow and reaches 99%.
  • Database connection count did not exceed 5700
  • During the high-load test, exporting 800,000 records concurrently for 62 tenants, an issue with the mod-data-export occurs
    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-927
    , which will be resolved and tested in the next release - Quesnelia 
    Jira Legacy
    serverSystem Jira
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyPERF-888
    . Other workflows performed normally. Resource usage was stable and corresponded to the load.
  • Test with Fiscal year rollover average duration 3:21:51 for 30 tenants and 6:26:33 for 60 tenants, finished without errors, shows severe performance degradation for data import and data export workflows due to high database CPU usage 99.8%.


Recommendations

Database parameter group configuration:

parameterchanged valuedefault valueComment
shared_buffersSUM({DBInstanceClassMemory/24076},-50003)SUM({DBInstanceClassMemory/12038},-50003)Increases freeable db memory from 25% to 50% and decreases cache memory limit from 75% to 50%
max_connectionsLEAST({DBInstanceClassMemory/9531392},9000)LEAST({DBInstanceClassMemory/9531392},5000)Increases maximal connection limit from 5000 to 9000 connections

Amazon OpenSearch Service

data node instance type - r6g.2xlarge.search

Services

mod-authtoken CPU - 1024, 6 tasks

okapi CPU-1536

nginx-okapi CPU-1536

For database size db.r6g.4xlarge

adjust task count to 6 for services:

  • mod-authtoken

adjust task count to 4 for services:

  • mod-permissions
  • mod-search
  • mod-patron
  • mod-inventory
  • mod-inventory-storage
  • mod-circulation
  • mod-circulation-storage
  • mod-order
  • mod-order-storage,
  • mod-invoice
  • mod-invoice-storage
  • mod-users

For database size db.r6g.8xlarge

adjust task count to 6 for services:

  • mod-authtoken
  • mod-search

adjust task count to 4 for services:

  • mod-permissions
  • mod-patron
  • mod-inventory
  • mod-inventory-storage
  • mod-circulation
  • mod-circulation-storage
  • mod-order
  • mod-order-storage,
  • mod-invoice
  • mod-invoice-storage
  • mod-organization
  • mod-organization-storage
  • mod-users
  • mod-finance
  • mod-finance-storage
  • mod-configuration


"Receiving: downloading" workflow optimization

Test Runs 

Test #

Workflows

Test Conditions

Results

1.

All workflows

2 tasks for all of the modules  (except the modules with 1 task by requirements) database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, 

Completed
2.All workflows

Adjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536

Completed
3.Without ERW: Exporting Receiving Information to CSV

Adjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536

Completed
4.All workflowsAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users, mod-organization, mod-organization-storage, mod-finance, mod-finance-storage, mod-configuration and task count 6 for mod-authtoken, mod-search. Database r6g.8xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed
5.Without ERW: Exporting Receiving Information to CSVAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users, mod-organization, mod-organization-storage, mod-finance, mod-finance-storage, mod-configuration and task count 6 for mod-authtoken, mod-search. Database r6g.8xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed
6.All workflowsAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed
7.All workflows, high-loadAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed with errors from data export
8.All no-acquisition workflows + Fiscal year rollover  30 tenantsAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed
9.All no-acquisition workflows + Fiscal year rollover  60 tenantsAdjasted task count to 4 for services: mod-permissions, mod-search, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage, mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, mod-users and task count 6 for mod-authtoken. Database r6g.4xlarge, shared_buffers   parameters  SUM({DBInstanceClassMemory/24076},-50003), Amazon OpenSearch Service r6g.2хlarge.search, mod-authtoken CPU - 1024, okapi CPU-1536, nginx-okapi CPU-1536Completed with errors from data import

...



Baseline (milliseconds)Tuned with Receiving: downloading (milliseconds)Tuned without Receiving: downloading (milliseconds)

db.r6g.4xlargeErrors

Test №1

db.r6g.4xlarge

2 tasks for all of the modules

Errors

Test №2

db.r6g.4xlarge

Test №4

db.r6g.8xlarge-configs for 8xlarge db

ErrorsDelta 4xlarge and 8xlarge db

Test №3

db.r6g.4xlarge

Test №5

db.r6g.8xlarge -configs for 8xlarge db

ErrorsDelta 4xlarge and 8xlarge db
AIE_TC: Create Invoices347300226010800648480-39.45325324980-23.21
AIE_TC: Invoices Approve331300221070703541920-40.41394130740-22.00
AIE_TC: Paying Invoices311190226620685743720-36.24487538340-21.35
CICO_TC_Check-In Controller6420147240225415500-31.23172813480-21.99
CICO_TC_Check-Out Controller10520478420403728610-29.13305523110-24.35
CSI_TC:Share local instance110184316491015945158530-0.5815816157200-0.61
DE_Exporting MARC Bib records custom workflow53296038085043997392510-10.7941901356990-14.80
DE_Exporting MARC Bib records workflow36563033504042815352150-17.7551828353790-31.74
EVA_TC: View Account-21183804753890-18.114143640-12.08
ILR_TC: Create ILR5085237400150510230-32.0310988170-25.59
MSF_TC: mod search by auth query
1880 (average for all search flows)
52
12044054458006.625575490-1.44
MSF_TC: mod search by boolean query3365016317205.521681630-2.98
MSF_TC: mod search by contributors5841038941306.173983910-1.76
MSF_TC: mod search by filter query3332027929806.812872830-1.39
MSF_TC: mod search by keyword query3376028029605.712892850-1.38
MSF_TC: mod search by subject query5107040642705.174134070-1.45
MSF_TC: mod search by title query642501040106202.12105010340-1.52
OPIH_/oai/records--74208495730-32.5111495440-52.65
POO_TC: Add Order Lines185765415(API- the same in UI)105721051776282680-45.40628149390-21.37
POO_TC: Approve Order146352088683040309209760-47.96254320210-20.53
POO_TC Create Order99937056442028708159100-44.58273421820-20.19
RTAC_TC: edge-rtac-027350241113550-43.80169312820-24.28
SDIC_Single Record Import (Create)30395442145901149158580-49.02544943100-20.90
SDIU_Single Record Import (Update)96046389233019259133910-30.4711500103500-10.00
TC: Receiving-an-Order-Line3072710167843088178474920-46.14932278730-15.54
Serials-Receiving-Workflow2823360141327083977460400-45.18968381560-15.77
Unreceiving-a-Piece106148059500029205158080-45.87278622950-17.62
ULR_TC: Users loan Renewal Transaction8023661190325622390-31.23243118960-22.01
TOTAL--18810482.84328.120-32.04103.9983.140-20.05


...

PTF - environment mcpt

  • 14 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

  • 1 database instance, writer

    one of

    NameMemory GIBvCPUs

    db.r6g.4xlarge

    		128 GiB16 vCPUs
    db.r6g.8xlarge256 GiB32 vCPUs


  • Amazon OpenSearch Service: ptf-test
    • Data nodes
      Instance type-r6g.2xlarge.search
      Number of nodes-4
    • Dedicated master nodes
      Instance type-r6g.large.search
      Number of nodes-3
  • MSK ptf-mobius-testing2
    • 2 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=2

...

Modules memory and CPU parameters:

db.r6g.4xlarge

Module
mcpt-pvt
Thu May 16 07:29:57 UTC 2024		Task Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
mod-remote-storage18579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.0.124920447210243960512512
mod-ncip12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.42102489612876888128
mod-finance-storage14579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.5.041024896102470088128
mod-agreements17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:6.0.2015921488128000
mod-ebsconet12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.1.1212481024128700128256
edge-sip212579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.1.12102489612876888128
mod-consortia18579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.0.325136477610244416384512
mod-organizations12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.8.02102489612876888128
mod-settings17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.22102489620076888128
edge-dematic15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.1.11102489612876888128
mod-data-import15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.0.71204818442561292384512
mod-search18579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.0.5425922480204814405121024
mod-tags12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.1.02102489612876888128
mod-authtoken22579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.14.1614401152102492288128
edge-courses12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.3.02102489612876888128
edge-inventory6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-inventory:1.4.02102489612876888128
mod-inventory-update12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.2.12102489612876888128
mod-notify12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.1.02102489612876888128
mod-configuration13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.9.24102489612876888128
mod-orders-storage13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.6.04102489651270088128
edge-caiasoft12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.1.02102489612876888128
mod-login-saml15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.7.22102489612876888128
mod-erm-usage-harvester7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.4.12102489612876888128
mod-password-validator12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.1.0214401298128768384512
mod-gobi13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.7.12102489612876888128
mod-licenses15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:5.0.22248023121281792384512
mod-fqm-manager12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:1.0.32102489612876888128
edge-dcb9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.0.22102489612876888128
mod-bulk-operations18579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:1.1.923072260010241536384512
mod-graphql12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.00102489612876888128
mod-finance12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.8.04102489612876888128
mod-erm-usage7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.6.02102489612876888128
mod-lists15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:1.0.52102489612876888128
mod-copycat12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.5.02102489612876888128
mod-permissions26579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.4.04168415445121024384512
mod-entities-links17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:2.0.4225922480400144001024
pub-edge6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.142102489612876800
mod-orders13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.7.142048144010241024384512
edge-patron14579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.0.0.1582102489625676888128
edge-ncip12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.22102489612876888128
mod-users-bl12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.6.021440115251292288128
edge-ea-data-export7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ea-data-export:4.1.02102489612876888128
mod-invoice13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.7.241440115251292288128
mod-inventory-storage19579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.0.444096369020483076384512
mod-user-import12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.02102489612876888128
mod-sender12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.11.02102489612876888128
edge-oai-pmh15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.7.121512136010241440384512
mod-data-export-worker15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.1.223072280010242048384512
mod-rtac12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.5.02102489612876888128
mod-task-list7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-task-list:1.9.22102489612876888128
mod-circulation-storage20579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.1.742880259215361814384512
mod-calendar13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:2.5.02102489612876888128
mod-source-record-storage21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.7.525600500020483500384512
mod-event-config12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.6.02102489612876888128
mod-courses12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.82102489612876888128
mod-circulation-item9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.02102489612876888128
mod-inventory19579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.1.642880259210241814384512
mod-email12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.16.02102489612876888128
mod-circulation20579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.0.1142880259215361814384512
mod-pubsub16579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.11.32153614401024922384512
mod-di-converter-storage17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.1.52102489612876888128
edge-rtac15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.6.22102489612876888128
edge-orders15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:2.9.12102489612876888128
mod-template-engine12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.19.12102489612876888128
mod-users17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.2.24102489612876888128
mod-patron-blocks14579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.9.021024896102476888128
mod-audit13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.8.02102489612876888128
edge-fqm13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:1.0.12102489612876888128
mod-source-record-manager18579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.7.725600500020483500384512
nginx-edge6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.1421024896128000
mod-quick-marc16579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.0.11228821761281664384512
nginx-okapi9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14210248961536000
okapi-b8579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.23168414401536922384512
mod-feesfines12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.0.02102489612876888128
mod-invoice-storage13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.7.041872153610241024384512
edge-users6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-users:1.2.02102489612876888128
mod-dcb9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.0.02102489612876888128
mod-service-interaction15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:3.0.22204818442561290384512
mod-data-export17579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:4.8.711024896102476888128
mod-patron12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.0.04102489612876888128
mod-oai-pmh16579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.12.824096369020483076384512
edge-connexion12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.1.12102489612876888128
mod-kb-ebsco-java12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.02102489612876888128
mod-notes13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.1.021024896128952384512
mod-data-export-spring15579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.0.21204818442561536384512
mod-login12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.10.12144012981024768384512
mod-organizations-storage13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.6.04102489612876888128
pub-okapi6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800
edge-erm7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-erm:1.0.02102489612876888128
mod-eusage-reports7579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.0.02102489612876888128

...

MOBIUS Tests: scenarios were started by carrier.io or from load generator instance in the same (us-east-1) AWS region. 

The database contains five times more generated data for large tenant IDs compared to small tenant IDs. The table below displays the average count of generated instances and items for both small and large tenants..

tenantinstancesitems
Small84754103137
Large416176543428

The attached file contains full info about generated data per tenant.

  • large tenants IDs = 0042, 0024, 0051, 0006

...

At the initial stage of testing, there were numerous errors from various workflows, most notably those involving search. While investigating the cause of the errors, it was found that there was insufficient CPU capacity (100% utilization) on the data nodes of the OpenSearch cluster. Initially, testing was done with r6g.large.search, then increased to r6g.4xlarge.search, but it was too large and only utilized 40% CPU. Therefore, it was downsized to r6g.2xlarge.search, with the number of containers (tasks) increased to 4 for mod-search.

However, after identifying the issue related to search, a problem with the database persisted, such as OOM (Out of Memory). Two options were tested to address this issue: increasing the database size to db.r6g.8xlarge and adjusting the shared_buffers configuration to SUM({DBInstanceClassMemory/24076},-50003) for the size of db.r6g.4xlarge. Both tests showed promising results; however, in both cases, there were errors related to an insufficient number of connections to the database in the logs ("remaining connection slots are reserved for non-replication superuser and rds_superuser connections"). To address this, the max_connections configuration was set to LEAST({DBInstanceClassMemory/9531392},9000) for both sizes of the database, increasing the upper limit of connections to the database to 9000.

All of these measures helped to eliminate errors, but response times remained high, especially for check-in and check-out. To improve response time, we increased the number of containers for mod-authtoken to 4, which yielded very good results. Additionally, we improved response times for all circulation workflows by setting 4 containers for each: mod-permissions, mod-patron, mod-inventory, mod-inventory-storage, mod-circulation, mod-circulation-storage


To improve the performance of acquisition workflows, the number of containers for mod-order, mod-order-storage, mod-invoice, mod-invoice-storage, and mod-users was increased to 4. However, this resulted in significantly increased load on mod-authtoken, causing other workflows to slow down again. Therefore, the number of containers for mod-authtoken was increased to 6 to alleviate the load.
The next issue we're facing is the operation of the "Receiving: downloading" workflow. Due to its design, it's processed by the frontend and sends frequent requests to the database in a very large quantity. When under a load of 70% of tenants simultaneously, this causes significant degradation in the performance of all other workflows. Therefore, results are provided in tables with additional tests conducted with this workflow and without it. The CPU usage by the database increases by more than 25% with this workflow and reaches 99%.