Versions Compared

Version Old Version 19 New Version Current
Changes made by

Olga Kondratenko

Roman_Fedynyshyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Table of Contents

Overview

Per

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-315
the following tests were performed using the data export workflow (bulk edits, eHoldings, and edifacts) to determine what load can the exports run successfully together 

Infrastructure

PTF -environment

  • 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfka-1
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=truec
    • log.retention.minutes=480
    • default.replication.factor=3

...

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-data-export-spring1.5.0-SNAPSHOT.5821256204818445121536 (1236 in MG)
mod-data-export-worker2.1.0-SNAPSHOT.85221024307226005122048 (1536 in MG)
mod-users19.0.0-SNAPSHOT.57342128 (258 in MG)1024896128768
mod-notes4.0.0-SNAPSHOT.237221281024896128m322 (768m in MG)
mod-inventory19.0.0-SNAPSHOT.38322102428802592512m1814m

mod-inventory-storage

25.0.0-SNAPSHOT.63152102422081952512m1440m
okapi4.14.423102416841440512m922m
mod-ordersmod-orders-12.5.0-SNAPSHOT.22932102420481440512m896m
mod-orders-storagemod-orders-storage-13.4.0-SNAPSHOT.154221281024896128m768m
mod-circulationmod-circulation-23.3.0-SNAPSHOT.5443210241024896128m768m
mod-circulation-storagemod-circulation-storage-15.1.0-SNAPSHOT.29662102415361440512m896m
mod-agreements5.4.0-SNAPSHOT.1042212815921488512m968m

...

  • Up to 10 different concurrent jobs can be performed successfully. Compared to individual bulk edit runs -10k records per user, 5 users simultaneously (50k records total) can be uploaded and updated in about 30 min which is successful. Bulk Edits with more than 10 concurrent jobs can fail with
    Jira Legacy
    serverFOLIO Issue TrackerSystem JIRA
    serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
    keyMODEXPW-306
    or
    Jira Legacy
    serverFOLIO Issue TrackerSystem JIRA
    serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
    keyPERF-335
  • Memory trend: memory usage increases for mod-data-export-worker by 1% (from 109% to 110%).  Stable for all other modules.
  • CPU - nginx-okapi is spiking up to 170% at the start of tests. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
  • For all tests - RDS CPU utilization did not exceed 80%.

In general, Nolana is more stable compared to the Morning Glory release. All were tested with a higher load and 

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyMODEXPW-306
occurs very rarely. As for
Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335
, it can be reproduced only if mod-data-export-worker is overloaded and will be solved in UI.

...

Test Run

EdifacteHoldings

Bulk-edit

user barcodes

Bulk edit

item barcodes

Bulk edit

Holdings HRIDs

BursarCirculation logMemory Behavior (Trend)CommentTime to process Results

#1

10 jobs 1K orders each7 jobs 9,194 titles each5 jobs -2500 records each5 jobs -10k records each-DNR35K records
memory usage increases for mod-data-export-worker during the test by 2% up to 111%. Stable for all other modules.		28 jobs totalAbout 27 min (time to process the longest one - 5 jobs -10k item records)

#2

10 jobs 1K orders each7 jobs 9,194 titles each5 jobs -2500 records each5 jobs -10k records each-DNR35K records28 jobs totalAbout 27 min (time to process the longest one - 5 jobs -10k item records)

24 successful, 4 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335

#3

10 jobs 1K orders each7 jobs 9,194 titles each5 jobs -2500 records each5 jobs -10k records each-DNR35K records28 jobs totalAbout 29 min (time to process the longest one - 5 jobs -10k item records)

24 successful, 4 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335













Test results

Test Run

EdifactTime to process ResultseHoldingsTime to process Results

Bulk-edit

user barcodes

Time to process Results

Bulk edit

item barcodes

Time to process ResultsComment
#110 jobs 1K orders each1 min 40 ssuccess7 jobs 9,194 titles each±10 minsuccess5 jobs -2500 records each3 min 45 secsuccess5 jobs -10k records each26-27 minsuccess


Sometimes Bulk edit jobs can be failed with

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyMODEXPW-306
or
Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335

#210 jobs 1K orders each1 min 50 ssuccess7 jobs 9,194 titles each±12 minsuccess5 jobs -2500 records each2 min 10 sec for a successful job

1 Successful - 4 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335

5 jobs -10k records each27 minsuccess
#310 jobs 1K orders each1 min 37 ssuccess7 jobs 9,194 titles each±12 minsuccess5 jobs -2500 records each2 min 50 sec for a successful job

1 Successful - 4 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335

5 jobs -10k records each28 min 30 secsuccess

...

 * - average job processing time for the number of records specified ( upload identifiers + editing)

Note- To manipulate with number of records to export in Circulation log workflow we just change dates in from-to section.


Resource Usage





Note: Between test #2 and test #3 we have restarted mod-data-export-worker manually. 

...

Test Run

Bulk-edit

user barcodes

Bulk edit

item barcodes

Memory Behavior (Trend)CommentTime to process Results

#1

5 jobs-5k records each+ 5 jobs -2500 records each5 jobs-5k records each+ 5 jobs -10k records each+ 5 jobs-1k records each+ 5 jobs-100 records eachmemory usage increases for mod-data-export-worker during the test by 2% up to 109%. Stable for all other modules.30 jobs totalAbout 29 min (time to process the longest one - 5 jobs -10k item records)

28 successful, 2 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyMODEXPW-306


Test results

Test Run

Bulk-edit

user barcodes

Time to process Results

Bulk edit

item barcodes

Time to process ResultsComment
#25 jobs -2500 records each 7 min 30 sec *

4 Successful -1 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyMODEXPW-306

5 jobs -100 records each1 min 30 sec *Successful


Sometimes jobs can be failed with

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyPERF-335

5 jobs -5k records each14 min *

4 Successful -1 failed

Jira Legacy
serverFOLIO Issue TrackerSystem JIRA
serverId6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc
keyMODEXPW-306

5 jobs -1k records each5 min 30 sec *Successful



5 jobs -5k records each16-18 minSuccessful



5 jobs -10k records each27-29 minSuccessful

...