Combined test for data export worker (Morning Glory)

Owned by Olga Kondratenko
Last updated: Aug 23, 2022

IN PROGRESS

Overview

Per PERF-267 - Getting issue details... STATUS the following tests were performed using the data export workflow (eHoldings, bulk edits, and edifacts) to determine what happens to the export workflows when mod-data-export-worker (with one node) fails and to what load can the exports be run successfully. 

Infrastructure

PTF -environment

  • 10 m6i.2xlarge EC2 instances  (changed. In Lotus it was m5.xlarge)
  • 2 instances of db.r6.xlarge database instances, one reader and one writer
  • MSK
    • 4 m5.2xlarge brokers in 2 zones
    • auto.create-topics.enable = true
    • log.retention.minutes=120
    • 2 partitions per DI topics
  • okapi (running tasks -3)
    • 1024 CPU units, 1360 MB mem
  • mod-users (running tasks -2) 
    • 128 CPU units, 896 MB mem
  • mod-data-export (running tasks -1) 
    • 1024 CPU units, 896 MB mem
  • mod-data-export-spring (running tasks -1) 
    • 256 CPU units, 1844 MB mem
  • mod-data-export-worker (running tasks -1) 
    • 256 CPU units, 1844 MB mem
  • mod-notes (running tasks -2) 
    • 128 CPU units, 896 MB mem
  • mod-agreements (running tasks -2) 
    • 128 CPU units, 1382 MB mem


Software Versions

  • mod-data-export-worker: 1.5.0-SNAPSHOT.54
  • mod-data-export-spring: 1.4.2
  • mod-agreements: 5.2.1
  • mod-notes: 3.1.0
  • mod-users: 18.3.0

Test Runs (initial assumption of the data sets for combo testing)

Bulk edit files are located in: Performance testing data Google Drive

Test Run

Duration

EdifacteHoldings

Bulk-edit

user barcodes

Bulk edit

item barcodes

Bursar

(optional)

Circulation logs (optional)Memory Behavior (Trend)Comment

Record identifier files

 name (bulk edit)		Time to process Results

#1

1-time checkExport 10 orders with 5 order lines each for 1 organization integration. 50 order lines exported to 1 .edi file

Export a "Package" record with small number of "Titles" (<1.000)

https://ptf-rmb.int.aws.folio.org/eholdings/packages/228-1126892 (744)

100 records

file name 100-user-barcodes.csv

100 records

file name 100-item-barcodes.csv








#2

1-time checkExport 10 orders for each account number with 10 order lines each. Each account represents a different organization integration. Meaning 100 order lines exported to 10 different .edi files

Export a "Package" record with close to limit number of "Titles" (between 9.000 and 10.000) without assigned "Notes" records

https://ptf-rmb.int.aws.folio.org/eholdings/packages/413-2605025 (9073)


1 000 records

file name 1000-user-barcodes.csv

1 000 records

file name 1000-item-barcodes.csv








#3

1-time checkExport 100 orders for each account number with 10 order lines each. Each account represents a different organization integration. Meaning 1,000 order lines exported to 10 different .edi files

export "Package" record with "Titles" (about 5k) and assigned 1 large "Note" record


https://ptf-rmb.int.aws.folio.org/eholdings/packages/413-2603523 (4,971 titles and one very large note) 

2 500 records

file name 2500-user-barcodes.csv

10 000 records

file name 10k-item-barcodes.csv








#41-time checkExport 100 orders for each account number with 100 order lines each. Each account represents a different organization integration. Meaning 10,000 order lines exported to 10 different .edi files

export "Package" record with "Titles" (between 9.000 and 10.000) and assigned 1 large "Note" record

https://ptf-rmb.int.aws.folio.org/eholdings/packages/36-6693? (9,825 titles and one very large note) 

5 000 records

file name 5000-user-barcodes.csv

100 000 records

file name 100k-item-barcodes.csv








#5 1-time check????






Test resulst

Test Run

Duration

EdifacteHoldings

Bulk-edit

user barcodes

Bulk edit

item barcodes

Bursar

(optional)

Circulation logs (optional)Memory Behavior (Trend)CommentTime to process Results
#1

3x 100 orders with 100 PO lines each (10k)

3[10k files as output

3x 4472 w/o Note

https://ptf-rmb.int.aws.folio.org/eholdings/packages/58-2110695?searchfield=title


1