Versions Compared

Old Version 2

changes.mady.by.user Mykhailo Petryshyn

Saved on

compared with

New Version 3

changes.mady.by.user Mykhailo Petryshyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

Bulk Edit - Establish a performance baseline for user status bulk updates. 

...

  • Can it be used with up to 5 concurrent users? 
  • Run consecutively four jobs editing 10k  item records
  • Run simultaneously four jobs editing 10k item records
  • Look for a memory trend and CPU usage

Summary 


Results

Total processing time of upload and edit - commit changes. Units =hours:minutes: seconds

onds

Number of virtual user/ Records

1VU

2VU

3VU

4VU

5VU

100 records




1000 records




5000 records




10000 records




100K

records

03:14:59

03:33:24

03:15:41

03:27:06

03:21:31

03:25:10

06:10:23 

03:33:23

03:20:39

03:21:24

04:04:24

04:02:45

04:03:11

04:06:32

04:12:04


Resource utilization

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. From the Instance CPU utilization graph, you can see that the bulk-edit process consists of 2 parts: file uploading(90-120 minutes) and records processing.
Image Added


Memory usage

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. 
Image Added

Service CPU usage

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. 

Image Added

Appendix

Infrastructure

PTF -environment pcp1

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3

...

ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
pcp1-pvt
Dec 06 13:08:42 UTC 2023
mod-authtoken13/mod-authtoken:2.14.121440115251292288128FALSE
mod-inventory-update9/mod-inventory-update:3.2.12102489612876888128FALSE
mod-bulk-operations8mod-bulk-operations:1.1.023072260010241536384512FALSE
mod-users-bl9/mod-users-bl:7.6.021440115251292288128FALSE
mod-inventory-storage12mod-inventory-storage:27.0.324096369020483076384512FALSE
mod-data-export-worker9mod-data-export-worker:3.1.023072280010242048384512FALSE
mod-source-record-storage15mod-source-record-storage:5.7.325600500020483500384512FALSE
mod-inventory11mod-inventory:20.1.322880259210241814384512FALSE
mod-users20mod-users:19.2.12102489612876888128FALSE
mod-patron-blocks9mod-patron-blocks:1.9.021024896102476888128FALSE
nginx-edge9nginx-edge:2023.06.1421024896128000FALSE
mod-quick-marc9mod-quick-marc:5.0.01228821761281664384512FALSE
nginx-okapi9nginx-okapi:2023.06.1421024896128000FALSE
okapi-b11okapi:5.1.23168414401024922384512FALSE
mod-data-export11mod-data-export:4.8.111024896102476888128FALSE
mod-data-export-spring11mod-data-export-spring:3.0.01204818442561536384512FALSE

Methodology/Approach

  1. Item records update:
    1. Upload file with item barcodes 
    2. Click the Start bulk edit option in the Action menu and make the following changes:
      1. Set Temporary location to Clear field
      2. Set Permanent location to < to the value available on test environment>
      3. Set Status to Unknown
      4. Set Temporary loan type to Clear field
      5. Set Permanent loan type to < to the value available on test environment>
      6. Add Administrative note by adding text: "This is a new administrative note"
      7. Add Action note by adding text: "This is a new action note"
      8. Suppress from discovery (set the value to true)
    3. Confirm the changes
    4. Commit the changes
    5. Verify the changes are correct
    6. Download the file with updated records
    7. Download the file with errors (if applicable)

...