Overview

Per PERF-267, test Bulk Edit (PERF-271) of 10K records to understand the workflow behavior before and when the mod-data-export-worker task crashes, if it crashes at all.

How long does it take to export 10K records?
What happens to the job that is running, will it be able to resume and complete successfully when the new task is spun up?
Look for a memory trend and use it to decide on the number of concurrent jobs needed to reach the tipping point.

Infrastructure

PTF -environment

10 m6i.2xlarge EC2 instances (changed. In Lotus it was m5.xlarge)
2 instances of db.r6.xlarge database instances, one reader and one writer
MSK
- 4 m5.2xlarge brokers in 2 zones
- auto.create-topics.enable = true
- log.retention.minutes=120
- 2 partitions per DI topics
okapi (running tasks -3)
- 1024 CPU units, 1360 MB mem
mod-users (running tasks -2)
- 128 CPU units, 896 MB mem
mod-data-export (running tasks -1)
- 1024 CPU units, 896 MB mem
mod-data-export-spring (running tasks -1)
- 256 CPU units, 1844 MB mem
mod-data-export-worker (running tasks -1)
- 256 CPU units, 1844 MB mem
mod-notes (running tasks -2)
- 128 CPU units, 896 MB mem
mod-agreements (running tasks -2)
- 128 CPU units, 1382 MB mem

Software Versions

mod-data-export-worker v 1.4.2
mod-data-export-spring v 1.4.2
mod-agreements:5.2.0
mod-notes: 3.1.0
mod-users: 18.3.0

Test Runs

Scenario

Items App - updating item status
1.Navigate to the Bulk edit app
2.Select "Inventory-Items" app => Select "Items identifier" ( item barcodes, item UUIDs, holdings UUIDs, item HRIDs, item accession numbers) from "Record identifier" dropdown
3.Upload a .csv file with items identifiers by dragging it on the file drag and drop area
4.Click "Actions" menu => Click "Download matched records"
5.Click "Actions" menu => Select "Start Bulk edit" option
6.Select "Item status" from "Options" dropdown
7.Select Item's status (Available, Missing,Withdraw)from "Select Itemstatus" dropdown
8.Click "Confirm changes" button
9.Click the "Download preview" button
10.Click "Confirm changes" button
11.Click "Actions" => Select "Download changed records (CSV)"

Record identifier files location - http://carrier-io.int.folio.ebsco.com/artifacts?q=export

				accession number				barcode
Test	Records number	Duration	Comment	Record identifier file name	Results for finding records	Time to upload	Time to update	Record identifier file name	Results for finding records	Time to upload	Time to update
1.	100	multiple time check	Always Pass	access_num_100.csv	PASS	6 sec	10 sec	100-item-barcodes_version.csv	PASS	5 sec	6 sec
2.	1000	multiple time check	Always Pass	access_num_1000.csv	PASS	1 min	1 min 30 sec	1000-item-barcodes_version.csv	PASS	40 sec	1 min
3.	2000	multiple time check	Always Pass	access_num_2000.csv	PASS	1 min 30 sec	2 min		-	-	-
4.	5000	multiple time check	Always Pass	access_num_5_000.csv	PASS	3 min	7 min	5000-item-barcodes_version.csv	PASS	4 min	7 min
5	7000	multiple time check			-	-	-	7k-item-barcodes_version.csv	PASS	5min	10min
6.	10000	multiple time check	Always Pass	access_num_10_000.csv	PASS	6 min	13 min	10k-item-barcodes_version.csv	PASS	8 min	13 min
7	20000	1-time check	Always Pass	access_num_20_000.csv	PASS	12 min	26 min
8	100000	1-time check						100k-item-barcodes_version.csv

Results

Summary

Initial test report for Bulk Edits items-app functionality.

10 K records can be exported in 6min (time to upload identifier file and find records). The time to commit edited records depends on the number of records to be changed for 10000 records it is about 13 min.
The files with identifiers should be strictly determined and do not allow data from files created with Excel (error for the first line of each file)
Files with barcodes: when trying to edit, give us 0 records to be edited the reason for error - For input string: "" .
Memory trend: memory usage increases over time for mod-data-export-worker increases from 37% to 46% in 2 steps at the start of the changes commitment process for 10000 and 20000 records. for mod-data-export-spring increases from 24% to 26% and become stable. All other services without changes.
CPU utilization for all mods did not exceed 43%.
5 parallel jobs for 10k records can be performed simultaneously only if started with the ramp-up of a minimum of 10sec (for both upload and editing processes). If the jobs started with fewer 10s intervals they will have IN_PROGRESS status forever.
The failover test - fail - we have an end_time in the database (the same time the mod-data-export-worker task was stopped) but the status is always "in_progress"

Memory usage

Memory trend: memory usage increases over time for mod-data-export-worker increases from 37% to 46% in 2 steps at the start of the changes commitment process for 10000 and 20000 records. for mod-data-export-spring increases from 24% to 26% and become stable. All other services without changes.

RDS CPU utilization

RDS CPU utilization did not exceed 11.5%

CPU utilization

CPU utilization for all mods did not exceed 43%

Notable observations

We have only 1 Job number (and 1 job in count) for both the identifier and updating processes named BULK_EDIT_IDENTIFIERS.
How many records have to be updated?
Has the update started or finished yet?
Have changes been committed yet (You can only observe 0 changes done. Is that all? No, in this case, it is only beginning)?

Bulk Edit Items App report [Morning Glory]