Overview

Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously.

Summary

Test report for Bulk Edits holdings-app functionality 2023-03-22.

Orchid release shows good performance in combination of jobs with different numbers of records 2-3 different jobs simultaneously.

It was no failed jobs during testing.

Jobs duration for combined tests with 2500 or 10k user records is approximately the same as the baseline number.
The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing). mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests. For all other modules, no memory leaks are suspected.
CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.

Recommendations & Jiras

For further testing Users' bulk editing can be performed with 10k records.

Results

For all of the tests used the BARECODE identifier type for ITEMS & USERS and the HRID identifier type for Holdings.

Test Runs

1# 2 concurrent jobs with 10k holdings records + 2500 user records

Job #	Job description	Duration	Baseline job time
2 concurrent jobs
1	10k holding records	18 min 35 s	19 min 33 s
2	2500 user records	3 min 37 s	3 min 32 s

2# 2 concurrent jobs with 10k item records + 2500 user records

Job #	Job description	Duration	Baseline job time
2 concurrent jobs
1	10k item records	17 min 41 s	17 min 50 s
2	2500 user records	3 min 38 s	3 min 32 s

3# 3 concurrent jobs with 10k item records + 10k holdings records + 2500 user records

Job #	Job description	Job duration	Baseline job time
3 concurrent jobs
1	10k item records	17 min 43 s	17 min 50 s
2	10k holding records	18 min 33 s	19 min 33 s
3	2500 user records	3 min 34 s	3 min 32 s

4# 2 Concurrent jobs with 10k item records + 10k holdings records

Job #	Job description	Orchid Duration	Baseline job time
1	10k item records	18 min 19 s	17 min 50 s
2	10k holding records	18 min 35 s	19 min 33 s

5# 2 Concurrent jobs with 10k item records + 10k user records

Job #	Job description	Orchid Duration	Baseline job time
1	10k item records	17 min 49 s	17 min 50 s
2	10k user records	11 min 26 s	11 min 17 s

6# 2 Concurrent jobs with 10k user records + 10k holdings records (This job was the first job with user records testing after the mod-bulk-operations update)

Job #	Job description	Orchid Duration	Baseline job time
1	10k holding records	18 min 34 s	19 min 33 s
2	10k user records	13 min 9 s	11 min 17 s

7# 3 Concurrent jobs with 10k item records + 10k holdings records +10k user records

Job #	Job description	Orchid Duration	Baseline job time
1	10k item records	17 min 48 s	17 min 50 s
2	10k holding records	18 min 56 s	19 min 33 s
3	10k user records	11 min 45 s	11 min 17 s

Memory usage

The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing).

mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests.

For all other modules, no memory leaks are suspected.

Instance CPU utilization

Instances CPU utilization did not exceed 16%

Service CPU utilization

CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.

RDS CPU utilization

For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.

Errors in logs during testing

2023-03-22T19:47:19.986Z
19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
Field				Value
@ingestionTime		1679514444708
@log				054267740449:ncp5-folio-eis
@logStream			ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
@message			19:47:19 [] [] [] [] ERROR ?                    HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
@timestamp			1679514439986

2023-03-22T19:47:19.985Z
19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
Field				Value
@ingestionTime		1679514442775
@log				054267740449:ncp5-folio-eis
@logStream			ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
@message			19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
@timestamp			1679514439985

Appendix

Infrastructure

PTF -environment ncp5 [ environment name]

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters:

Module	SoftLimit	XMX	Revision	Version	desiredCount	CPUUnits	RWSplitEnabled	HardLimit	Metaspace	MaxMetaspaceSize
mod-inventory-storage-b	1952	1440	3	mod-inventory-storage:26.1.0-SNAPSHOT.644	2	1024	False	2208	384	512
mod-inventory-b	2592	1814	7	mod-inventory:20.0.0-SNAPSHOT.392	2	1024	False	2880	384	512
okapi-b	1440	922	1	okapi:5.1.0-SNAPSHOT.1352	3	1024	False	1684	384	512
mod-users-b	896	768	4	mod-users:19.2.0-SNAPSHOT.584	2	128	False	1024	88	128
mod-data-export-worker	2600	2048	3	mod-data-export-worker:3.0.0-SNAPSHOT.104	2	1024	False	3072	384	512
mod-data-export-spring	1844	1292	3	mod-data-export-spring:2.0.0-SNAPSHOT.67	1	256	False	2048	200	256
mod-bulk-operations	3864	0	10	mod-bulk-operations:1.0.2	2	400	False	4096	384	512
mod-notes	896	322	3	mod-notes:5.1.0-SNAPSHOT.245	2	128	False	1024	128	128
mod-agreements	2580	2048	3	mod-agreements:5.6.0-SNAPSHOT.117	2	128	False	3096	384	512
nginx-okapi	896	0	3	nginx-okapi:2022.03.02	2	128	False	1024	0	0

Combined Bulk Edit report [Orchid] 20/03/2023