Combined Bulk Edit report [Orchid] 20/03/2023
- Olga Kondratenko
Overview
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously.
Summary
Test report for Bulk Edits holdings-app functionality 2023-03-22.
Orchid release shows good performance in combination of jobs with different numbers of records 2-3 different jobs simultaneously.
It was no failed jobs during testing.
- Jobs duration for combined tests with 2500 or 10k user records is approximately the same as the baseline number.
- The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing). mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests. For all other modules, no memory leaks are suspected.
- CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
- For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Recommendations & Jiras
For further testing Users' bulk editing can be performed with 10k records.
Results
For all of the tests used the BARECODE identifier type for ITEMS & USERS and the HRID identifier type for Holdings.
Test Runs
1# 2 concurrent jobs with 10k holdings records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k holding records | 18 min 35 s | 19 min 33 s |
| 2 | 2500 user records | 3 min 37 s | 3 min 32 s |
2# 2 concurrent jobs with 10k item records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 41 s | 17 min 50 s |
2 | 2500 user records | 3 min 38 s | 3 min 32 s |
3# 3 concurrent jobs with 10k item records + 10k holdings records + 2500 user records
| 3 concurrent jobs | |||
| Job # | Job description | Job duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 43 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 33 s | 19 min 33 s |
| 3 | 2500 user records | 3 min 34 s | 3 min 32 s |
4# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 18 min 19 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 35 s | 19 min 33 s |
5# 2 Concurrent jobs with 10k item records + 10k user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 49 s | 17 min 50 s |
| 2 | 10k user records | 11 min 26 s | 11 min 17 s |
6# 2 Concurrent jobs with 10k user records + 10k holdings records (This job was the first job with user records testing after the mod-bulk-operations update)
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k holding records | 18 min 34 s | 19 min 33 s |
2 | 10k user records | 13 min 9 s | 11 min 17 s |
7# 3 Concurrent jobs with 10k item records + 10k holdings records +10k user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 48 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 56 s | 19 min 33 s |
| 3 | 10k user records | 11 min 45 s | 11 min 17 s |
Memory usage
The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing).
mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests.
For all other modules, no memory leaks are suspected.
Instance CPU utilization
Instances CPU utilization did not exceed 16%
Service CPU utilization
CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
RDS CPU utilization
For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Errors in logs during testing
2023-03-22T19:47:19.986Z 19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000 ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0 Field Value @ingestionTime 1679514444708 @log 054267740449:ncp5-folio-eis @logStream ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0 @message 19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000 @timestamp 1679514439986
2023-03-22T19:47:19.985Z
19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
Field Value
@ingestionTime 1679514442775
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
@message 19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
@timestamp 1679514439985
12023-03-23T17:21:26.756Z
17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
Field Value
@ingestionTime 1679592091556
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
@message 17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
@timestamp 1679592086756
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Module | SoftLimit | XMX | Revision | Version | desiredCount | CPUUnits | RWSplitEnabled | HardLimit | Metaspace | MaxMetaspaceSize |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage-b | 1952 | 1440 | 3 | mod-inventory-storage:26.1.0-SNAPSHOT.644 | 2 | 1024 | False | 2208 | 384 | 512 |
| mod-inventory-b | 2592 | 1814 | 7 | mod-inventory:20.0.0-SNAPSHOT.392 | 2 | 1024 | False | 2880 | 384 | 512 |
| okapi-b | 1440 | 922 | 1 | okapi:5.1.0-SNAPSHOT.1352 | 3 | 1024 | False | 1684 | 384 | 512 |
| mod-users-b | 896 | 768 | 4 | mod-users:19.2.0-SNAPSHOT.584 | 2 | 128 | False | 1024 | 88 | 128 |
| mod-data-export-worker | 2600 | 2048 | 3 | mod-data-export-worker:3.0.0-SNAPSHOT.104 | 2 | 1024 | False | 3072 | 384 | 512 |
| mod-data-export-spring | 1844 | 1292 | 3 | mod-data-export-spring:2.0.0-SNAPSHOT.67 | 1 | 256 | False | 2048 | 200 | 256 |
| mod-bulk-operations | 3864 | 0 | 10 | mod-bulk-operations:1.0.2 | 2 | 400 | False | 4096 | 384 | 512 |
| mod-notes | 896 | 322 | 3 | mod-notes:5.1.0-SNAPSHOT.245 | 2 | 128 | False | 1024 | 128 | 128 |
| mod-agreements | 2580 | 2048 | 3 | mod-agreements:5.6.0-SNAPSHOT.117 | 2 | 128 | False | 3096 | 384 | 512 |
| nginx-okapi | 896 | 0 | 3 | nginx-okapi:2022.03.02 | 2 | 128 | False | 1024 | 0 | 0 |