Combined Bulk Edit report [Orchid] 20/03/2023

Overview

Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously. 

Summary 

Test report for Bulk Edits holdings-app functionality 2023-03-22. 

Orchid release shows good performance in combination of jobs with different numbers of records 2-3 different jobs simultaneously.

It was no failed jobs during testing.

  • Jobs duration for combined tests with 2500 or 10k user records is approximately the same as the baseline number.
  • The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing). mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests. For all other modules, no memory leaks are suspected.
  • CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
  • For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.


Recommendations & Jiras

For further testing Users' bulk editing can be performed with 10k records.

Results

For all of the tests used the BARECODE identifier type for ITEMS & USERS  and the HRID identifier type for Holdings.

Test Runs

1# 2 concurrent jobs with 10k holdings records  + 2500 user records
2 concurrent jobs
Job #Job descriptionDurationBaseline job time
1

10k holding records

18 min 35 s19 min 33 s
2

2500 user records

3 min 37 s3 min 32 s
2#  2 concurrent jobs with 10k item records  + 2500 user records
2 concurrent jobs
Job #Job descriptionDurationBaseline job time
1

10k item records

17 min 41 s

17 min 50 s

2

2500 user records

3 min 38 s

3 min 32 s

3# 3 concurrent jobs with 10k item records + 10k holdings records  + 2500 user records
3 concurrent jobs
Job #Job descriptionJob durationBaseline job time
1

10k item records

17 min 43 s17 min 50 s
2

10k holding records

18 min 33 s19 min 33 s
3

2500 user records

3 min 34 s3 min 32 s
4# 2 Concurrent jobs with 10k item records + 10k holdings records
Job #Job descriptionOrchid DurationBaseline job time
1

10k item records

18 min 19 s17 min 50 s
2

10k holding records

18 min 35 s19 min 33 s
5# 2 Concurrent jobs with 10k item records + 10k user records
Job #Job descriptionOrchid DurationBaseline job time
1

10k item records

17 min 49 s17 min 50 s
2

10k user records

11 min 26 s11 min 17 s
6# 2 Concurrent jobs with 10k user records + 10k holdings records (This job was the first job with user records testing after the mod-bulk-operations update)
Job #Job descriptionOrchid DurationBaseline job time
1

10k holding records

18 min 34 s

19 min 33 s

2

10k user records

13 min 9 s11 min 17 s
7# 3 Concurrent jobs with 10k item records + 10k holdings records +10k user records
Job #Job descriptionOrchid DurationBaseline job time
1

10k item records

17 min 48 s17 min 50 s
2

10k holding records

18 min 56 s19 min 33 s
3

10k user records

11 min 45 s11 min 17 s

Memory usage

The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing).

mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests.

For all other modules, no memory leaks are suspected.

Instance CPU utilization

Instances CPU utilization did not exceed 16%


Service CPU utilization

CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.


RDS CPU utilization

For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.

Errors in logs during testing

2023-03-22T19:47:19.986Z
19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
Field				Value
@ingestionTime		1679514444708
@log				054267740449:ncp5-folio-eis
@logStream			ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
@message			19:47:19 [] [] [] [] ERROR ?                    HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
@timestamp			1679514439986
2023-03-22T19:47:19.985Z
19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
Field				Value
@ingestionTime		1679514442775
@log				054267740449:ncp5-folio-eis
@logStream			ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
@message			19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
@timestamp			1679514439985
12023-03-23T17:21:26.756Z
17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f

Field				Value
@ingestionTime		1679592091556
@log				054267740449:ncp5-folio-eis
@logStream			ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
@message			17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController   A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, typ