...
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that different bulk edits can be performed simultaneously.
...
Test report for Bulk Edits holdings-app functionality 2023-03-22.
Orchid release works It shows good performance in combination of jobs with different numbers of records 2-3 different jobs simultaneously.
It was no failed jobs during testing.
- Jobs duration for combined tests with 2500 or 10k user records is approximately the same
...
- as
...
- the baseline number.
- The memory utilization of mod-bulk operation increases from 23% to 25% (the memory trend will be investigated in further testing). mod-users memory utilization increases from 45 to 47% during the first test with 10k users and was stable for all other tests. For all other modules, no memory leaks are suspected.
- CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
- For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Recommendations & Jiras
For further testing Users' bulk editing can be performed with 10k records.
Results
For all of the tests used the BARECODE identifier type for ITEMS & USERS and the HRID identifier type for Holdings.
...
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k holding records | 18 min 35 s | 19 min 33 s |
| 2 | 2500 user records | 3 min 37 s | 3 min 32 s |
2# 2 concurrent jobs with 10k item records + 2500 user records
| 2 concurrent jobs | |||
| Job # | Job description | Duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 41 s | 17 min 50 s |
2 | 2500 user records | 3 min 38 s | 3 min 32 s |
3# 3 concurrent jobs with 10k item records + 10k holdings records + 2500 user records
| 3 concurrent jobs | |||
| Job # | Job description | Job duration | Baseline job time |
|---|---|---|---|
| 1 | 10k item records | 17 min 43 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 33 s | 19 min 33 s |
| 3 | 2500 user records | 3 min 34 s | 3 min 32 s |
4# 2 Concurrent jobs with 10k item records + 10k holdings records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 18 min 19 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 35 s | 19 min 33 s |
5# 2 Concurrent jobs with 10k item records + 10k
...
user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 49 s | 17 min 50 s |
| 2 | 10k user records | 11 min 26 s | 11 min 17 s |
6# 2 Concurrent jobs with 10k
...
user records + 10k holdings records (This job was the first job with user records testing after the mod-bulk-operations update)
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k holding records | 18 min 34 s | 19 min 33 s |
2 | 10k user records | 13 min 9 s | 11 min 17 s |
7# 3 Concurrent jobs with 10k item records + 10k holdings records +10k user records
| Job # | Job description | Orchid Duration | Baseline job time |
| 1 | 10k item records | 17 min 48 s | 17 min 50 s |
| 2 | 10k holding records | 18 min 56 s | 19 min 33 s |
| 3 | 10k user records | 11 min 45 s | 11 min 17 s |
Memory usage
...
For all tests - RDS CPU utilization did not exceed 59% (for tests with 10k users records). For the tests with 2500 user records did not exceed 27%.
Errors in logs during testing
| Code Block |
|---|
2023-03-22T19:47:19.986Z
19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
Field Value
@ingestionTime 1679514444708
@log 054267740449:ncp5-folio-eis
@logStream ncp5/okapi-b/8dcac0276f1c46cba21d6e5814ec6cd0
@message 19:47:19 [] [] [] [] ERROR ? HTTP response code=404 msg=No suitable module found for path /holdings-sources/ for tenant fs09000000
@timestamp 1679514439986 |
| Code Block |
|---|
2023-03-22T19:47:19.985Z
19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
Field Value
@ingestionTime 1679514442775
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
@message 19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
@timestamp 1679514439985 |
| Code Block |
|---|
12023-03-23T17:21:26.756Z
17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
Field Value
@ingestionTime 1679592091556
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
@message 17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
@timestamp 1679592086756 |
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
...