Overview

Bulk Edits - Establish a performance baseline for Holdings bulk updates PERF-408 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure the performance did not deteriorate in comparison to Nolana release.  Some questions can help us to determine the performance and stability of the new Bulk Edits implementation:

• How long does it take to export 100, 1000, 10k, and 100K records?
• Can it be used with up to 5 concurrent users? 
• Run consecutively four jobs editing 10k holdings records
• Run simultaneously four jobs editing 10k holdings records
• Look for a memory trend and CPU usage

Summary 

Test report for Bulk Edits holdings-app functionality 2023-03-20. 

Orchid release works about 40% slower for holdings bulk editing than Nolana. One of the possible root causes of performance degradation could be a too long time to get a preview of changes. MODBULKOPS-86 - Getting issue details... STATUS

It is approximately the same stable as Nolana.

• Jobs duration: For 1 concurrent job, 100 records can be edited in 1 min 9 s which is 19 s slower compared to Nolana (50 s), and 1000 records editing could be performed in 2 min 54 s  which is 40 s slower compared to Nolana(2 min 10 s), and 10k records bulk editing is about 36% slower 100k records editing unavailable due to PERF-334 - Getting issue details... STATUS .
• 5 concurrent jobs run is successful: 10k records per user, 5 users simultaneously (50k records total) can be uploaded and edited in about 22 minutes which is about 9 min 30 s slower compared to Nolana (about 12 min 25 s). Slowness Could be a result of the changes UXPROD-3842 and MODBULKOPS-86 - Getting issue details... STATUS
• Run consecutively four jobs editing 10k Holdings records did not show any performance degradation.
• Simultaneously four jobs editing 10k holdings records could be run 20 min 40 s which is about 1 min or 5,7% slower than 1 concurrent job (19 min 33 s).
• The memory utilization of mod-bulk operation increases from 20% to 23% (The service was updated before the test, probably it is reaching a steady state- the memory trend will be investigated in further testing). For all other modules, no memory leaks are suspected.
• CPU for all modules did not exceed 56% for all of the tests. Compared to Nolana mod-data-export-worker has no spikes anymore and the average CPU utilization of other modules is approximately the same, except nginx-okapi - which is about 15% higher.
• For all records number (100, 1k,10k), and 5 concurrent jobs - RDS CPU utilization did not exceed 41%. Better compared to Nolana(it was up to 50%).

Recommendations & Jiras

More than 50% of jobs with 10k + records FAILED in about 30 min - 1 hour with the error "Connection reset (SocketException)". PERF-334 - Getting issue details... STATUS

Results

Test Runs

1# One (concurrent) Job

2#  Holdings App 10k records 1, 4, and 5 concurrent jobs

3# Editing four jobs consecutively with 10k holdings records each

4# 5 Concurrent Holdings Apps jobs

 * "-" test was not performed due to errors that occurred

Memory usage

For all test runs

The memory utilization of mod-bulk operation increases from 20% to 23% (The service was updated before the test, probably it is reaching a steady state- the memory trend will be investigated in further testing).

The memory of mod-data-export worker was 58 % at the beginning of the tests and grows after the tests finished then goes back to normal and was 55% at

 the end of testing (Looks like the work of the Garbage collectors).

Instance CPU utilization

Did not exceed 17%.

Service CPU utilization

CPU for all modules did not exceed 56% for all of the tests.

RDS CPU utilization

Maximum RDS CPU utilization is 41% for 5 concurrent jobs with 10k holdings records.

The more concurrent jobs are running -the higher RDS CPU usage. The maximum number of concurrent jobs will be investigated.