Overview

Bulk Edits - Establish a performance baseline for Items bulk updates PERF-406 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure the performance did not deteriorate in comparison to Nolana release.  Some questions can help us to determine the performance and stability of the new Bulk Edits implementation:

• How long does it take to export 100, 1000, 10k, and 100K records?

• Can it be used with up to 5 concurrent users? 

• Run consecutively four jobs editing 10k  item records

• Run simultaneously four jobs editing 10k item records

• Look for a memory trend and CPU usage

Summary 

Test report for Bulk Edits items-app functionality 2023-03-08. 

Orchid release works 30% faster for 10k items bulk editing than Nolana.

It is approximately the same stable as Nolana.

• For 1 concurrent job 100 records can be edited in 1 min 9 s which is 3 times slower than in Nolana, 1000 records editing could be performed in approximately the same time as Nolana(2 min 40 s), and 10k records bulk editing is about 30% faster.

• 10k records per user, 5 users simultaneously (50k records total) can be uploaded and edited in about 20 minutes which is about 8 min faster compared to Nolana (about 28 min).

• The memory of mod-inventory-storage was high at 109% but stable (It was 109% even before the test). No memory leaks were found.

• CPU for mod-users was up to 125% (5 concurrent jobs 10k records updating) increased compared to Nolana (was about 40% with the same configurations) Need to be investigated in further testing. For all other modules did not exceed 65% for all of the tests.

• For all records number (100, 1k,10k), and 5 concurrent jobs - RDS CPU utilization did not exceed 60%.

Recommendations & Jiras

More than 50% of jobs FAILED in about 28-33 min with the error "Connection reset (SocketException)". https://folio-org.atlassian.net/browse/PERF-334

From time to time Job FAILED with the error from the s3 bucket https://folio-org.atlassian.net/browse/MODBULKOPS-76

The high CPU usage of mod-users (up to 125% ) needs to be investigated.

Results

Test Runs

1# One (concurrent) Job

2#  Items App 10k records 3, 4, and 5 concurrent jobs

3# Editing four jobs consecutively with 10k item records each

4# 5 Concurrent Item Apps jobs

 * "-" test was not performed due to errors that occurred

Memory usage

For all test runs

The memory of mod-inventory-storage was high at 109% but stable (It was 109% even before the test). No memory leaks were found.

Open

Instance CPU utilization

Run #1

Open

Run #2 & #3

Open

Service CPU utilization

Run #1

Open

Run #2 & #3

Open

RDS CPU utilization

Run #1

Open

Run #2 & #3

Maximum RDS CPU utilization is 61% for 5 concurrent jobs with 10k item records.

The more concurrent jobs are running -the higher RDS CPU usage - it looks like it should handle up to 7 concurrent jobs without any issues. The maximum number of jobs will be investigated. 

Open

Open

Appendix

Infrastructure

PTF -environment ncp5 [ environment name] 

• 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

• 2 instances of db.r6.xlarge database instances: Writer & reader instances

• MSK ptf-kakfa-3 [ kafka configurations]

  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

Modules memory and CPU parameters: