Overview

Bulk Edit - Establish a performance baseline for Items bulk updates PERF-406

• How long does it take to export 100, 1000, 10k, and 100K records?
• Use it for up to 5 concurrent users.  
• Run consecutively four jobs editing 10k  item records
• Run simultaneously four jobs editing 10k item records
• Look for a memory trend and CPU usage

Summary 

Test report for Bulk Edits items-app functionality 2023-03-08. 

• For 1 concurrent job100 records can be edited in 1 min 9 s which is 3 times slower than in Nolana, 1000 records editing could be performed in approximately the same time as Nolana(2 min 40 s), and 10k records bulk editing is about 30% faster.
• 10k records per user, 5 users simultaneously (50k records total) can be uploaded and edited in about 20 minutes which is about 8 min faster compared to Nolana (about 28 min).
• The memory of mod-inventory-storage was high at 109% but stable. No memory leaks were found.
• CPU for mod-users was up to 125% (5 concurrent jobs 10k records updating) increased compared to Nolana (was about 40% with the same configurations). For all other modules did not exceed 65% for all of the tests.
• For all records number (100, 1k,10k), and 5 concurrent jobs - RDS CPU utilization did not exceed 60%.

Recommendations & Jiras

More than 50% of jobs FAILED in about 28-33 min PERF-334 - Getting issue details... STATUS

From time to time Job FAILED with the error from the s3 bucket MODBULKOPS-76 - Getting issue details... STATUS

Results

Test Runs

1#

2#

Items App 10k records 3, 4, and 5 concurrent jobs

3#

consecutively four jobs editing 10k  item records

4#

Items App 5 concurrent jobs

 * "-" test was not performed due to errors that occurred

Memory usage

For all test runs

The memory of mod-inventory-storage was high at 109% but stable. No memory leaks were found.

Instance CPU utilization

Run #1

Run #2 & #3

Service CPU utilization

Run #1

Run #2 & #3

RDS CPU utilization

Run #1

Run #2 & #3

Maximum RDS CPU utilization is 61% for 5 concurrent jobs with 10k item records.

The more concurrent jobs are running -the higher RDS CPU usage - it looks like it should handle up to 7 concurrent jobs without any issues. The maximum number of jobs will be investigated. 

Appendix

Infrastructure

PTF -environment ncp5 [ environment name] 

• 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
• 2 instances of db.r6.xlarge database instances: Writer & reader instances
• MSK ptf-kakfa-3 [ kafka configurations]
  
  • 4 kafka.m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Modules memory and CPU parameters: