Overview

Per PERF-315 - Getting issue details... STATUS the following tests were performed using the data export workflow (bulk edits, eHoldings, and edifacts) to determine what load can the exports run successfully together 

Infrastructure

PTF -environment

• 9 m6i.2xlarge EC2 instances located in US West (Oregon)us-west-2 AWS region (comparing to 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 for Morning Glory release)
• 2 instances of db.r6.xlarge database instances, one reader, and one writer
• MSK ptf-kakfka-1
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=truec
  • log.retention.minutes=480
  • default.replication.factor=3

Modules memory and CPU parameters

MG- Morning Glory release

Summary

Test report for combined mod-data-export-worker functionality 2022-11-08. 

• Up to 10 different concurrent jobs can be performed successfully. Compared to individual bulk edit runs -10k records per user, 5 users simultaneously (50k records total) can be uploaded and updated in about 30 min which is successful. Bulk Edits with more than 10 concurrent jobs can fail with MODEXPW-306 - Getting issue details... STATUS or PERF-335 - Getting issue details... STATUS
• Memory trend: memory usage increases for mod-data-export-worker by 1% (from 109% to 110%).  Stable for all other modules.
• CPU - nginx-okapi is spiking up to 170% at the start of tests. For all other modules did not exceed 75%. CPU utilization increases proportionally to the number of jobs started.
• For all tests - RDS CPU utilization did not exceed 80%.

In general, Nolana is more stable compared to the Morning Glory release. All were tested with a higher load and  MODEXPW-306 - Getting issue details... STATUS occurs very rarely. As for PERF-335 - Getting issue details... STATUS , it can be reproduced only if mod-data-export-worker is overloaded and will be solved in UI.

Results

Test Runs (initial assumption of the data sets for combo testing)

Bulk edit files are located at https://github.com/folio-org/perf-testing/tree/Bulk-edit/workflows-scripts/Bulk-edit