Overview

Bulk Edit - Establish a performance baseline for user status bulk updates. PERF-750: [Poppy] [Bulk edit] Item recordsClosedPreview

• How long does it take to export 100, 1000, 5000, 10k, and 100K records?

• Use it for up to 5 concurrent users.  

• Look for a memory trend and CPU usage

• Pay attention to any RTR token expiration messages and observe how/if BE is affected by expiring tokens. If needed set the Access token's expiration time to 300s or less to trigger the Access token's quick expiration. 

Summary 

• All tests were successful, and 100K records files after bulk-edit were downloaded(test completed successfully). System has not reached maximum capacity therefore, the number of VU can be increased to 7.

• No errors with messages like "Invalid token" or messages like "Access token has expired" during 2 tests.

• With an increase in the number of virtual users, the operating time of bulk-edit increases(All results are in the first table).

• Comparing the test results on Poppy and Orchid releases, we can conclude that the processing time increased by up to 10%(However, the previous report lacked data on the duration of Bulk-edit ).

• The system was stable during the test. Maximal resource utilization was during 100K records bulk-edit and 5 VU concurrently.

  • Max CPU utilization was on nginx-okapi(60%) and okapi(50%) services. 

  • Service memory usage was without memory leaks. Except the mod-search service during Test 2.

  • Average DB CPU usage was about 63%

  • Number of DB connections ~ 200

• Comparing the resource utilization graphs, we can say that the system behavior on the Poppy release is the same as on Orchid.

• From Orchid JIRAS "The high CPU usage of mod-users (up to 125% ) needs to be investigated."  During 2 tests max CPU consumption was about 40% for mod-users service.
  
  

  Recommendations & Jiras
  
  

• The high memory usage of mod-search service during test 2, needs to be investigated
  

Results

Total processing time of upload and edit - commit changes. Units =hours:minutes: seconds

Comparison table of bulk-edit process duration on Poppy and Orchid releases

Link to the report with Orchid items apps testing results Bulk Edit Items App report [Orchid] 08/03/2023

Resource utilization

Instance CPU Utilization

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. From the Instance CPU utilization graph, you can see that the bulk-edit process consists of 2 parts: file uploading(90-120 minutes) and records processing.
*During the Bulk-edit for 4 VU, one process was uploading 100K records file for more than 4 hours( jobs finished processing at about 21:30, and the grey graph was running up to 2 a.m.), but on all other bulk-edits jobs, there were no problems with this file.

Test 2.   Bulk-edit 100-1000-5000-10k records successively from 1VU up to 5VU. Blurred areas contain errors from the load generator.

Open

Memory usage

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. 

Test 2.   Bulk-edit 100-1000-5000-10k records successively from 1VU up to 5VU

Service CPU usage

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. 

Open

Test 2.   Bulk-edit 100-1000-5000-10k records successively from 1VU up to 5VU

Open

RDS CPU utilization

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU.  Average CPU for 1VU ~ 32%; 2VU ~ 43%; 3VU ~ 50%; 4VU ~ 52%; 5VU ~ 63%; 

Open

Test 2.   Bulk-edit 100-1000-5000-10k records successively from 1VU up to 5VU. Average CPU for 1VU ~ 40%; 2VU ~ 51%; 3VU ~ 53%; 4VU ~ 62%; 5VU ~ 80%; 

Open

Database connections

Test 1. Bulk-edit 5 consecutive runs of 100K records, starting from 1VU up to 5VU. The number of connections to the database did not exceed 200

Open

Test 2.   Bulk-edit 100-1000-5000-10k records successively from 1VU up to 5VU. The number of connections to the database did not exceed 200

Open

Database load 

Part of test 1. Bulk-edit 100K records 5VU. 

Open

TOP SQL Queries

Appendix

Infrastructure

PTF -environment pcp1

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 instances of db.r6.xlarge database instances, one reader, and one writer

• MSK tenant

  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

Modules memory and CPU parameters