Bulk Edit Users App report [Orchid] September 2023
- 1 Overview
- 1.1 Summary.
- 1.2 Recommendations & Jiras
- 2 Test Runs
- 3 Errors in logs during testing
- 4 Appendix
- 4.1 Infrastructure
- 4.1.1 mod-agreements:5.5.2
- 4.1.2 mod-bulk-operations:1.0.6
- 4.1.3 mod-users-bl:7.5.0
- 4.1.4 mod-inventory-storage:26.0.0
- 4.1.5 mod-data-export-worker:3.0.13
- 4.1.6 mod-inventory:20.0.6
- 4.1.7 nginx-okapi:2023.06.14
- 4.1.8 okapi:5.0.1
- 4.1.9 mod-notes:5.0.1
- 4.1.10 mod-data-export-spring:2.0.2
- 4.1 Infrastructure
Overview
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that bulk edits can be performed simultaneously.
How long does it take to export 100, 1000, 2500, 5000 records?
Use it for up to 5 concurrent users.
Look for a memory trnend and CPU usage
Summary.
Test report for Bulk Edits users-app functionality 2023-09-15.
5k records per user, 5 users simultaneously (25k records total) can be uploaded in about 6 min 13 seconds,
Possible job duration degradation compared to NOLANA realese is caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.The files with identifiers should be strictly determined.
The memory of all modules during the tests for 5000 records with 5 parallel was stable, the memory gap in Figure 1 was caused by restarting several modules before DI job run.
Instance CPU usage
maximal value for text 4VU (0.1-1-2.5k-5k) was 26%
maximal value for text 5VU (0.1-1-2.5k-5k) was 27%
Service CPU usage for test 5VU (0.1-1-2.5k-5k)
CPU of mod-bulk-operations 126%, for all other modules, did not exceed 22%.
RDS CPU utilization did not exceed 43% for 5jobs 5000 records and 34% for 4jobs 5k.
Recommendations & Jiras
For further testing Users' bulk editing can be performed with 10k records
Test Runs
Total processing time of upload and edit - commit changes. Units =minutes:seconds
Number of virtual user/ Records | 1VU | 2VU | 3VU | 4VU | 5VU |
|---|---|---|---|---|---|
100 records | 00:01:05 | 00:01:03 00:01:03 | 00:01:03 | 00:01:03 | 00:01:04 |
1000 records | 00:01:37 | 00:01:59 | 00:02:02 | 00:02:02 | 00:02:11 |
2500 records | 00:03:28 | 00:03:30 | 00:03:42 | 00:00:23(*2) | 00:03:48 |
5000 records | 00:05:13 | 00:06:36 (*3) | 00:06:48 | 00:06:44 | 00:06:13 |
(*1) Duplicate barcodes in CSV input data, 1200 out of 2500 records were processed
(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException) 
(*3) Was running as a separate test to avoid duplication of input data
Comparison with previous results
5VU, | Nolana | Orchid |
|---|---|---|
2500 | 2 min 9 sec | 4 minutes 4 seconds |
5000 records | 3 min 47 sec | 6minutes 13 |
Possible job duration degradation caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.
Memory usage
The time range marked with # Several modules were restarted and then data import jobs were running on the environment (DI jobs were not part of Bulk Edits testing).
Memory usage during the testing with 1-5 concurrent and 100-1000-2500-5000 records
Memory usage during the testing with 2-5 concurrent and 5000 records
Memory usage for the same time range as the figure below, but only involved services were selected.
Instance CPU utilization
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
The test 5VU(0.1-1-2.5k-5k) was restarted because there were in the input data(Barcodes duplication)
CPU instance utilization for 2 up to 5 Vu with 5000 records
Service CPU utilization
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
CPU instance utilization for 2 up to 5 VU with 5000 records
RDS CPU utilization
For all tests - RDS CPU utilization exceeds 40% (for tests with 5k user records). For the tests with 2500 user records did not exceed 27%.
Database connections
Errors in logs during testing
(*1) Duplicate barcodes in CSV input data, 1200 out of 2500 records were processed
(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException)
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Modules memory and CPU parameters:
Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
ncp5-pvt | |||||||||
Wed Sep 13 09:54:48 UTC 2023 | |||||||||
mod-agreements | 6 | 2 | 1592 | 1488 | 128 | 968 | 384 | 512 | |
mod-bulk-operations | 17 | 2 | 3072 | 2600 | 1024 | 1536 | 384 | 512 | |
mod-users-bl | 8 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | |
mod-inventory-storage | 12 | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 | |
mod-data-export-worker | 10 | 1 | 3072 | 2800 | 1024 | 2048 | 384 | 512 | |
mod-inventory | 12 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | |
nginx-okapi | 7 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | |
okapi-b | 8 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | |
mod-notes | 6 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 | |
mod-data-export-spring | 8 | 1 | 2048 | 1844 | 256 | 1536 | 384 | 512 |