...
Test report for Bulk Edits items-app functionality 2023-03-08.
- For 1 concurrent job100 records can be edited in 1 min 9 s which is 3 times slower than in Nolana, 1000 records editing could be performed in approximately the same time as Nolana(2 min 40 s), and 10k records bulk editing is about 30% faster.
- 10k records per user, 5 users simultaneously (50k records total) can be uploaded and edited in about 20 minutes which is about 8 min faster compared to Nolana (about 28 min).
- The memory of mod-inventory-storage was high at 109% but stable. No memory leaks were found.
- CPU for mod-users was up to 125% (5 concurrent jobs 10k records updating) increased compared to Nolana (was about 40% with the same configurations). For all other modules did not exceed 65% for all of the tests.
- For all records number (100, 1k,10k), and 5 concurrent jobs - RDS CPU utilization did not exceed 60%.
Recommendations & Jiras
In More than 50% of jobs FAILED in about 28-33 min
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
From time to time Job FAILED with the error from the s3 bucket
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Results
Test Runs
1#
| 1 concurrent job | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Number of records | Duration | Comments | ||||||||
| 100 | 1 min 9 s | |||||||||
| 1000 | 2 min 36 s | |||||||||
| 10k | 17 min 50 s | |||||||||
| 50k | 1 hour 58 min | or Error in about 28-33 min Connection reset (SocketException)
| ||||||||
| 100k | always FAILD | Error in about 28-33 min Connection reset (SocketException)
| ||||||||
...
The more concurrent jobs are running -the higher RDS CPU usage - it looks like it should handle up to 7 concurrent jobs without any issues. The maximum number of jobs will be investigated.
The more concurrent jobs are running -the higher RDS CPU usage - it looks like it should handle up to 7 concurrent jobs without any issues. The maximum number of jobs will be investigated.
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
- 8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
- 2 instances of db.r6.xlarge database instances: Writer & reader instances
- MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
Modules memory and CPU parameters:
Module | SoftLimit | XMX | Revision | Version | desiredCount | CPUUnits | RWSplitEnabled | HardLimit | Metaspace | MaxMetaspaceSize |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage-b | 1952 | 1440 | 3 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.1.0-SNAPSHOT.644 | 2 | 1024 | False | 2208 | 384 | 512 |
| mod-inventory-b | 2592 | 1814 | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.0-SNAPSHOT.392 | 2 | 1024 | False | 2880 | 384 | 512 |
| okapi-b | 1440 | 922 | 1 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.1.0-SNAPSHOT.1352 | 3 | 1024 | False | 1684 | 384 | 512 |
| mod-users-b | 896 | 768 | 4 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.2.0-SNAPSHOT.584 | 2 | 128 | False | 1024 | 88 | 128 |
| mod-data-export-worker | ||||||||||
| mod-data-export-spring | ||||||||||
| mod-bulk-operations | ||||||||||
| mod-notes | ||||||||||
| mod-agreements | ||||||||||
| nginx-okapi |
...