[Sunflower] [non-ECS] Bulk Edit Parameters Testing with Item records
Overview
This document summarizes the results of Bulk Edit parameter testing performed changing the mod-data-export-worker task definition parameters. Various parameter configurations were evaluated using a file with 50,000 item records. The test scenarios involved applying a single change to each record—specifically, the addition of an extra note to Note field.
Sunflower release on a non-ecs environment ( secp1 in our case, central tenant - fs09000000).
Test results contain two parts. First is file upload and retrieving data. The second part is applying changes. Parameter value changes intend to help reduce duration of the first part of test.
Summary
All tests completed successfully.
Best results achieved in test number 5 with MERGE_CSV_JSON_MRC_POOL_SIZE parameter equal 2. Combining parameters with best cases reveal negative effect. All other test results and observations exposed in the table below:
Parameters | Tests | Total Duration (min) | Notes |
|---|---|---|---|
QUANTITY_PER_PARTITION | Test2–4 | 49, 50, 48 | Slightly better than baseline; stable performance |
MERGE_CSV_JSON_MRC_POOL_SIZE | Test5–7 | 47, 62, 56 | High variability; possible tuning impact |
NUM_PARTITIONS | Test8–10 | 67, 51, 53 | One test caused significant slowdown (Test8: 67) |
DB_POOL_SIZE | Test11–13 | 52, 52, 51 | Very consistent results; no performance gain |
QUANTITY_PER_PARTITION + MERGE_CSV_JSON_MRC_POOL_SIZE | Test15 | 62 | Combined change shows degraded performance |
All Parameters Combined | Test14 | 72 | Worst performance; suggests negative cumulative effect |
The 50k file upload and retrieving data part perform better in test #4 - 4 minutes. Test #5 show the same results as default - 5 minutes.
Memory consumption for all modules shows a stable trend. No memory leaks are suspected.
Service CPU utilization corresponded to the load.
Top 5 services CPU utilization: mod-inventory-storage-b=121%, mod-inventory-b=53%, mod-data-export-worker-b=31%, mod-inn-reach-b=13%, mod-audit-b=10%
Service memory consumption
mod-bulk-operations-b=61%, edge-users-b=42%, mod-search-b=40%, mod-data-export-worker-b=40%, mod-source-record-storage-b=34%, mod-inventory-b=33%, mod-data-export-b=29%, mod-circulation-b=26%, mod-inventory-storage-b=25%, kong-b=12%
DB CPU utilization was 29% mostly. Combined scenario and test #6 which is performed worse - 36%. During test #6 there's a spike to 52% once after data retrieved and before applying changes.
DB connections - 1280 all tests.
Retest of test #14 was conducted only for the first part - retrieving data. It took 5 minutes and 35 seconds. Retest of test #5 took 4 minutes and 40 seconds. So cumulative negative affect confirmed. Degradation in average - 1 minute or 20%.
Jiras
Test description
Identifier | Record identifier | Number of records | Action | mod-data-export-worker parameter | records source |
|---|---|---|---|---|---|
Inventory-Items | Item barcodes | 50 000 |
| Applying single variable each test QUANTITY_PER_PARTITION MERGE_CSV_JSON_MRC_POOL_SIZE NUM_PARTITIONS DB_POOL_SIZE | from .csv file |
Test runs
Tests | Task definition | mod-data-export-worker parameter | Value |
|---|---|---|---|
Test #1 | Default values | QUANTITY_PER_PARTITION | 5000 |
MERGE_CSV_JSON_MRC_POOL_SIZE | 3 | ||
NUM_PARTITIONS | 8 | ||
DB_POOL_SIZE | 20 | ||
Test #2 | secp1-mod-data-export-worker:10 | QUANTITY_PER_PARTITION | 3000 |
Test #3 | secp1-mod-data-export-worker:11 | QUANTITY_PER_PARTITION | 4000 |
Test #4 | secp1-mod-data-export-worker:12 | QUANTITY_PER_PARTITION | 7000 |
Test #5 | secp1-mod-data-export-worker:14 | MERGE_CSV_JSON_MRC_POOL_SIZE | 2 |
Test #6 | secp1-mod-data-export-worker:15 | MERGE_CSV_JSON_MRC_POOL_SIZE | 5 |
Test #7 | secp1-mod-data-export-worker:16 | MERGE_CSV_JSON_MRC_POOL_SIZE | 7 |
Test #8 | secp1-mod-data-export-worker:17 | NUM_PARTITIONS | 2 |
Test #9 | secp1-mod-data-export-worker:18 | NUM_PARTITIONS | 10 |
Test #10 | secp1-mod-data-export-worker:19 | NUM_PARTITIONS | 15 |
Test #11 | secp1-mod-data-export-worker:20 | DB_POOL_SIZE | 15 |
Test #12 | secp1-mod-data-export-worker:21 | DB_POOL_SIZE | 30 |
Test #13 | secp1-mod-data-export-worker:22 | DB_POOL_SIZE | 40 |
Test #14 | secp1-mod-data-export-worker:23 | QUANTITY_PER_PARTITION | 7000 |
MERGE_CSV_JSON_MRC_POOL_SIZE | 2 | ||
NUM_PARTITIONS | 10 | ||
DB_POOL_SIZE | 40 | ||
Test #15 | secp1-mod-data-export-worker:24 | QUANTITY_PER_PARTITION | 7000 |
MERGE_CSV_JSON_MRC_POOL_SIZE | 2 | ||
Test #16 - retest #14 | secp1-mod-data-export-worker:23 |
|
|
Test Results
Test # | Parameter change | Upload Duration (minutes) | Applying changes Duration (minutes) | Total time | Results |
|---|---|---|---|---|---|
Test1 | default | 5 | 47 | 52 | Completed |
Test2 | QUANTITY_PER_PARTITION | 6 | 43 | 49 | Completed |
Test3 | QUANTITY_PER_PARTITION | 7 | 43 | 50 | Completed |
Test4 | QUANTITY_PER_PARTITION | 5 | 43 | 48 | Completed |
Test5 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 43 | 47 | Completed |
Test6 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 58 | 62 | Completed |
Test7 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 52 | 56 | Completed |
Test8 | NUM_PARTITIONS | 7 | 60 | 67 | Completed |
Test9 | NUM_PARTITIONS | 5 | 46 | 51 | Completed |
Test10 | NUM_PARTITIONS | 5 | 48 | 53 | Completed |
Test11 | DB_POOL_SIZE | 5 | 47 | 52 | Completed |
Test12 | DB_POOL_SIZE | 5 | 47 | 52 | Completed |
Test13 | DB_POOL_SIZE | 5 | 46 | 51 | Completed |
Test14 | QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE/NUM_PARTITIONS/DB_POOL_SIZE | 7 | 65 | 72 | Completed |
Test15 | QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE | 7 | 55 | 62 | Completed |
Test16 | Retest #14 | 6 | not tested |
| Completed |
Resource utilization
Service CPU Utilization
Service Memory Utilization
Tests #1 - #7
Tests #8 - #14
DB CPU Utilization
Tests #1 - #7
Tests #8 - #14
DB Connections
Tests #1 - #7
Tests #8 - #14
DB load
Tests #1 - #7
Tests #8 - #14
Top SQL-queries
Tests #1 - #7
Tests #8 - #14
Appendix
Infrastructure
PTF - environment Ramsons (secp1)
14 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
MSK fse-test
4 kafka.m7g.xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=2