[Sunflower] [non-ECS] Bulk Edit Parameters Testing with Item records
Overview
- This document summarizes the results of Bulk Edit parameter testing performed changing the mod-data-export-worker task definition parameters. Various parameter configurations were evaluated using a file with 50,000 item records. The test scenarios involved applying a single change to each record—specifically, the addition of an extra note to Note field.
- Sunflower release on a non-ecs environment ( secp1 in our case, central tenant - fs09000000).
- Test results contain two parts. First is file upload and retrieving data. The second part is applying changes. Parameter value changes intend to help reduce duration of the first part of test.
- Jira PERF-1164 - Getting issue details... STATUS
Summary
- All tests completed successfully.
- Best results achieved in test number 5 with MERGE_CSV_JSON_MRC_POOL_SIZE parameter equal 2. Combining parameters with best cases reveal negative effect. All other test results and observations exposed in the table below:
| Parameters | Tests | Total Duration (min) | Notes |
|---|---|---|---|
| QUANTITY_PER_PARTITION | Test2–4 | 49, 50, 48 | Slightly better than baseline; stable performance |
| MERGE_CSV_JSON_MRC_POOL_SIZE | Test5–7 | 47, 62, 56 | High variability; possible tuning impact |
| NUM_PARTITIONS | Test8–10 | 67, 51, 53 | One test caused significant slowdown (Test8: 67) |
| DB_POOL_SIZE | Test11–13 | 52, 52, 51 | Very consistent results; no performance gain |
| QUANTITY_PER_PARTITION + MERGE_CSV_JSON_MRC_POOL_SIZE | Test15 | 62 | Combined change shows degraded performance |
| All Parameters Combined | Test14 | 72 | Worst performance; suggests negative cumulative effect |
- The 50k file upload and retrieving data part perform better in test #4 - 4 minutes. Test #5 show the same results as default - 5 minutes.
- Memory consumption for all modules shows a stable trend. No memory leaks are suspected.
- Service CPU utilization corresponded to the load.
- Top 5 services CPU utilization: mod-inventory-storage-b=121%, mod-inventory-b=53%, mod-data-export-worker-b=31%, mod-inn-reach-b=13%, mod-audit-b=10%
- Service memory consumption
- mod-bulk-operations-b=61%, edge-users-b=42%, mod-search-b=40%, mod-data-export-worker-b=40%, mod-source-record-storage-b=34%, mod-inventory-b=33%, mod-data-export-b=29%, mod-circulation-b=26%, mod-inventory-storage-b=25%, kong-b=12%
- DB CPU utilization was 29% mostly. Combined scenario and test #6 which is performed worse - 36%. During test #6 there's a spike to 52% once after data retrieved and before applying changes.
- DB connections - 1280 all tests.
- Retest of test #14 was conducted only for the first part - retrieving data. It took 5 minutes and 35 seconds. Retest of test #5 took 4 minutes and 40 seconds. So cumulative negative affect confirmed. Degradation in average - 1 minute or 20%.
Jiras
Test description
| Identifier | Record identifier | Number of records | Action | mod-data-export-worker parameter | records source |
|---|---|---|---|---|---|
| Inventory-Items | Item barcodes | 50 000 |
| Applying single variable each test QUANTITY_PER_PARTITION MERGE_CSV_JSON_MRC_POOL_SIZE NUM_PARTITIONS DB_POOL_SIZE | from .csv file |
Test runs
| Tests | Task definition | mod-data-export-worker parameter | Value |
|---|---|---|---|
| Test #1 | Default values | QUANTITY_PER_PARTITION | 5000 |
| MERGE_CSV_JSON_MRC_POOL_SIZE | 3 | ||
| NUM_PARTITIONS | 8 | ||
| DB_POOL_SIZE | 20 | ||
| Test #2 | secp1-mod-data-export-worker:10 | QUANTITY_PER_PARTITION | 3000 |
| Test #3 | secp1-mod-data-export-worker:11 | QUANTITY_PER_PARTITION | 4000 |
| Test #4 | secp1-mod-data-export-worker:12 | QUANTITY_PER_PARTITION | 7000 |
| Test #5 | secp1-mod-data-export-worker:14 | MERGE_CSV_JSON_MRC_POOL_SIZE | 2 |
| Test #6 | secp1-mod-data-export-worker:15 | MERGE_CSV_JSON_MRC_POOL_SIZE | 5 |
| Test #7 | secp1-mod-data-export-worker:16 | MERGE_CSV_JSON_MRC_POOL_SIZE | 7 |
| Test #8 | secp1-mod-data-export-worker:17 | NUM_PARTITIONS | 2 |
| Test #9 | secp1-mod-data-export-worker:18 | NUM_PARTITIONS | 10 |
| Test #10 | secp1-mod-data-export-worker:19 | NUM_PARTITIONS | 15 |
| Test #11 | secp1-mod-data-export-worker:20 | DB_POOL_SIZE | 15 |
| Test #12 | secp1-mod-data-export-worker:21 | DB_POOL_SIZE | 30 |
| Test #13 | secp1-mod-data-export-worker:22 | DB_POOL_SIZE | 40 |
| Test #14 | secp1-mod-data-export-worker:23 | QUANTITY_PER_PARTITION | 7000 |
| MERGE_CSV_JSON_MRC_POOL_SIZE | 2 | ||
| NUM_PARTITIONS | 10 | ||
| DB_POOL_SIZE | 40 | ||
| Test #15 | secp1-mod-data-export-worker:24 | QUANTITY_PER_PARTITION | 7000 |
| MERGE_CSV_JSON_MRC_POOL_SIZE | 2 | ||
| Test #16 - retest #14 | secp1-mod-data-export-worker:23 |
Test Results
| Test # | Parameter change | Upload Duration (minutes) | Applying changes Duration (minutes) | Total time | Results |
|---|---|---|---|---|---|
| Test1 | default | 5 | 47 | 52 | Completed |
| Test2 | QUANTITY_PER_PARTITION | 6 | 43 | 49 | Completed |
| Test3 | QUANTITY_PER_PARTITION | 7 | 43 | 50 | Completed |
| Test4 | QUANTITY_PER_PARTITION | 5 | 43 | 48 | Completed |
| Test5 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 43 | 47 | Completed |
| Test6 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 58 | 62 | Completed |
| Test7 | MERGE_CSV_JSON_MRC_POOL_SIZE | 4 | 52 | 56 | Completed |
| Test8 | NUM_PARTITIONS | 7 | 60 | 67 | Completed |
| Test9 | NUM_PARTITIONS | 5 | 46 | 51 | Completed |
| Test10 | NUM_PARTITIONS | 5 | 48 | 53 | Completed |
| Test11 | DB_POOL_SIZE | 5 | 47 | 52 | Completed |
| Test12 | DB_POOL_SIZE | 5 | 47 | 52 | Completed |
| Test13 | DB_POOL_SIZE | 5 | 46 | 51 | Completed |
| Test14 | QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE/NUM_PARTITIONS/DB_POOL_SIZE | 7 | 65 | 72 | Completed |
| Test15 | QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE | 7 | 55 | 62 | Completed |
| Test16 | Retest #14 | 6 | not tested | Completed |
Resource utilization
Service CPU Utilization
Service Memory Utilization
Tests #1 - #7
Tests #8 - #14
DB CPU Utilization
Tests #1 - #7
Tests #8 - #14
DB Connections
Tests #1 - #7
Tests #8 - #14
DB load
Tests #1 - #7
Tests #8 - #14
Top SQL-queries
Tests #1 - #7
Tests #8 - #14
AppendixInfrastructure
PTF - environment Ramsons (secp1)
- 14 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance, writer
Name Memory GIB vCPUs db.r6g.xlarge
32 GiB 4 vCPUs - MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=2
Methodology/Approach
Test set
Data preparation: prepare . csv file with 50000 item barcodes for testing
The script to prepare files is here.
In case of testing with instances - populate the database with FOLIO records if needed with this script.
- Manually tested. All tests triggered on one tenant only:
- Trigger bulk edit by submitting identifiers for 50000 item records
- Wait until Bulk edit finds all matched records and record the startTime and endTime of the process from the response.
- Edit instances and start the second part of the bulk edit to commit the changes.
- Record the start time of committing changes from Headers
- Database preparation script after the test run required to restore Suppress from discovery and Staff suppress parameters.
Additional screenshots
, multiple selections available,