[Sunflower] [non-ECS] Bulk Edit Parameters Testing with Item records

[Sunflower] [non-ECS] Bulk Edit Parameters Testing with Item records

Overview

  • This document summarizes the results of Bulk Edit parameter testing performed changing the mod-data-export-worker task definition parameters. Various parameter configurations were evaluated using a file with 50,000 item records. The test scenarios involved applying a single change to each record—specifically, the addition of an extra note to Note field.
  •  Sunflower release on a non-ecs environment ( secp1 in our case, central tenant - fs09000000).
  • Test results contain two parts. First is file upload and retrieving data. The second part is applying changes. Parameter value changes intend to help reduce duration of the first part of test.
  • Jira PERF-1164 - Getting issue details... STATUS

Summary

  • All tests completed successfully.
  • Best results achieved in test number 5 with MERGE_CSV_JSON_MRC_POOL_SIZE parameter equal 2. Combining parameters with best cases reveal negative effect. All other test results and observations exposed in the table below:
ParametersTestsTotal Duration (min)Notes
QUANTITY_PER_PARTITIONTest2–449, 50, 48Slightly better than baseline; stable performance
MERGE_CSV_JSON_MRC_POOL_SIZETest5–747, 62, 56High variability; possible tuning impact
NUM_PARTITIONSTest8–1067, 51, 53One test caused significant slowdown (Test8: 67)
DB_POOL_SIZETest11–1352, 52, 51Very consistent results; no performance gain
QUANTITY_PER_PARTITION + MERGE_CSV_JSON_MRC_POOL_SIZETest1562Combined change shows degraded performance
All Parameters CombinedTest1472Worst performance; suggests negative cumulative effect
  • The 50k file upload and retrieving data part perform better in test #4 - 4 minutes. Test #5 show the same results as default - 5 minutes.
  • Memory consumption for all modules shows a stable trend. No memory leaks are suspected.
  • Service CPU utilization corresponded to the load.
    • Top 5 services CPU utilization: mod-inventory-storage-b=121%, mod-inventory-b=53%, mod-data-export-worker-b=31%, mod-inn-reach-b=13%, mod-audit-b=10%
  • Service memory consumption
    • mod-bulk-operations-b=61%, edge-users-b=42%, mod-search-b=40%, mod-data-export-worker-b=40%, mod-source-record-storage-b=34%, mod-inventory-b=33%, mod-data-export-b=29%, mod-circulation-b=26%, mod-inventory-storage-b=25%, kong-b=12%
  • DB CPU utilization was 29% mostly. Combined scenario and test #6 which is performed worse  - 36%. During test #6 there's a spike to 52% once after data retrieved and before applying changes.
  • DB connections - 1280 all tests.
  • Retest of test #14 was conducted only for the first part - retrieving data. It took 5 minutes and 35 seconds. Retest of test #5 took 4 minutes and 40 seconds. So cumulative negative affect confirmed. Degradation in average - 1 minute or 20%.

Jiras


Test description

IdentifierRecord identifierNumber of recordsActionmod-data-export-worker parameterrecords source
Inventory-ItemsItem barcodes50 000
  1. Adding a note "ptf_test" in Note field

Applying single variable each test 

QUANTITY_PER_PARTITION

MERGE_CSV_JSON_MRC_POOL_SIZE

NUM_PARTITIONS

DB_POOL_SIZE

from .csv file

Test runs

TestsTask definitionmod-data-export-worker parameterValue
Test #1


Default values



QUANTITY_PER_PARTITION5000
MERGE_CSV_JSON_MRC_POOL_SIZE3
NUM_PARTITIONS8
DB_POOL_SIZE20
Test #2 secp1-mod-data-export-worker:10QUANTITY_PER_PARTITION3000
Test #3secp1-mod-data-export-worker:11QUANTITY_PER_PARTITION4000
Test #4secp1-mod-data-export-worker:12QUANTITY_PER_PARTITION7000
Test #5secp1-mod-data-export-worker:14MERGE_CSV_JSON_MRC_POOL_SIZE2
Test #6secp1-mod-data-export-worker:15MERGE_CSV_JSON_MRC_POOL_SIZE5
Test #7secp1-mod-data-export-worker:16MERGE_CSV_JSON_MRC_POOL_SIZE7
Test #8secp1-mod-data-export-worker:17NUM_PARTITIONS2
Test #9secp1-mod-data-export-worker:18NUM_PARTITIONS10
Test #10secp1-mod-data-export-worker:19NUM_PARTITIONS15
Test #11secp1-mod-data-export-worker:20DB_POOL_SIZE15
Test #12secp1-mod-data-export-worker:21DB_POOL_SIZE30
Test #13secp1-mod-data-export-worker:22DB_POOL_SIZE40
Test #14


secp1-mod-data-export-worker:23


QUANTITY_PER_PARTITION7000
MERGE_CSV_JSON_MRC_POOL_SIZE2
NUM_PARTITIONS10
DB_POOL_SIZE40
Test #15secp1-mod-data-export-worker:24QUANTITY_PER_PARTITION7000
MERGE_CSV_JSON_MRC_POOL_SIZE2
Test #16 - retest #14secp1-mod-data-export-worker:23

Test Results

Test #Parameter changeUpload Duration (minutes)Applying changes  Duration  (minutes)Total timeResults
Test1default54752Completed
Test2QUANTITY_PER_PARTITION64349Completed
Test3QUANTITY_PER_PARTITION74350Completed
Test4QUANTITY_PER_PARTITION54348Completed
Test5MERGE_CSV_JSON_MRC_POOL_SIZE44347Completed
Test6MERGE_CSV_JSON_MRC_POOL_SIZE45862Completed
Test7MERGE_CSV_JSON_MRC_POOL_SIZE45256Completed
Test8NUM_PARTITIONS76067Completed
Test9NUM_PARTITIONS54651Completed
Test10NUM_PARTITIONS54853Completed
Test11DB_POOL_SIZE54752Completed
Test12DB_POOL_SIZE54752Completed
Test13DB_POOL_SIZE54651Completed
Test14QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE/NUM_PARTITIONS/DB_POOL_SIZE76572Completed
Test15QUANTITY_PER_PARTITION/MERGE_CSV_JSON_MRC_POOL_SIZE75562Completed
Test16Retest #146not tested
Completed


Resource utilization

Service CPU Utilization


Service Memory Utilization

Tests #1 - #7

Tests #8 - #14

DB CPU Utilization

Tests #1 - #7

Tests #8 - #14

DB Connections

Tests #1 - #7

Tests #8 - #14

DB load

Tests #1 - #7

Tests #8 - #14

Top SQL-queries

Tests #1 - #7

Tests #8 - #14


Appendix

Infrastructure

PTF - environment Ramsons (secp1)

  • 14  r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database instance, writer

    NameMemory GIBvCPUs

    db.r6g.xlarge

    32 GiBvCPUs
  • MSK fse-test
    • 4 kafka.m7g.xlarge brokers in 2 zones
    • Apache Kafka version 3.7.x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=2


 Sunflower services resources
Cluster Resources - secp1-pvt (Mon Aug 04 08:03:13 UTC 2025)


















ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage8mod-remote-storage:3.4.32492044721283960512512
mod-remote-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-finance-storage8mod-finance-storage:8.8.42102489612870088128
mod-finance-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-ebsconet7mod-ebsconet:2.4.0212481024128700128256
mod-ebsconet - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-sip27edge-sip2:3.4.12102489612876888128
mod-consortia-keycloak6mod-consortia-keycloak:1.7.12513647765124416384512
mod-consortia-keycloak - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-tags8mod-tags:2.4.12102489612876888128
mod-tags - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-courses7edge-courses:1.6.02102489612876888128
mod-inventory-update7mod-inventory-update:4.1.02102489612876888128
mod-inventory-update - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-notify8mod-notify:3.4.02102489612876888128
mod-notify - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-configuration8mod-configuration:5.12.02102489612876888128
mod-configuration - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
folio-keycloak11folio-keycloak:26.1.31500030002048204800
edge-caiasoft7edge-caiasoft:2.4.02102489612876888128
mod-licenses8mod-licenses:6.2.22248023125121792384512
mod-licenses - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-gobi8mod-gobi:3.0.02102489612876888128
mod-gobi - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-graphql8mod-graphql:1.13.12102489612876888128
mod-graphql - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-roles-keycloak8mod-roles-keycloak:3.0.32102489612851288256
mod-roles-keycloak - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-erm-usage5mod-erm-usage:5.0.02280025501281800384512
mod-erm-usage - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-batch-print7mod-batch-print:1.3.02102489612876888128
mod-batch-print - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-copycat8mod-copycat:1.8.02102489612876888128
mod-copycat - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-entities-links8mod-entities-links:4.0.3225922483512144001024
mod-entities-links - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
pub-edge8pub-edge:2023.06.142102489612876800
mod-orders8mod-orders:13.0.522048174010241024384512
mod-orders - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
kong13kong:3.9.0161446144204876888128
edge-patron8edge-patron:5.3.02102489625676888128
edge-ncip7edge-ncip:1.10.22102489612876888128
edge-inn-reach1edge-inn-reach:3.4.02102489612876888128
mod-users-bl8mod-users-bl:7.10.221440115212892288128
mod-users-bl - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-invoice8mod-invoice:6.0.121440115251292288128
mod-invoice - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-inventory-storage8mod-inventory-storage:29.0.1124096369010243067384512
mod-inventory-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A10247685122560128
mod-user-import7mod-user-import:3.10.12102489612876888128
mod-user-import - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-sender8mod-sender:1.14.12102489612876888128
mod-sender - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mgr-tenant-entitlements6mgr-tenant-entitlements:3.1.111024896102451288256
mod-data-export-worker24mod-data-export-worker:3.4.623072280010242048384512
mod-data-export-worker - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A10247685122560128
mod-circulation-storage8mod-circulation-storage:17.4.022880259210241814384512
mod-circulation-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-calendar8mod-calendar:3.3.022048153612876888128
mod-calendar - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-event-config8mod-event-config:2.9.02102489612876888128
mod-event-config - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-courses8mod-courses:1.4.122102489612876888128
mod-courses - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-circulation-item5mod-circulation-item:1.1.021024896128000
mod-circulation-item - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-email8mod-email:1.19.02280025501281800384512
mod-email - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-pubsub8mod-pubsub:2.16.12153614401024922384512
mod-pubsub - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-di-converter-storage8mod-di-converter-storage:2.4.22102489612876888128
mod-di-converter-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-circulation8mod-circulation:24.4.722880259210241814384512
mod-circulation - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
edge-orders7edge-orders:3.2.02102489612876888128
edge-rtac7edge-rtac:2.9.12102489612876888128
mod-users10mod-users:19.5.12102489625676888128
mod-users - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-template-engine8mod-template-engine:1.22.02102489612876888128
mod-template-engine - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-audit9mod-audit:2.11.12102489612876888128
mod-audit - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-source-record-manager8mod-source-record-manager:3.10.624096368810243076384512
mod-source-record-manager - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-quick-marc8mod-quick-marc:7.0.01228821761281664384512
mod-quick-marc - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-feesfines8mod-feesfines:19.3.22102489612876888128
mod-feesfines - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-users4edge-users:2.0.02102489612876888128
mod-dcb8mod-dcb:1.2.112204818441281292384512
mod-dcb - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-service-interaction7mod-service-interaction:4.2.22204818442561290384512
mod-service-interaction - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-patron8mod-patron:6.3.12102489612876888128
mod-patron - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-connexion8edge-connexion:1.4.12102489612876888128
mod-data-export-spring8mod-data-export-spring:3.5.21204818442561536384512
mod-data-export-spring - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-erm7edge-erm:1.4.02102489612876888128
mod-ncip7mod-ncip:1.15.72102489612876888128
mod-ncip - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-agreements8mod-agreements:7.2.22409640961024409600
mod-agreements - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-organizations8mod-organizations:2.1.02102489612862088128
mod-organizations - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-settings8mod-settings:1.2.02102489620076888128
mod-settings - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-search8mod-search:5.0.4225922480102414405121024
mod-search - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-dematic7edge-dematic:2.4.21102489612876888128
mod-data-import8mod-data-import:3.3.41204818442561292384512
mod-data-import - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-record-specifications7mod-record-specifications:2.0.1210248964001440256512
mod-record-specifications - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-inn-reach2mod-inn-reach:3.4.023600324012828805121024
mod-inn-reach - Sidecar 1N/Afolio-module-sidecar:3.0.4N/A1024512128256096
edge-inventory4edge-inventory:2.1.32102489612876888128
mod-orders-storage8mod-orders-storage:13.9.42102489651270088128
mod-orders-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-erm-usage-harvester5mod-erm-usage-harvester:5.0.22102489612876888128
mod-erm-usage-harvester - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-password-validator8mod-password-validator:3.4.0214401298128768384512
mod-password-validator - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-bulk-operations8mod-bulk-operations:2.2.523072260010242048384512
mod-bulk-operations - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A15007685122560128
mod-fqm-manager7mod-fqm-manager:4.0.10230002600128204888512
mod-fqm-manager - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
edge-dcb5edge-dcb:1.2.42102489612876888128
mod-finance8mod-finance:5.1.12102489625676888128
mod-finance - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-tlr2mod-tlr:1.0.172102489612876888256
mod-tlr - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-lists7mod-lists:3.1.5260002600128204888512
mod-lists - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A768715128256096
mod-permissions8mod-permissions:6.7.02168415445121024384512
mod-permissions - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-users-keycloak8mod-users-keycloak:3.0.32102489612851288256
mod-users-keycloak - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-marc-migrations5mod-marc-migrations:2.0.122880259210241814384512
mod-marc-migrations - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mgr-applications5mgr-applications:3.0.011024896102451288256
mgr-tenants5mgr-tenants:3.0.011024896102451288256
edge-oai-pmh7edge-oai-pmh:2.11.021512136010241440384512
mod-okapi-facade8mod-okapi-facade:3.0.02102489612876888128
mod-okapi-facade - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-rtac8mod-rtac:3.8.02102489612876888128
mod-rtac - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-source-record-storage8mod-source-record-storage:5.10.924096368810243076384512
mod-source-record-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-scheduler8mod-scheduler:3.0.12102489612851288256
mod-scheduler - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-circulation-bff2mod-circulation-bff:1.0.202102489612876888128
mod-circulation-bff - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-inventory8mod-inventory:21.1.724096368810243076384512
mod-inventory - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A10247685122560128
mod-requests-mediated2mod-requests-mediated:1.0.1721500102412876888192
mod-requests-mediated - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096
mod-linked-data2mod-linked-data:1.0.422048179225676888128
mod-linked-data - Sidecar 1N/Afolio-module-sidecar:3.0.4N/A1024512128256096
mod-patron-blocks8mod-patron-blocks:1.12.121024896102462088128
mod-patron-blocks - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
edge-fqm7edge-fqm:4.0.12102489612876888128
nginx-edge7nginx-edge:2023.06.1421024896128000
mod-invoice-storage8mod-invoice-storage:6.0.021872153601024384512
mod-invoice-storage - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-reading-room7mod-reading-room:1.2.12102489612876888128
mod-reading-room - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-data-export8mod-data-export:5.2.212592248020481440881024
mod-data-export - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-oai-pmh8mod-oai-pmh:3.15.224096369020483076384512
mod-oai-pmh - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512256256096
mod-notes8mod-notes:7.0.021024896128952384512
mod-notes - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-kb-ebsco-java7mod-kb-ebsco-java:6.0.02102489612876888128
mod-kb-ebsco-java - Sidecar 1N/Afolio-module-sidecar:3.0.5N/A1024512128256096
mod-eusage-reports5mod-eusage-reports:3.0.02102489612876888128
mod-eusage-reports - Sidecar 1N/Afolio-module-sidecar:3.0.4.455.nbN/A1024512128256096


Methodology/Approach

Test set

Data preparation: prepare . csv file with 50000 item barcodes for testing
The script to prepare files is here.

In case of testing with instances - populate the database with FOLIO records if needed with this script.

  1.  Manually tested. All tests triggered on one tenant only:
    1. Trigger bulk edit by submitting identifiers for 50000 item records
    2. Wait until Bulk edit finds all matched records and record the startTime and endTime of the process from the response.
    3. Edit instances and start the second part of the bulk edit to commit the changes.
    4. Record the start time of committing changes from Headers
  2. Database preparation script after the test run required to restore Suppress from discovery and Staff suppress parameters.

Additional screenshots