Overview The purpose of this document is to highlight test result of full reindex of ECS cluster in scope of https://folio-org.atlassian.net/browse/PERF-1255
Reindex done without errors;
Reindex time has not changed compared to previous tests;
Sub-resources are not locked in the database after reindexing has finished (in other words, after reindexing is finished successfully, users can create a new Instance with subject, contributor, classification, and call number and find them via Browse).
Additional resources:
https://folio-org.atlassian.net/browse/MSEARCH-1161
Ticket created to investigate background query that was running during 5 hours after reindex finished https://folio-org.atlassian.net/browse/MSEARCH-1181
Summary
According to latest reindex report - reindex took 3 hours 4 minutes with scaled environment. In current test we used setup “as is“ (Xlarge DB, r7g.2xlarge.search, 2 tasks for mod-inventory-sorage and mod-search) and together with merge and upload it took 3 hours 47 minutes. So we may conclude that results are about the same compared to the previous test.
Sub resources are not locked and Instance with subject, contributor, classification, and call number may be created and found via Browse
Even that reindex done after approximately 4 hours - there are still background job running that occupy DB CPU for 45% during 4-5 hours.
Resource utilization
Service CPU Utilization High CPU utilization of mod search may be explained by allocated 64 CPU units.
As PTF previously recommended https://folio-org.atlassian.net/wiki/x/IgCZU
Memory Utilization Memory utilization showed stable trend.
mod-search memory usage increased from 35% to max 69% during upload phase. After phase was passed memory returned to ±40%.
DB Resources DB CPU usage reached 84 % max during inventory reindex. It is visible that DB was occupied even after reindex is done for next 4-5 hours using 45% CPU
Query that load DB for 40% during next 5-6 hours after reindex is done.
WITH cte AS (
SELECT s.id,
s.value,
s.authority_id,
s.source_id,
s.type_id,
s.last_updated_date
FROM fs09000000_mod_search.subject s
WHERE (last_updated_date, id) > ($1, $2)
ORDER BY last_updated_date, id
LIMIT $3
)
SELECT s.id,
s.value,
s.authority_id,
s.source_id,
s.type_id,
s.last_updated_date,
json_agg(
CASE
WHEN sub.instance_count IS NULL THEN NULL
ELSE json_build_object(
'count', sub.instance_count,
'shared', sub.shared,
'tenantId', sub.tenant_id
)
END
) AS instances
FROM cte s
LEFT JOIN (
SELECT cte.id,
ins.tenant_id,
ins.shared,
count(1) AS instance_count
FROM fs09000000_mod_search.instance_subject ins
INNER JOIN cte ON ins.subject_id = cte.id
GROUP BY cte.id, ins.tenant_id, ins.shared
) sub ON s.id = sub.id
GROUP BY s.id, s.value, s.authority_id, s.source_id, s.type_id, s.last_updated_date
ORDER BY last_updated_date ASC, id ASC
Freeable memory
Even though the DB was heavily utilizing CPU for an extended period of time, free memory did not drop to critical levels.
MSK Kafka metriсs
Open Search metrics Max indexing rate was reached during inventory reindex - 144K operations/min
Max CPU utilization during inventory reindex was 44%.
Appendix Infrastructure 10 r7g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r7g.xlarge database, writer instance.
MSK - fse-test
Apache Kafka version 3.7.x
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Open Search
modules versions Cluster Resources - secon-pvt (Tue Feb 24 11:32:40 UTC 2026) Module Task Definition Revision Module Version Task Count Mem Hard Limit Mem Soft Limit CPU Units Xmx Metaspace Size Max Metaspace Size R/W Split Enabled mod-remote-storage 9 mod-remote-storage:3.4.3 2 4920 4472 64 3960 512 512 false mod-remote-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-finance-storage 9 mod-finance-storage:8.8.4 2 1024 896 64 700 88 128 false mod-finance-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-ebsconet 9 mod-ebsconet:2.4.0 2 1248 1024 64 700 128 256 false mod-ebsconet - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-mosaic 1 mod-mosaic:1.0.0 2 1024 896 64 768 88 128 false mod-mosaic - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-consortia-keycloak 10 mod-consortia-keycloak:1.7.3 2 5136 4776 512 4416 384 512 false mod-consortia-keycloak - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-tags 9 mod-tags:2.4.1 2 1024 896 64 768 88 128 false mod-tags - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-courses 5 edge-courses:1.6.0 2 1024 896 64 768 88 128 false mod-inventory-update 9 mod-inventory-update:4.1.0 2 1024 896 64 768 88 128 false mod-inventory-update - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-notify 9 mod-notify:3.4.1 2 1024 896 64 768 88 128 false mod-notify - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-configuration 9 mod-configuration:5.12.0 2 1024 896 64 768 88 128 false mod-configuration - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-caiasoft 5 edge-caiasoft:2.4.1 2 1024 896 64 768 88 128 false folio-keycloak 8 folio-keycloak:26.4.3 3 2750 2250 1536 1536 0 0 false mod-licenses 9 mod-licenses:6.2.3 2 2480 2312 64 1792 384 512 false mod-licenses - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-gobi 9 mod-gobi:3.0.1 2 1024 896 64 768 88 128 false mod-gobi - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-graphql 9 mod-graphql:1.13.1 2 1024 896 64 768 88 128 false mod-graphql - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-roles-keycloak 9 mod-roles-keycloak:3.0.14 4 4096 3688 128 3076 88 256 false mod-roles-keycloak - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-erm-usage 9 mod-erm-usage:5.1.1 2 2800 2550 64 1800 384 512 false mod-erm-usage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-batch-print 12 mod-batch-print:1.3.0 2 1024 896 64 768 88 128 false mod-batch-print - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-copycat 9 mod-copycat:1.8.1 2 1024 896 64 768 88 128 false mod-copycat - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-entities-links 10 mod-entities-links:4.0.3 2 2592 2483 64 1440 0 1024 false mod-entities-links - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false pub-edge 6 pub-edge:2025.09.19 2 1024 896 64 768 0 0 false mod-orders 9 mod-orders:13.0.9 2 2048 1740 64 1024 384 512 false mod-orders - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false kong 7 kong:3.9.2 2 6144 6144 64 768 88 128 false edge-patron 8 edge-patron:5.3.1 2 1024 896 64 768 88 128 false edge-ncip 5 edge-ncip:1.10.2 2 1024 896 64 768 88 128 false edge-inn-reach 2 edge-inn-reach:3.4.0 2 1024 896 128 768 88 128 false mod-users-bl 9 mod-users-bl:7.10.3 2 1440 1152 64 922 88 128 false mod-users-bl - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-inventory-storage 10 mod-inventory-storage:29.0.16 4 8192 7372 1024 5160 384 1024 false mod-inventory-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 768 512 256 0 128 false mod-invoice 9 mod-invoice:6.0.3 2 1440 1152 64 922 88 128 false mod-invoice - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-user-import 9 mod-user-import:3.10.2 2 1024 896 64 768 88 128 false mod-user-import - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-sender 9 mod-sender:1.14.2 2 1024 896 64 768 88 128 false mod-sender - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mgr-tenant-entitlements 6 mgr-tenant-entitlements:3.1.9 2 1024 896 64 512 88 256 false mod-login-keycloak 9 mod-login-keycloak:3.0.4 2 1024 896 128 512 88 256 false mod-login-keycloak - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-data-export-worker 9 mod-data-export-worker:3.4.12 2 3072 2800 64 2048 384 512 false mod-data-export-worker - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 768 64 256 0 128 false mod-circulation-storage 9 mod-circulation-storage:17.4.1 2 2880 2592 64 1814 384 512 false mod-circulation-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-calendar 9 mod-calendar:3.3.1 2 2048 1536 64 1024 96 256 false mod-calendar - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-event-config 9 mod-event-config:2.9.0 2 1024 896 64 768 88 128 false mod-event-config - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-courses 9 mod-courses:1.4.12 2 1024 896 64 768 88 128 false mod-courses - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-circulation-item 9 mod-circulation-item:1.2.2 2 1024 896 64 0 0 0 false mod-circulation-item - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-email 9 mod-email:1.19.1 2 2800 2550 64 1800 384 512 false mod-email - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-pubsub 9 mod-pubsub:2.16.2 2 1536 1440 1024 922 384 512 false mod-pubsub - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 512 256 0 96 false mod-circulation 9 mod-circulation:24.4.14 2 2880 2592 64 1814 384 512 false mod-circulation - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-di-converter-storage 9 mod-di-converter-storage:2.4.2 2 1024 896 64 768 88 128 false mod-di-converter-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-rtac 5 edge-rtac:2.9.3 2 1024 896 64 768 88 128 false edge-orders 5 edge-orders:3.2.2 2 1024 896 64 768 88 128 false mod-users 9 mod-users:19.5.4 2 1024 896 64 768 88 128 false mod-users - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-template-engine 9 mod-template-engine:1.22.0 2 1024 896 64 768 88 128 false mod-template-engine - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-audit 12 mod-audit:2.11.4 2 1024 896 64 768 88 128 false mod-audit - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-source-record-manager 9 mod-source-record-manager:3.10.11 2 4500 4000 64 3000 384 512 false mod-source-record-manager - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-quick-marc 9 mod-quick-marc:7.0.0 1 2288 2176 64 1664 384 512 false mod-quick-marc - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-feesfines 9 mod-feesfines:19.3.2 2 1024 896 64 768 88 128 false mod-feesfines - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-dcb 11 mod-dcb:1.3.10 2 2048 1844 64 1292 384 512 false mod-dcb - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-service-interaction 9 mod-service-interaction:4.2.3 2 2048 1844 64 1290 384 512 false mod-service-interaction - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-patron 9 mod-patron:6.3.5 2 1024 896 64 768 88 128 false mod-patron - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-data-export-spring 9 mod-data-export-spring:3.5.2 1 2048 1844 64 1536 384 512 false mod-data-export-spring - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-organizations-storage 9 mod-organizations-storage:4.9.1 2 1024 896 64 700 88 128 false mod-organizations-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-erm 5 edge-erm:1.4.0 2 1024 896 64 768 88 128 false mod-ncip 9 mod-ncip:1.16.1 2 1024 896 64 768 88 128 false mod-ncip - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-agreements 9 mod-agreements:7.2.3 2 4096 4096 64 4096 0 0 false mod-agreements - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-organizations 9 mod-organizations:2.1.0 2 1024 896 64 768 88 128 false mod-organizations - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-serials-management 9 mod-serials-management:2.0.4 2 2780 2312 64 1792 384 896 false mod-serials-management - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-settings 9 mod-settings:1.2.0 2 1024 896 64 768 88 128 false mod-settings - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-data-import 10 mod-data-import:3.3.5 1 2048 1844 64 1292 384 512 false mod-data-import - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-search 23 mod-search:5.0.11 2 2592 2480 64 1440 512 1024 false mod-search - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-dematic 5 edge-dematic:2.4.2 1 1024 896 64 768 88 128 false mod-record-specifications 9 mod-record-specifications:2.0.3 2 2048 1592 64 1440 256 512 false mod-record-specifications - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-inn-reach 3 mod-inn-reach:3.4.0 2 3600 3240 128 2880 512 1024 false mod-inn-reach - Sidecar 1 N/A folio-module-sidecar:3.0.4 N/A 1024 512 128 256 0 96 false mod-orders-storage 9 mod-orders-storage:13.9.5 2 1024 896 64 700 88 128 false mod-orders-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-erm-usage-harvester 9 mod-erm-usage-harvester:5.1.1 2 1024 896 64 768 88 128 false mod-erm-usage-harvester - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-password-validator 9 mod-password-validator:3.4.0 2 1440 1298 64 768 384 512 false mod-password-validator - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-dcb 5 edge-dcb:1.3.5 2 1024 896 64 768 88 128 false mod-fqm-manager 9 mod-fqm-manager:4.0.15 2 3000 2600 64 2048 88 512 false mod-fqm-manager - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-bulk-operations 9 mod-bulk-operations:2.2.8 2 3072 2600 64 2048 384 512 false mod-bulk-operations - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1500 768 64 256 0 128 false mod-finance 9 mod-finance:5.1.1 2 1024 896 64 768 88 128 false mod-finance - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-lists 9 mod-lists:3.1.5 2 6000 2600 64 2048 88 512 false mod-lists - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 768 715 64 256 0 96 false mod-permissions 9 mod-permissions:6.7.1 2 1684 1544 64 1024 384 512 false mod-permissions - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-users-keycloak 9 mod-users-keycloak:3.0.12 4 4096 3688 128 3076 88 256 false mod-users-keycloak - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-marc-migrations 9 mod-marc-migrations:2.0.5 2 2880 2592 64 1814 384 512 false mod-marc-migrations - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mgr-applications 6 mgr-applications:3.0.6 2 1024 896 64 512 88 256 false mgr-tenants 6 mgr-tenants:3.0.6 2 1024 896 64 512 88 256 false edge-oai-pmh 5 edge-oai-pmh:2.11.0 2 1512 1360 64 1440 384 512 false mod-okapi-facade 9 mod-okapi-facade:3.0.0 2 1024 896 64 768 88 128 false mod-okapi-facade - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-rtac 9 mod-rtac:3.8.2 2 1024 896 64 768 88 128 false mod-rtac - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-source-record-storage 11 mod-source-record-storage:5.10.15 2 4500 4000 64 3000 384 512 false mod-source-record-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-scheduler 9 mod-scheduler:3.0.8 2 1024 896 64 512 88 256 false mod-scheduler - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-inventory 11 mod-inventory:21.1.19 2 4096 3688 512 3076 384 512 false mod-inventory - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 768 512 256 0 128 false mod-linked-data 9 mod-linked-data:1.0.4 2 2048 1792 64 768 88 128 false mod-linked-data - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-patron-blocks 9 mod-patron-blocks:1.12.2 2 1024 896 64 768 88 128 false mod-patron-blocks - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false edge-fqm 5 edge-fqm:4.0.1 2 1024 896 64 768 88 128 false nginx-edge 6 nginx-edge:2025.09.19 2 1024 896 64 0 0 0 false mod-invoice-storage 9 mod-invoice-storage:6.0.0 2 1872 1536 64 1024 384 512 false mod-invoice-storage - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-reading-room 9 mod-reading-room:1.2.1 2 1024 896 64 768 88 128 false mod-reading-room - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-data-export 9 mod-data-export:5.2.3 1 2592 2480 64 1440 88 1024 false mod-data-export - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-oai-pmh 9 mod-oai-pmh:3.15.4 2 4096 3690 64 3076 384 512 false mod-oai-pmh - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-notes 9 mod-notes:7.0.0 2 1024 896 64 952 384 512 false mod-notes - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-kb-ebsco-java 9 mod-kb-ebsco-java:6.0.1 2 1024 896 64 768 88 128 false mod-kb-ebsco-java - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false mod-eusage-reports 9 mod-eusage-reports:3.0.0 2 1024 896 64 768 88 128 false mod-eusage-reports - Sidecar 1 N/A folio-module-sidecar:3.0.18.nb N/A 1024 512 64 256 0 96 false
Data Volume
cs00000int
instancecount
holdingscount
itemcount
authoritycount
cs00000int
1874245
22
25
1460136
cs00000int_0001
6190015
6115107
6807974
2043417
cs00000int_0002
250656
423843
845449
1763189
cs00000int_0003
252910
430519
856456
1368817
cs00000int_0004
233675
263948
611709
1368816
cs00000int_0005
435241
789574
1652178
1378816
