2026-06-03 Sys Ops & Management SIG Agenda and Meeting Notes

Shelley: Taking a copy of production databases ( Kong, Keycloak, Vault ) to put into our dev/test/stage env and we couldn’t b/c we were hitting the max_locks_per_transaction setting. We take copy of prod data to practice the Production upgrade . Couldn’t do pgdump, because there were about 800,000 empty tables in the mod_search schema. @Joshua Greben opened a Ticket. Other related ticket: https://folio-org.atlassian.net/browse/MSEARCH-1233

Example:

• mod-search create empty tables ; mod-search-5.0.8

• The AI overview in the folio wiki about it:
  Streaming resource IDs refers to a specialized API and workflow in FOLIO's mod-search module designed to handle large-scale data operations like reindexing and bulk ID retrieval.
  Key Components

  • resource-ids-streaming API: A new module (v1.0) that replaces the deprecated GET /search/instances/ids and GET /search/holdings/ids endpoints  .

  • Streaming Reindex Service: Orchestrates full reindexing by populating a new Operational Index via a streaming ingestion pipeline from mod-inventory-storage  .

  • Delta Reconciliation: A critical workflow that logs resource IDs modified via Kafka events during active reindexing into a reindex_deltas table to ensure data consistency  .

  • Temporary Table Management: Jobs submitted via POST /search/resources/jobs create temporary database tables with random names that require scheduled cleanup to prevent resource leaks  .

  Technical Implementation

  • Dual-Index Model: Uses a flat Operational Index optimized for high-throughput writes and a nested Search View Index for complex reads  .

  • Performance: The system utilizes PgUtil.streamGet() to handle exceptionally large datasets efficiently by reducing memory consumption and network bandwidth 

  • Versioning: Each streaming operation is identified by a time-ordered UUIDv7 known as a stream_id  .

• fixed in Trillium; Trillium cleans up the empty tables PR for fix to cleanup empty tables

• this has been around since Sunflower or Ramsons

Mahrad: Creating new Helm Charts. These should be similar to folio-helm-v2. Same Helm Charts for all instances; only mininally auto-generation.

Ingolf: Issue with certifcates for keycloak

• KC_HOSTNAME field when installing keycloak should be the external server name

• The problem is KC_HOSTNAME is set to the folio-k8s11 (a node in the cluster).

• Ingress is going to the same folio-k8s11 domain and not “eureka-testing”

• Where to put the certificate and how does it look in k8s. What does the proxy_pass do.

• Sascha uses nodeport; IP addresses of the nodes and nodeport are all in nginx-conf.

• Why not use metal LB?

• load balancers use nodeport in the background.

• Using a load balancer is the easiest thing to do.

• Jason always sets everything to https all the way down whenever he puts a certificate on a load balancer in front of the k8s cluster.

•  

Next Meeting

Chat Log

Topics for next meetings