2025-12-03 Sys Ops & Management SIG Agenda and Meeting Notes

Welcome

Replacing Kong in FOLIO

Jason

See Jason’s POST in the chat !

Jason: Kong will be replaced as the Gateway in FOLIO. Investigated alternatives were APISIX and Gloo.

Shelley: We will also replace nginx.

nginx = an Ingress Controller or a Proxy (here: used as an ingress controller).
Kong is also an Ingress Controller
Envoy

Shelley: we will exchange nginx by Envoy as an Ingress Controller.
Jason/Ingolf : This will integrate smoothly. You have to re-deploy the ingresses, but those don’t interact much with the other components of FOLIO.

Jason: We have about a dozen Ingresses.

Ingolf: I have confgured 7 ingresses using Traefik as the ingress controller.

Jason: Old Ingress Controller (Kong) will still work, it just will not have community control anymore. You will have to build it yourself from source.

RMB Log discussion

Julian an Jason

Jason: We have a solution to the log analysis which became costly becasue the RMB modules log so much.

See Julian’s post in #folio-sys-ops of today: Julian propsed to reduce the amount of RMB module logs by PUTs to the /admin/loglevel API.

Jason: Are the log configs persisted ?

Julian: It's a shell script, the PUT ist not being stored.
One has to use some kind of (Kubernetes) cron script or something to do the PUTs.

log4j configs: Some modules have log4j configs. Some modules don't have a request id pattern.
Jason: Set mode to WARN, DEBUG, FATAL or something. It is easier to track issues if you just track warnings and above.
it would be nice to have log level in env.

mod-source-record-storage times out

Shelley

Shelley: Restarting all of the srs stuff. mod-source-record-storage times out. Usually when one tries to import a record from oclc. The error ends up as a time-out.
Jason: Kafka messaging is not flowing properly.
Shelley: it is not Kafka. It times out at a GET request.
Jason: For source-record-storage ... set connection time to 40; for source-record-manager we set it to 300.000 => see Jason’s POST in the chat!
it put so much load on the database that it was timing out.

entitlement fails in Sunflower

Ingolf

Entitlement of mod-platform-minimal fails in Euerka with

Caused by: org.folio.tools.store.exception.SecretNotFoundException: Attribute: mod-users-keycloak not set for eureka-testing/hbztest

this is being logged in mod-users-keycloak sidecar.

See Shelley’s post in #folio-eureka-early-adapters of December 1st !

… But also, we found that the "system users" weren't getting created in keycloak. We ended up creating mod-users-keycloak and mod-users in Vault and Keycloak because it looked like the process wasn't doing it for us. After adding those, things went more smoothly.  …

Ingolf: How to create the system users in Vault and Keycloak ?

Shelley: In Vault, create a Secret “mod-users-keycloak” with some random 32digit password. Then in Keycloak, create a user “mod-users-keycloak” with the same Credentials. Use the same 32digit password. Set password at Credentials → Reset Password and choose “temporary: off”.

Also, set add “-DSYSTEM_USER_ENABLED=false “ to JAVA_OPTS in mod_users_keycloak and all modules that create system users!

Also: Need to set OKAPI_URL=http://localhost:8082 in mod-users-keycloak,

Shelley: not sure if it worked because we have created the users mod-users and mod-users-keycloak manually in Vault and Keycloak before, or if it worked because we set the env vars.

Ingolf: I did the the env vars beforehand and it didn’t work. So, for me, it is obviously a fault in Sunflower. It works in the snapshot versions, which are newer than sunflower.

Ingolf : to creat Random 32 characters in a Linux console:

Chat Log

Heute

Jason Root an Alle 18:03
POC - Building Kong from source (What if we stick with Kong?):  https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1190428721
Spike - Investigate Kong Alternatives (initial breadth-first review of options):  https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1203601409
Deep Dive - APISIX: https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1228047278
Deep Dive - Gloo: https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1297612801
API Gateway Comparison: APISIX vs Gloo vs Kong:  https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1408172033

Shelley Doljack an Alle 18:07
Envoy

Jason Root an Alle 18:32
SRM Ques:

• name: "DB_EXPLAIN_QUERY_THRESHOLD"
  value: "300000"

Sie an Alle 18:50
OKAPI_URL=http://localhost:8082

Sie 19:05
head -c 300 /dev/urandom | tr -cd '[a-zA-Z0-9-_]' | head -c 32

Topics for next meetings