2025-12-17 Sys Ops & Management SIG Agenda and Meeting Notes

New member Ghassen Tajini

Ghassen

Ghassen is from BSZ (South-West German Library Consortium) in Konstanz, Germany. He has deployed the Environment (Kong, Keycloak, Vault, Postgres, Kafka,…) on a single-node Kubernetes setup on a Laptop. The Laptop has 60 GB of RAM and 2 Terabyte hard disk. He managed to deploy Keycloak. He is about to now install the manager components and deply the modules. He will install the latest snapshot version.

Dump Email addresses from the database , when setting up a test instance from production data

Florian, Jason, Trent, Magnus

When setting up a test system from the production system, it is not a good practice to dump e-mail adresses from the (mod-users) database. The system will still send emails to those addresses.
Trent: we use a tool called smtp 4 dev
We remove the smtp config of mod-email
Florian: We re-route all emails based on the sender

EDIFACT Issue in Update to Okapi Sunflower

Trent

Trent: EDIFACT issue, restart the modules . We observe that in Okapi Sunflower.

Jason: I observed the same issue in Eureka Sunflower.

Vault needs to be unsealed every few weeks

Ingolf

Vault needs to be unsealed every few weeks, otherwise the Vault pob goes red.

I also observed increased CPU consumption in my cluser. Pods frequently restart.

Shelley: Configure in Vault values file:

server:

unsealConfig:

enabled: false

entitlement fails on Eureka Sunflower

Ingolf

In my cluster, entitlement of app-platform-minimal still fails on Eureka Sunflower CSP-2.

The System seems to habe resource problems.

Group: Look at Kong pod. Kong pod is green, but down. Log says “Kong is waiting to start…”

Shelley: Increase Kong Resource Request and Resource Limits:

These are our Kong resources:
resources:
limits:
ephemeral-storage: 1Gi
memory: 12Gi
requests:
ephemeral-storage: 50Mi
memory: 9Gi

Delete all SNAPSHOT versions (if you want to entitle Sunflower CSP-2).

Jason: Kong does not recover very well.
Force Kong to run on another node, the pod will move "Waiting für Kong to start"
Hopefully APISIX will fix that.

Chat Log

Heute

Jason Root an Alle 18:04
RIP local laptop ;)

Shelley Doljack an Alle 18:11
update <tenantid>_mod_users.users set jsonb = jsonb_set(jsonb, '{personal, email}', ‘”<“’whatever fake email>) where not jsonb #>> '{personal, email}' is null;

Florian Kreft (LRZ) an Alle 18:18
https://gitlab.bib-bvb.de/folio-public/okapi-build/-/blob/v7.0.0/Dockerfile?ref_type=heads

Shelley Doljack an Alle 18:20
Sorry, had to take a phone call.

Jason Root an Alle 18:11
In my experience, you need to run this a few times. I don’t exactly know why but I’ve seen it not update every record…

Shelley Doljack 18:12
Really?

Jason Root 18:12
Yeah, let me find the exact SQL I was using
Here is what I was using back when I was at TAMU:

UPDATE tamu_mod_users.users
SET jsonb = Jsonb_set(jsonb, '{personal, email}', '"folio_user@tamu.org"')
WHERE jsonb -> 'personal' ->> 'email' != 'folio_user@tamu.org';
Ah yours is very similar - but has a “where not"

Shelley Doljack 18:22
The module system users don’t have email addresses, so their records have email is null. I do counts before and after and the update stmt changes all of the user records to the fake email address.

Jason Root an Alle 18:24
I have to drop a little bit before 11:30 to another meeting.

Jason Root an Alle 18:24 (Bearbeitet)
Jenkins CI builds Okapi from here: https://jenkins-aws.indexdata.com/job/folio-org/job/okapi/

Jason Root an Alle 18:30
Gotta drop, adios all!

minus an Alle 18:34
https://developer.hashicorp.com/vault/docs/concepts/seal

Trent Spears an Alle 18:37
https://github.com/lrstanley/vault-unseal

Shelley Doljack an Alle 18:37
resources:
limits:
cpu: 1024m
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi

Jason Root an Alle 18:42
I’m back! Meeting was short.

Jason Root an Alle 18:47
kubectl --kubeconfig .kube/<YOUR_CLUSTER_CONFIG> describe pod <YOUR_POD> -n <YOUR_NAMESPACE>

That output will tell you why it’s crashing

Shelley Doljack an Alle 18:48
These are our Kong resources:
resources:
limits:
ephemeral-storage: 1Gi
memory: 12Gi
requests:
ephemeral-storage: 50Mi
memory: 9Gi

Jason Root an Alle 19:02
I remember the days when FOLIO was 10ish modules ;)
Can confirm now they don’t all need to be running to upgrade.
But if you have other tenants you don’t want to upgrade, or can’t - then you’ll need to keep the old ones u

Topics for next meetings

Next Meeting January 7th 12-13 EST

Happy Holidays !