2026-07-15 Sys Ops & Management SIG Agenda and Meeting Notes

Meeting Notes

Trent (in Slack): “I also seem to be having issues with a few Capability Sets from app-authority-records and app-quick-marc . …”

• mod-entities-links no longer entitled, having it included in a different app

• the backend permission name didn't change but the path did

• app-acquisition was split up between Ramsons and Sunflower

• the ability to upgrade has not been fixed

• the api docs are for the latest version. In mod-entities-links github, it is there.

It is after entitlement, when you try to assign the capability to a role.
something is definitely missing

Error from mod-seach when upgrading to Trillium. Julian wrote a ticket.

Jason: Multiple tenants in the same name space. Waiting for the environment to get spun up, Kitfox team.
Something going on in the Kafka topics. An issue with the module version suffixes.

Jason Root 18:17
https://folio-org.atlassian.net/browse/MGRENTITLE-183
https://folio-org.atlassian.net/browse/MODUSERSKC-167
[DEPRECATED] Cookbook: Environment Variables for Mixed Version Deployment

Something going on with the system user password generation, when connecting to a Secret store like Vault.
This is in QA currently. How to set the env variables up when you are running the env with multiple versions.

Magnus: Will this take care of CSPs, too ? Yes.

Jason: By default, there is an API count limit on the AWS SSM endpoint. They are hitting the 10,000 limit.
Trent: I would suspect the sidecars.
Jason: The limit is hit frequently and often in a 15 tenant system.

Ingolf: Problems during entitlement. The modules seem to have come up, but there are silent fails. Kafka communication is missing. Modules can not connect to the Vault Secret Store, then fail to generate a system user. Reminds me of what Jason stated about the multi tenant multi versions problem.

When I reboot the server, all modules will be redeployed. If Vault has not come up yet, or Vault has not been de-sealed, yet, the modules can not connect to the Secret Store and silently fail.

One needs to look at ~ 140 module logs, ~ 70 modules plus their side cars. In Kubernetes, everything seems to be O.K., modules are green.

It is crucial to set these env var in every module:

Sie 18:22
extraEnvVars:
name: FOLIO_SYSTEM_USER_ENABLED
value: 'false'
name: SYSTEM_USER_CREATE
value: 'false'
name: SYSTEM_USER_ENABLED
value: 'false'

Jason: A selected number of modules worked, we tried to figure out which modules.

Shelley: We do that in every module, it is in a Secret for each module. We use SYSTEM_USER_CREATED (with a 'D' at the end).

Shelley Doljack 18:23
extraEnvVars:

• name: FOLIO_SYSTEM_USER_ENABLED
  value: 'false'

• name: SYSTEM_USER_CREATED
  value: 'false'

• name: SYSTEM_USER_ENABLED
  value: 'false'

Jason Root 18:25
FOLIO_SYSTEM_USER_ENABLED: "false"
SYSTEM_USER_CREATE: "false"
SYSTEM_USER_ENABLED: "false"
SYSTEM_USER_NAME: "mod-bulk-operations"
SYSTEM_USER_USERNAME: "mod-bulk-operations"

Shelley Doljack 18:35
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain

Magnis: Drain the cluster. Draw Kubernetes away from this Node.
It is a kubectl command. It is good operational maintanance for Kubernetes.

minus 18:36
checkout 'cordon' and 'drain' before reboots

Kong has a health endpoint. And all modules have a health endpoint.
You can have lots of errors but the health is still O.K.
Magnus: It takes 2 hours to restart all modules and check if they are all healthy. We do that by hand.
Magnus: Quite a lot of modules have problems with S3; they mention this in a comment.
modules have become a little bit better at testing when they start.
some reading to do: every module and every sidecar.

Trent: I write documentation; Getting from Okapi to Eureka Sunflower; Getting from Sunflower to Trillium.
I will share it when it is ready.

Umbrellaleaf delayed until 2027.
Jason: Getting everybody on to Sunflower. First to Okapi, then to Eureka. Then to Trillium.
Some are still on Ramsons.

Shelley: Sink cookies with postman. A Eureka postman bundle. It is called postman interceptor. You just have to configure it to allow domains in order to sink the cookies.

Next Meeting

22.07.2026

Chat Log

Jason Root 18:17
https://folio-org.atlassian.net/browse/MGRENTITLE-183
https://folio-org.atlassian.net/browse/MODUSERSKC-167
[DEPRECATED] Cookbook: Environment Variables for Mixed Version Deployment

Sie 18:22
extraEnvVars:
name: FOLIO_SYSTEM_USER_ENABLED
value: 'false'
name: SYSTEM_USER_CREATE
value: 'false'
name: SYSTEM_USER_ENABLED
value: 'false'

Shelley Doljack 18:23
extraEnvVars:

• name: FOLIO_SYSTEM_USER_ENABLED
  value: 'false'

• name: SYSTEM_USER_CREATED
  value: 'false'

• name: SYSTEM_USER_ENABLED
  value: 'false'

Jason Root 18:25
FOLIO_SYSTEM_USER_ENABLED: "false"
SYSTEM_USER_CREATE: "false"
SYSTEM_USER_ENABLED: "false"
SYSTEM_USER_NAME: "mod-bulk-operations"
SYSTEM_USER_USERNAME: "mod-bulk-operations"

Shelley Doljack 18:35
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain

minus 18:36
checkout 'cordon' and 'drain' before reboots

Sie 18:56
This was the session with the PC initiative "community development": 2026-02-04 Sys Ops & Management SIG Agenda and Meeting Notes

Topics for next meetings