2026-07-29 Sys Ops & Management SIG Agenda and Meeting Notes

Meeting Notes

Mahrad: Sometimes entitlement fails upon the first attempt. But the second or third time, entitlement succeeds.

Shelley: It it hitting some timeouts. At the 2nd time, it goes faster.

Mahrad: But in our case it fails within a minute. And it only happens every tenth time.

Shelley: We have pgadmin in front.

Mahrad: But our biggest problem is the user management. It works unreliably. Sometimes it fails, sometimes it works. You can remove a user, but sometimes the correspondig user in keycloak stays. I can’t set the user credentials one more time using the same password.

Sometimes there is a descrepany of what the sidecar know about Keycloak and what is actually in Keycloak.

Endpoints are either from users-keycloak, roles-keycloak or login-keycloak. It is very odd. Normally it works as it should. But sometimes this happends. All I can do is restart.

Shelley: There are Kafka topics.

Mahrad: In some cases a request to Keycloak gives me “forbidden”. A restart fixes it.

Shelley: We have a bunch of app users for integrations. We had to re-create their passwords when we migrate. I set the password in the UI, settings - developers.

Mahrad: We have a script for this for admin passwords.

Shelley: When we upgrade, we have to post the passwords again.

Mahrad: One should not touch these APIs for a given user too often, because [ the system ] doesn’t like this.

Shelley: There is a script from EBSCO which creates roles and permissions. It is worth looking at.

Shelley: platform-lsp for Sunflower CSP-7 is missing some of the modules, mod-reading-room, …, from app-reading-room and app-linked-data. CSP-8 is not missing them.

Shelley: In data-import, when you upload large MARC2-files, it just timed out. Increased the sidecar env var MAX_BODY_SIZE to 200 MB. The default value for that is 10 MB. There is a bunch of timeout setting: the modules have timeouts, sidecars have timeouts, Kong has timeouts …

Shelley: Another thing: Trying to get OpenSearch work instead of ElasticSearch. Probably not going to install the OpenSearch operator. It installs https and certificates. We don’t need to do that for our purposes. How to deploy OpenSearch without the secure things.

Jason: I have a manifest file for my opensearch deployment. We use AWS persistent storage.

Florian G: We use the OpenSearch operator. It could work without https but we want to use https. The library admins should be able to look into it [ in our scenario ] . We added the certificate for the OpenSearch operator and injected it into the Keystore of mod-search. The bitnami elasticsearch chart does something that we don’t need in FOLIO. Ingest Nodes are good, but here there are of no use for us. Also Coordinating Nodes [ in elasticsearch ] are not needed. We have very static data. That is my thinking of it. You are good with some Master Nodes and some Data Nodes and that’s all. If you run without https you are all good with the default mod-search.

Jason: I sent you my manifest. Try with the single node. You point it to the master node and it handles the charting. I think they are statful sets.

Shelley: I put a link to the values file in the Zoom chat.

Shelley: Going to OpenSearch, we will have to re-index everything. If you are going from Elasticsearch version 7, you can probably go without re-indexing.

Florian G: I guess you always have to re-index. There are too many differences from Elasticsearch to OpenSearch.

Florian G. There are problems with OpenSearch 3 and Elasticsearch 9 in FOLIO. But this was in Quesnelia. You have to stay on major versions OpenSearch 2 or Elasticsearch 8. Remove the opensearch security plugin if you want to go without https.

Jason: You can have one master node and data nodes. But for HA, you want to scale this all out.

Next Meeting

Ingolf is out on August 12th.

Next regular meeting is on 26.08.2026

Chat Log

Shelley Doljack 18:40
https://github.com/folio-org/folio-helm-v2/blob/d2912993af5815359b354b634230c689dc458c00/charts/folio-common/templates/_sidecar.tpl#L61-L62

Shelley Doljack 18:45
This is what I have so far: https://github.com/sul-dlss/folio-eureka/blob/opensearch/folio-dev/infrastructure/opensearch.yaml

Topics for next meetings