2022-02-04 - Sys Ops & Management SIG Agenda and Notes

Went live January 10 2022. Migration went well after one weekend data migration and some days testing. Juniper Hotfix 3.

15 Nodes Rancher-provisioned Kubernetes Cluster
3 etcd Control Plane Nodes
Most Worker Nodes have 4 Cores, 16 GB Mem, some Nodes have 8 Cores, 24 GB Mem

Cluster is dedicated to Folio
Database Stack is on VMware outside Kubernetes with connection pooling and load-balancing with one master and 2 replicas. Asynchronous processes for replication.
Decision not to run database in Kubernetes was because HA Postgres was not very good tested at time of decision, and database administrations best practice was using VMware.

No outages so far (fantastic!), but some performance issues.
Regular searching in Inventory app is slow, users should use Inventory-ES (Elasticsearch).
Issues with Elasticsearch:

• Reindexing did not index all the first time
• Corrupt Index forced to reindex again. Folio showed a "general search error", hard to trace down in logs.

Checkin/out, circulation tasks were slow. Many modules needed to scale to 5 Pods each, which helped for performance issues.
Reporting with LDP did not meet expectations of users. Frontend is a homegrown PHP application.

Upgrade Plans: Deploy everything in the same namespace. Then upgrade folio/Modules, no blue-green Deployment. Hint: Update Pubsub first!
TAMU uses a steering committee to decide which versions/updates have to be deployed.

Deployment is done using flat YAM files described in https://github.com/folio-org/folio-install/tree/tamu-r2-2021/alternative-install/kubernetes-rancher/TAMU
Eventually Ansible will be used in future for deployment.

Gone live with single server installation, Juniper without Elasticsearch.

Issue reported by users: https://issues.folio.org/browse/UIIN-1902

Switch to Elasticsearch using docker-compose is planned.