2024-01-12 Sys Ops & Management SIG Agenda and Meeting notes

jpnelson :  Stanford is migrating to Poppy over Spring break with first deploying Poppy to our test systems for review by library staff. Likely we will only be able to do one or two releases per year.

Florian Gleixner : One or two releases per year, some libraries want to have new features early, some want to stick on old versions. Doing more than two releases per year is not possible.

Florian Gleixner : Because only two releases are supported backwards, we have to force clients to migrate if a new release comes out.

Need to have a good balance between librarian's needs (want to use new, previously missing features, quickly) and sys ops (minimize work for (and risks in) conducting release changes). Once or twice a year sounds most reasonable. 

If releases appear three or four times a year, sys ops will likely skip one release. Which means they will combine two release upgrades into one system upgrade.

The bottleneck is not so much the effort in bringing the releases to the server, but the testing period. Chicago has many workflows that need to be tested when a new release has been played to the server. Linköping has one test instance with real data and one test instance without data. Others have similar test instance setups. The testing period on the test servers takes several weeks. Only after that - and if all tests passed, maybe iteratively - the production instance may be upgraded. This process can not be performed more than twice a year, regarding also availability of testing staff.

CSP releases are not much of an issue, because the testing period usually does not take more than one week (TAMU). Re-deplyoing the services is not so much of an issue, it is the amount of new features (in a flower release) and the testing of those.

The view of large hosting providers is missing / Separate feedback from Wayne Schneider :

I'm sorry, I'm not available right now. But I can give you my (not necessarily Index Data's) perspective quickly.

I think the cadence of 2-3 releases a year is fine, but I worry that it doesn't deliver features (or, indeed, bug fixes) needed by libraries quickly enough. And the monolithic nature of releases is problematic. The Poppy release in particular is very labor-intensive, as it requires a number of out-of-band procedures outside the regular upgrade procedure (e.g. database scripts, etc.).

IMO a large issue with the cadence of releases is that we treat releases as large monoliths, which require extensive integration testing and suck up many resources, rather than prioritizing backwards-compatibility of interfaces and taking advantage of the microservices architecture to release features more incrementally. Of course this is a challenge as the domain boundaries of many modules are not hard and fast, and so delivering a feature for e.g. check out may require changes in multiple modules. As the project matures, rapid interface changes should be less necessary and I would hope we could deliver on this more successfully.

...I'm speaking for myself as a FOLIO sysop, not for Index Data as a hosting provider. Corrie Hutchinson would have a better perspective on what it is like to manage upgrades for many libraries simultaneously.