2024-01-12 Sys Ops & Management SIG Agenda and Meeting notes

Date and time

9-10 CT

https://openlibraryfoundation.zoom.us/j/591934220?pwd=dXhuVFZoSllHU09qamZoZzZiTWhmQT09

Topics

see below

Attendees

TimeItemWhoNotes

Question from the Technical CouncilIngolf

How many releases per year do we want from FOLIO ?

Nils Olof Paulsson : Best would be not to have any new releases

jroot : Average one release per year is conducted. That means, one time a year, most time in summer is done

Ingolf Kuss : Two releases per years sounds fine. 

jpnelson :  Stanford is migrating to Poppy over Spring break with first deploying Poppy to our test systems for review by library staff. Likely we will only be able to do one or two releases per year.

Florian Gleixner : One or two releases per year, some libraries want to have new features early, some want to stick on old versions. Doing more than two releases per year is not possible.

Florian Gleixner : Because only two releases are supported backwards, we have to force clients to migrate if a new release comes out.

Need to have a good balance between librarian's needs (want to use new, previously missing features, quickly) and sys ops (minimize work for (and risks in) conducting release changes). Once or twice a year sounds most reasonable. 

If releases appear three or four times a year, sys ops will likely skip one release. Which means they will combine two release upgrades into one system upgrade.

The bottleneck is not so much the effort in bringing the releases to the server, but the testing period. Chicago has many workflows that need to be tested when a new release has been played to the server. Linköping has one test instance with real data and one test instance without data. Others have similar test instance setups. The testing period on the test servers takes several weeks. Only after that - and if all tests passed, maybe iteratively - the production instance may be upgraded. This process can not be performed more than twice a year, regarding also availability of testing staff.

CSP releases are not much of an issue, because the testing period usually does not take more than one week (TAMU). Re-deplyoing the services is not so much of an issue, it is the amount of new features (in a flower release) and the testing of those.

The view of large hosting providers is missing / Separate feedback from Wayne Schneider :

I'm sorry, I'm not available right now. But I can give you my (not necessarily Index Data's) perspective quickly.

I think the cadence of 2-3 releases a year is fine, but I worry that it doesn't deliver features (or, indeed, bug fixes) needed by libraries quickly enough. And the monolithic nature of releases is problematic. The Poppy release in particular is very labor-intensive, as it requires a number of out-of-band procedures outside the regular upgrade procedure (e.g. database scripts, etc.).

IMO a large issue with the cadence of releases is that we treat releases as large monoliths, which require extensive integration testing and suck up many resources, rather than prioritizing backwards-compatibility of interfaces and taking advantage of the microservices architecture to release features more incrementally. Of course this is a challenge as the domain boundaries of many modules are not hard and fast, and so delivering a feature for e.g. check out may require changes in multiple modules. As the project matures, rapid interface changes should be less necessary and I would hope we could deliver on this more successfully.

...I'm speaking for myself as a FOLIO sysop, not for Index Data as a hosting provider. Corrie Hutchinson (Unlicensed) would have a better perspective on what it is like to manage upgrades for many libraries simultaneously.




Provide any feedback on the post by Craig of Dec. 18 (see the post verbatim on the right)Ingolf

Post of Dec 18th: "The Technical Council has an important update regarding PosgtreSQL and the Quesnelia & Ramsons releases.

  • PostgreSQL 12 will reach EOL on November 14, 2024 and FOLIO needs to transition to a current version, PostgreSQL 16.  Quesnelia will be a transition release which must run under both PostgreSQL 12 and PostgreSQL 16, this will allow deployed environments to transition.
  • It is imperative that when deployed environments switch from PostgreSQL 12 to PostgreSQL 16, nothing breaks due to this update.
  • For Quesnelia, modules should be constrained to PostgreSQL 12 features which are forward-compatible (not broken by) PostgreSQL 16.
  • Making this transition will have implications across the technical side of the project, and the The Technical Council will collaborate with affected audiences / stakeholders to assess any impact and potential implementation issues.


Please do not hesitate to ask questions in this thread."

This decision of the TC has some consequences to system operators.

Please read also here at "summary for sysops": DR-000038 - PostgreSQL Upgrade to 16 - Technical Council - FOLIO Wiki

Discussion:

Ingolf Kuss Does anyone run unsupported Postgres Versions?

Florian Gleixner Yes, we run Postgres 13

Florian Gleixner Idea of the Decision Record is to make postgres upgrade independent from tenant upgrades.

Ingolf Kuss Enough feedback for the TC?

Tod Olson Seems not to be a big issue

jroot (and the whole group): No problems expected. Not a big issue.



Action items

  • Type your task here, using "@" to assign to a user and "//" to select a due date