2023-11-29 - Officially Supported Technologies

Marc Johnson is next, followed by Ingolf Kuss 

Recording

Background:

• We decided on 27 Nov 2023 that we will not upgrade Postgres in Quesnelia.  The upgrade path and timing are still TBD.
• See DR-000038 - PostgreSQL Upgrade to 16
• Considerations:
  • Support period - Postgres 12 support ends in Nov. `24
  • Impact on testing
  • How are upgrades usually conducted?  "In place" or "parallel"?
  • Lead time for devs, sysOps, etc.
  • Test results from testcontainer
  • Test results from Kitfox evaluation
  • This isn't just about code in folio-org....  External code exists, we shouldn't make assumptions based solely on the folio-org repos.
  • Can/should we support multiple versions of Postgres for a given flower release?  
    • What does that mean for TCR evals?
    • How do we make expectations clear to devs?  sysOps?  etc.

Notes:

• Craig McNally reiterated that the TC decided not to upgrade PostgreSQL for Quesnelia and stay with version 12, asking how we address questions described in the considerations above
• Julian Ladisch asked if we should consider the other decisions for the official supported technologies list for Quesnelia
• Marc Johnson advised that firstly we should update the PostgreSQL version in that document. And suggested it is strange that we've settled on this version and that Julian Ladisch (the original proposer) is now advocating we change topic
• Craig McNally agreed that the decision had been made and any further conversation should be for Ramsons, unless there are further conversations relevant to Quesnelia?
• Jenn Colt? asked if folks understood the support period challenge, as they considered the end of the PostgreSQL support period ended close to the [likely] end of the support period for Quesnelia?
• Craig McNally advised that the support period for PostgreSQL ends in Nov 2024, which is around the time of the release of Ramsons, however Quesnalia support period would run till the next release, because FOLIO supports two releases at a time
• Tod Olson described how the proposal was that we continue to support PostgreSQL 12 in Quesnelia and later, in a hot fix, support version 16 as well. This would address the in-place upgrade challenge where support for two versions is needed
• Craig McNally asked whether any of this is applicable to Quesnelia or not? Which would be the case if we intend to alter supported versions during to CSP (referred to as a hot fix above), which would be abnormal behaviour
• Marc Johnson asked if Tod Olson was suggesting we evaluate the multi-version proposal?
• Julian Ladisch advised that developers must only use features from PostgreSQL 12, however we would support up to 16 as well, which we've seen evidence for from the testing we've done
• Craig McNally emphasised that we don't know about code that is outside of the FOLIO organisation's github, under the incomplete assumption that the tests we have pass
• Taras Spashchenko suggested that we move to support PostgreSQL 16 because we have confirmation that there are no significant problems in the folio-org code base and we have time to fix any issues during the rest of the release
• Craig McNally expressed concern that those kinds of decisions had got us into trouble before, e.g. the recent RTR decision
• Taras Spashchenko suggested that this change is different because we are only changing an underlying technology rather than the behaviour of the FOLIO platform. And that if we do not use any new features, then we can safely declare support for PostgreSQL 16 in a backwards compatible way
• Craig McNally suggested that if an external developer uses features from 16 then it no longer backwards compatible. That means that developers would need to understand this expectation
• Marc Johnson asked how to do we tell folks that we support version 16 and communicate to them to not use any new features since 12? And how do we test that range of versions to gain confidence? What version of PostgreSQL should BugFest run on? Thanks to the work that Kitfox did, we know that the Karate tests run against 16 with similar success to 12, however we know that those tests don't cover much of the functionality that we do regression testing on for each flower release. As the community has never officially supported multiple versions of PostgreSQL, the in-place upgrade across database versions is officially (though often not practically) the responsibility of the system operators performing the upgrade
• Taras Spashchenko asked what would be different if we deferred the upgrade to PostgreSQL 16 to the Ransoms release? Marc Johnson advised that it would be different because we are already late for making this decision for Quesnelia and thus Ransoms would be different because of the amount of notice we would be giving
• Julian Ladisch advised that we already constrain the PostgreSQL version to 12 for the automated tests and BugFest. And that it is sufficient to do a one time test of the system using PostgreSQL 16
• Marc Johnson disagrees that we should only do a limited amount of testing on the new version of PostgreSQL 16, because it currently takes a long time to regression test FOLIO. And advised that additional environments and builds would need consultation with the Community Council (or AWS costs group). And that it feels strange that on Monday we decided we weren't comfortable to go to PostgreSQL 13 and yet today we are discussing supporting all versions between 12 and 16
• Julian Ladisch advised we can easily run the automated tests on 16 and we don't need continuous testing because we are confident that there probably won't be any issues
• Craig McNally suggested that the communication is a consideration than the testing because that is a departure from where we have been in the past. And thought that an upgrade to 13 in Quesnalia, to avoid the support challenge, and an upgrade to 16 in a release after Ramsons to avoid multiple upgrades in subsequent releases. And raised concerns about how the current approach could come back to bite us. And asked what is involved in the automated testing changes that need to be made?
• Julian Ladisch advised that the unit tests would need the version of the PostgreSQL container bumping in RMB, spring tools and other libraries, and then require all module implementors to use these new versions
• Florian Gleixner advised that it is necessary to support multiple versions of infrastructure within releases to facilitate upgrades without disrupting the system
• Marc Johnson agreed that it would be beneficial for FOLIO to support multiple versions of infrastructure for each release, however it is not where FOLIO is today. And raised concerns that the more configuration options we support the more we have to do to be confidence that the system works with all configurations. And advised that my experience is that periodic testing does not give us that confidence. And asked if it is imperative for FOLIO to support multiple versions for Quesnelia (when we are already late for making this decision), when that hasn't been the case previously?
• Taras Spashchenko suggested that as a trade off, Quesnelia could support PostgreSQL 13 because it only has limited changes from 12 and we already have environments running 13, and that in the Ransoms release we move to the latest version
• Craig McNally advised that the concern raised previously was that would require system operators to upgrade multiple releases in a row, however we could mitigate that by deferring for another release
• Tod Olson suggested that the competing needs are the need for operators to upgrade to a supported version yet developers only use features from version 12. And that the effort involved will be similar whether we do this now or later. And that we don't need to do a full regression test because we won't be affecting the front end
• Craig McNally asked how to communicate support for multiple versions? Julian Ladisch suggested this was answered in the proposed decision record
• Marc Johnson asked if that included a communication plan? Julian Ladisch advised that developers and hosted reference environments would still use 12 and so nothing would need to change
• Craig McNally asked if anyone has time to come up with a way to communicate these kinds of changes, please do