[FOLIO-3094] SPIKE: Investigate Upgrade to Postgres 12 Created: 29/Mar/21  Updated: 20/May/21  Resolved: 20/May/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Craig McNally Assignee: Steve Ellis
Resolution: Done Votes: 0
Labels: R2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File run-tests.sh     Zip Archive test-outcomes.zip    
Issue links:
Relates
relates to FOLIO-2145 Upgrade to PostgreSQL 12 Closed
Sprint: CP: sprint 113, CP: sprint 114
Story Points: 3
Development Team: Core: Platform
Release: R2 2021

 Description   

Overview

Investigate the breakinging changes in postgres 12 and evaluate what, if anything might need to be done in order to upgrade.

Scope

FOLIO as a whole, not isolated to a given framework or module.

Acceptance Criteria

  • Findings are documented (probably on the wiki) and shared with the team
  • User stories are created as needed


 Comments   
Comment by Steve Ellis [ 06/May/21 ]

I created a wiki page documenting my process and progress. I'll update that as things progress. See https://folio-org.atlassian.net/wiki/display/FOLIJET/Upgrading+to+Postgres+12.

Comment by Steve Ellis [ 07/May/21 ]

Julian Ladisch I've made some edits to the wiki page. Unfortunately I don't see a clear path to running tests against postgres 12 for modules that use embedded. I saw your edits to the wiki page suggesting that you could use 12 or 13 with embedded but I don't see a way to do that using their code.

I do see that you made a contribution to the embedded pg repo back in 2018 to add version 11 support. But nobody seems to have added 12 or 13. I'm wondering if you're thinking that might be something we want to do here (fork embedded pg and add support for 12?).

In a perfect world embedded pg would just let you point to the pg binary you wanted to use. But I don't see a way to do that. It's "embedded" too deep perhaps.

 

 

Comment by Julian Ladisch [ 07/May/21 ]

[https://github.com/yandex-qatools/postgresql-embedded
]downloads the postgres binary from
https://www.enterprisedb.com/downloads/postgres-postgresql-downloads

There is a configuration option to load any version you want:
https://github.com/yandex-qatools/postgresql-embedded#how-to-use-your-custom-version-of-postgresql

However, enterprisedb.com offers PostgreSQL download of version >= 11 for Mac OS X and Windows x86-64 only.
Not for Linux.

Embedded postgresql is deprecated. FOLIO switches to the more flexible use of Docker containers.

Therefore we shouldn't waist time trying to make embedded postgres work with PostgreSQL 12.

Removing startEmbeddedPostgres() in the unit tests and using DB_* is probably more easy.

Comment by Steve Ellis [ 08/May/21 ]

For modules that haven't yet made the switch to RMB 32.2+ and are still using embedded, testing against version 12 still seems be possible by bypassing embedded postgres by using a postgres-conf.json file or environment variables. For more info: https://github.com/folio-org/raml-module-builder#postgresql-integration. Using the conf file I'm able to have the tests in mod-finance-storage run against my local postgres 9.6. However a lot of tests that don't fail with embedded are failing with local non-embedded, even though I appear to be connecting to the db and queries from the tests are hitting it. Will continue to investigate this.

Comment by Steve Ellis [ 09/May/21 ]

I'm able to see all tests pass in mod-finance-storage when running against a local postgres version 10.16, so I'm thinking that embedded pg in RMB 32.1 isn't hitting 9.6, but rather 10.6, which is the default in embedded if no version is selected, and which appears to be the case in vertex-pg-client.

Comment by Steve Ellis [ 11/May/21 ]

In our discussion on monday we decided that version 10 is what I should be testing against, not 9.6. I've now run most tests using 10.16. Attaching the script I'm using to run the tests for a given module. This approach works reasonably well, and I'm able to bypass embedded postgres with a local postgres. So far most modules where the approach succeeds (it doesn't work for everything) seem to indicate that upgrading to 12 will be no problem. Of course there are cases which this may not be the case, some of which may be due to the postgres version change, and some of which may just be because I'm not running the tests in the right way, because the module's configuration is slightly different than what my script expects.

See the current state of things in the wiki:

https://folio-org.atlassian.net/wiki/display/FOLIJET/Upgrading+to+Postgres+12

 

 

Comment by Steve Ellis [ 14/May/21 ]

Adding the testing-outcomes.zip which contains detailed logs (4 files) for each of the tests run. I changed my approach to logging part way through my testing as I tried to script more things. Mosts test produced 4 log files:

  • The output of the postgres log with logging level set to info for version 10
  • Same for version 12
  • The output from mvn clean install invoked on 10
  • Same for version 12

If your  module has a warning sign in the wiki table these logs might provide meaningful information.

Generated at Thu Feb 08 23:25:34 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.