Skip to end of banner
Go to start of banner

Upgrading to Postgres 12

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 86 Next »

This page documents the findings of my investigation of upgrading the FOLIO project as a whole to postgres 12. The goal is to determine if an upgrade to version 12 causes any module unit tests to break. For background there is a ticket associated with this: FOLIO-3094 - Getting issue details... STATUS .

The scope of the investigation is FOLIO-wide. This means we're interested in seeing how an upgrade to postgres 12 affects all modules.

Summary of findings

TODO

Test method

Determine the impact of database upgrade by doing the following:

  1. Determine if the module has persistence.
  2. Clone the module code.
  3. Run the unit tests for the module, making a note of any failed tests.
  4. Determine a method for running the unit tests using postgres 12.
  5. Run the tests and compare the results with the results prior to the upgrade.
  6. If there are any breakages investigate them and document them.

Notes on code this investigation used

I'm working off of clones of modules that were made on 2021-05-06, so the findings should be only relevant to the code in that snapshot.

Should we try to go to version 13?

The answer appears to be no, since, although RDS now supports version 13, the Aurora managed service does not:

Determining if the module has persistence

See if the pom file containing domain-models-runtime or vertx-pg-client seem like good indications. A number of modules appear to not have persistence roles. For example mod-inventory doesn't, but mod-inventory-storage does. See below for a summary of the status of each module.

Determining if a module has been upgraded to postgres 12

Modules that new up the RMB's PostgresTesterContainer class, have been upgraded. PostgresTesterContainer is where the postgres version gets passed into the constructor on the RMB level (so not on the module level). See for example mod-configuration, mod-circulation-storage, mod-permissions, etc. This is documented in the table below for each module.

Note that not all modules that are using RMB 32.2, are using PostgresTesterContainer (for example mod-calendar is using RMB 32.2 but still uses embedded postgres for its tests).

Modules that depend on RMB 33

Modules that use RMB 33 use PostgresTesterContainer that uses postgres 12.

Modules that depend on RMB 32.2

Using 32.2 opened up the possibility for modules to use PostgresTesterContainer, but it isn't a guarantee. For example, mod-calendar uses 32.2, but still uses embedded postgres.

Open questions

Should we expand the scope of this spike to investigate other FOLIO code beyond modules?

Determining a method for running the unit tests using postgres 12

If a module has switched to testcontainers (moving away from embedded postgres), swapping out a version of postgres appears to be a simple matter of swapping out the docker image name in code like this:

postgresSQLContainer = new PostgreSQLContainer<>("postgres:12-alpine")

For some discussion on making the change to testcontainers see FOLIO-1845 - Getting issue details... STATUS .

For modules that haven't yet made the switch to RMB 32.2+ and are still using embedded, testing against version 12 still seems be possible by bypassing embedded postgres by using a postgres-conf.json file or environment variables. For more info see: https://github.com/folio-org/raml-module-builder#postgresql-integration. Using this file I'm able to have the tests in mod-finance-storage run against my local postgres 9.6. However a lot of tests that don't fail with embedded are failing with local non-embedded, even though I appear to be connecting to the db and queries are hitting it.

Testing against a local version of 10.16, all tests pass, which leads me to believe that we're not using 9.6 anymore.

Module readiness for postgres 12

The following table goes into detail about where modules stand as far as upgrading to postgres 12 is concerned. A module is anything with "mod-"  in its name. If we want to go beyond this definition of modules to include other FOLIO code, let me know.

Legend: (tick) - Everything is good. The module is ready for an upgrade to postgres or it has already upgraded. (warning) - Either a problem was encountered in testing or unit tests due to testing the upgrade failed. Probably means more work. Empty status - Means that I wasn't able to find anything that would indicate work needing to be done (because for example the module doesn't talk to postgres). In all cases see the notes for details.

Module

Current supported Postgres version per readme

Version info

Verification method

Status

Notes

Work impact

raml-module-builder

N/A

Uses postgres:12-alpine in its test container.

Source code contains a reference to postgres:12-alpine.

(tick)Already using 12. No failed tests. This is the first row because is used by most of the modules below. The version of RMB that modules use has something to say about the state of postgres. See above for notes on that.None
mod-agreements


(tick)

It's a grails app. See the services sentry tools/testing/docker-compose.yml. It is using postgres 12.None
mod-audit
RMB 32.2.0

TODOTODO
mod-authtoken


(tick)Does not appear to talk to postgres in any way. See pom.xml. No reference to vertx-db-client or domain model runtime (RMB).None
mod-calendar
RMB 32.2.0
(warning)Unable to build.Unknown.
mod-camunda



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.
mod-circulation



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None

mod-circulation-storage

Readme mentions 9.6.1. Probably needs to be updated to reflect the use of RMB 32.2.0.

RMB 32.2.0

Uses PostgresTesterContainer

(tick)

RMB 32.2 uses pg 12.

None
mod-codex-ekb
RMB 32.1.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-codex-inventory
RMB 32.2.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-codex-mux
RMB 32.1.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-configuration
RMB 33.0.0Uses PostgresTesterContainer(tick)No references to embedded pg. Uses RMB 33.None
mod-courses
RMB 32.2.2
(warning)Hasn't switched to test containers. TODO
mod-datasets



Readme says it runs a glint server as an Okapi module. Unclear what that is.Unknown
mod-data-export
RMB 32.2.0
(warning)Tests do make use of embedded postgres. Unable to get tests to hit local postgres however. Does not appear to respect postgres-conf.json even though it is loaded.Investigate why we can't bypass embedded postgres locally.
mod-data-export-worker



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.
mod-data-import
RMB 32.1.0
(warning)Uses PostgresClient with embedded in tests. Tests run against version 12 fail to complete. This may be a compatibility issue with pg 12.Investigate test failure after upgrade to postgres 12.
mod-data-import-converter-storage
RMB 32.1.0
(warning)Uses embedded postgres in tests. Able to bypass with local postgres via DB_* environment variables.Having trouble comparing the logs from the two test runs. The info messages with test results summaries from surefire are not the same, although both runs appear to have 0 errors or failures.
mod-ebsconet


(warning)No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. No reference to RMB.May still be using postgres. Unclear how to run tests against different version of pg, although the connection string appears to be something set in the ENV. See run-env.sh.
mod-erm-usage-harverster
RMB 32.1.0
(warning)Tests do make use of embedded postgres. Unable to get tests to hit local postgres however. Does not appear to respect postgres-conf.json even though it is loaded.Investigate why we can't bypass embedded postgres locally.
mod-eusage-reports



Tests do not appear to hit postgres. No apparent usage of embedded pg in tests.None
mod-feesfines
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Same tests that failed with version 10 failed with version 12. Tests run: 399. Same 3 failed in both cases. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-finance-storage
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 277. Errors: 4. Failures:0. Same errors with version 10 and version 12. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-gobi
RMB 32.1.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.
mod-graphql



Don't think it will be affected since it appears to sit on top of modules like mod-inventory-storage.None
mod-inn-reach



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-inventory



Does not appear to talk to postgres or reference RMB in its pom file.None

mod-inventory-storage

Readme mentions 9.6.1. Probably needs to be updated to reflect the use of RMB 32.2.0.

RMB 32.2.0

Uses PostgresTesterContainer

(tick)

RMB 32.2 uses postgres 12.

None
mod-invoice
RMB 32.1.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-invoice-storage
RMB 32.1.0
(warning)1 test fails with verison 10. One additional test fails when testing against version 12.Investigate test failure.
mod-licenses


(warning)It is a grails app, so no pom file. Uses postgres. Not able to run tests locally.Figure out how to run tests locally.
mod-login
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Same tests that failed with version 10 failed with version 12. Tests run: 31. Errors: 12. Failures: 0. Saame errors with version 10 and 12. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-login-saml
RMB 33.0.0-pre1
(tick)No references to embedded postgres in source. Using RMB 33 which uses postgres 12. None
mod-notes
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. No tests failed with 10 or 12. Upgrading to 12 should be fine.Upgrade to newest RMB. 
mod-notify
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. One test (the same test) failed with 10 and 12. Upgrading to 12 should be fine.Upgrade to newest RMB. 
mod-oai-pmh
RMB 31.1.5
(warning)Uses embedded for tests. But appears to use embedded postgres in a way that is different from other modules. Logs indicate that maybe pg should be started manually?Figure out how to get tests to run.
mod-orders-storage
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 236. One test failed (the same test) with 10 and 12. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-organizations
RMB 31.1.0

Although it has RMB in the pom, there's no reference to vertx-pg-client or vertx-sql-client. Tests do not appear to make an reference to embedded or PostgresClient.None
mod-patron
RMB 32.1.0

I don't see any reference to embedded postgres in the tests.
mod-patron-blocks
RMB 32.2.0

Still starts embedded postgres.
mod-permissionsReadme doesn't mention postgres.RMB 33.0.0-pre2. This should mean it uses 12 for postgres access.Uses PostgresTesterContainer(tick)Dependencies reference org.testcontainers (probably the RMB). No reference to embedded pg. It looks like mod-permissions only talks to postgres through the RMB, and the RMB version uses postgres 12 in the test container.None
mod-pubsub
RMB 32.1.0

No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-remote-storage



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-rtacNo reference to postres in readme.RMB 32.1.0

Not seeing any reference to embedded pg in source.  No reference to vertx-sql or vertx-pg in pom.None
mod-search



No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests.None
mod-sender
RMB 32.1.0

Has vertx-sql or vertx-pg in pom. But no reference to embedded pg in tests. No usage of PostgresClient in tests.None
mod-source-record-storage
RMB 32.1.0
(tick)Even though it is using 32.1, it has already upgraded to postgres 12 by running pg in a container. See https://github.com/folio-org/mod-source-record-storage#compilingThe amount of work here is fully known, since this module appears to not be taking the path of using testcontainers via RMB.
mod-tags
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 3. No tests failed with either version. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-user-import
RMB 32.1.0

Not seeing anything in source about embedded postgres.Upgrade to newest RMB.
mod-template-engine
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 43. No difference between postgres versions.Upgrade to newest RMB.
mod-users
RMB 32.1.0
(tick)Able to bypass embedded postgres locally. Same two tests fail when running 10 and 12. Upgrading to 12 should be fine.Upgrade to newest RMB.
mod-users-bl
RMB 32.1.0

Not seeing anything in source about embedded postgres.None
  • No labels