This page documents the findings of my investigation of upgrading the FOLIO project as a whole to postgres 12. The goal is to determine if an upgrade to version 12 causes any module unit tests to break. For background there is a ticket associated with this:
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
The scope of the investigation is FOLIO-wide. This means we're interested in seeing how an upgrade to postgres 12 affects all modules.
...
For some discussion on making the change to testcontainers see
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
For modules that haven't yet made the switch to RMB 32.2+ and are still using embedded, testing against version 12 still seems be possible by bypassing embedded postgres by using a postgres-conf.json file or environment variables. For more info see: https://github.com/folio-org/raml-module-builder#postgresql-integration. Using this file I'm able to have the tests in mod-finance-storage run against my local postgres 9.6. However a lot of tests that don't fail with embedded are failing with local non-embedded, even though I appear to be connecting to the db and queries are hitting it.
...
Module | Current supported Postgres version per readme | Version info | Verification method | Status | Notes | Work impact |
raml-module-builder | N/A | Uses postgres:12-alpine in its test container. | Source code contains a reference to postgres:12-alpine. |  | Already using 12. No failed tests. This is the first row because is used by most of the modules below. The version of RMB that modules use has something to say about the state of postgres. See above for notes on that. | None |
| mod-agreements |
| It's a grails app. See the services sentry tools/testing/docker-compose.yml. It is using postgres 12. | None | |||
| mod-audit | RMB 32.2.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. No failed tests. Upgrading to 12 should be fine. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. | Upgrade to newest RMB. | ||
| mod-authtoken | | Does not appear to talk to postgres in any way. See pom.xml. No reference to vertx-db-client or domain model runtime (RMB). | None | |||
| mod-calendar | RMB 32.2.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 34. No errors or failures with 10 or 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-camunda | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | ||||
| mod-circulation | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | ||||
mod-circulation-storage | Readme mentions 9.6.1. Probably needs to be updated to reflect the use of RMB 32.2.0. | RMB 32.2.0 | Uses PostgresTesterContainer |
| RMB 32.2 uses pg 12. | None |
| mod-codex-ekb | RMB 32.1.0 | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | |||
| mod-codex-inventory | RMB 32.2.0 | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | |||
| mod-codex-mux | RMB 32.1.0 | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | |||
| mod-configuration | RMB 33.0.0 | Uses PostgresTesterContainer | | No references to embedded pg. Uses RMB 33. | None | |
| mod-courses | RMB 32.2.2 |  | Able to bypass embedded postgres locally. Needed to remove PostgresClient.startEmbeddedPostgres (multiple calls) to get tests against local pg to work. Ran tests against version 10.16 and 12. Many test errors (not failures) with both 10 and 12. | Need to investigate large number of errors. These errors do not occur when running against embeded pg, so I have likely broken them. | ||
| mod-datasets | Readme says it runs a glint server as an Okapi module. Unclear what that is. | Unknown | ||||
| mod-data-export | RMB 32.2.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. One failed test when running tests with pg 10. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-data-export-worker | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | |||||
| mod-data-import | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. One failed test when running tests with pg 10. Two different tests failed after upgrading to 12. See logs for details. | Investigate two failures (see logs) after upgrading to 12. | ||
| mod-data-import-converter-storage | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 159. No errors or failures with 10 or 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-ebsconet | | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. No reference to RMB. | May still be using postgres. Unclear how to run tests against different version of pg, although the connection string appears to be something set in the ENV. See run-env.sh. | |||
| mod-erm-usage-harvester | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Same tests that failed with version 10 failed with version 12. Same 1 test failed in both cases. Upgrading to 12 should be fine. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. | Upgrade to newest RMB. | ||
| mod-eusage-reports | Tests do not appear to hit postgres. No apparent usage of embedded pg in tests. | None | ||||
| mod-feesfines | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Same tests that failed with version 10 failed with version 12. Tests run: 399. Same 3 failed in both cases. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-finance-storage | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 277. Errors: 4. Failures:0. Same errors with version 10 and version 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-gobi | RMB 32.1.0 | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | ||||
| mod-graphql | Don't think it will be affected since it appears to sit on top of modules like mod-inventory-storage. | None | ||||
| mod-inn-reach | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | ||||
| mod-inventory | Does not appear to talk to postgres or reference RMB in its pom file. | None | ||||
mod-inventory-storage | Readme mentions 9.6.1. Probably needs to be updated to reflect the use of RMB 32.2.0. | RMB 32.2.0 | Uses PostgresTesterContainer |
| RMB 32.2 uses postgres 12. | None |
| mod-invoice | RMB 32.1.0 | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | |||
| mod-invoice-storage | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. Tests run: 129. No tests failed. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-licenses | | It is a grails app, so no pom file. Uses postgres. Not able to run tests locally. | Figure out how to run tests locally. | |||
| mod-login | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Same tests that failed with version 10 failed with version 12. Tests run: 31. Errors: 12. Failures: 0. Saame errors with version 10 and 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-login-saml | RMB 33.0.0-pre1 | | No references to embedded postgres in source. Using RMB 33 which uses postgres 12. | None | ||
| mod-notes | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. No tests failed with 10 or 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-notify | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. One test (the same test) failed with 10 and 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-oai-pmh | RMB 31.1.5 | | Uses embedded for tests. But appears to use embedded postgres in a way that is different from other modules. Logs indicate that maybe embedded pg (or some other pg process?) should be started manually ?using pg_ctl. | TODO - Figure out how to get run tests to run. | ||
| mod-orders-storage | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 236. One test failed (the same test) with 10 and 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-organizations | RMB 31.1.0 | Although it has RMB in the pom, there's no reference to vertx-pg-client or vertx-sql-client. Tests do not appear to make an reference to embedded or PostgresClient. | None | |||
| mod-patron | RMB 32.1.0 | No reference to vertx-pg or vertex-sql in pom. I don't see any reference to embedded postgres in the tests. | None | |||
| mod-patron-blocks | RMB 32.2.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Needed to remove PostgresClient.startEmbeddedPostgres call to get tests against local pg to work. No test errors or failures. 248 tests run. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-permissions | Readme doesn't mention postgres. | RMB 33.0.0-pre2. This should mean it uses 12 for postgres access. | Uses PostgresTesterContainer | | Dependencies reference org.testcontainers (probably the RMB). No reference to embedded pg. It looks like mod-permissions only talks to postgres through the RMB, and the RMB version uses postgres 12 in the test container. | None |
| mod-pubsub | RMB 32.1.0 | | Was able to run tests against my local postgres by passing the value "external" in using org.folio.pubsub.test.database. I think this gets passed as a maven param. All tests pass when running against either version of postgres. | Upgrade to newest RMB. | ||
| mod-remote-storage | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | ||||
| mod-rtac | No reference to postres in readme. | RMB 32.1.0 | Not seeing any reference to embedded pg in source. No reference to vertx-sql or vertx-pg in pom. | None | ||
| mod-search | No vertx-sql or vertx-pg in pom. No reference to embedded pg in tests. | None | ||||
| mod-sender | RMB 32.1.0 | Has vertx-sql or vertx-pg in pom. But no reference to embedded pg in tests. No usage of PostgresClient in tests. | None | |||
| mod-source-record-storage | RMB 32.1.0 | | Even though it is using 32.1, it has already upgraded to postgres 12 by running pg in a container. See https://github.com/folio-org/mod-source-record-storage#compiling. | The amount of work here is fully known, since this module appears to not be taking the path of using testcontainers via RMB. | ||
| mod-tags | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 3. No tests failed with either version. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-user-import | RMB 32.1.0 | Not seeing anything in source about embedded postgres. | Upgrade to newest RMB. | |||
| mod-template-engine | RMB 32.1.0 | | Able to bypass embedded postgres locally. Ran tests against version 10.16 and 12. Tests run: 43. No difference between postgres versions. | Upgrade to newest RMB. | ||
| mod-users | RMB 32.1.0 | | Able to bypass embedded postgres locally. Same two tests fail when running 10 and 12. Upgrading to 12 should be fine. | Upgrade to newest RMB. | ||
| mod-users-bl | RMB 32.1.0 | Not seeing anything in source about embedded postgres. | None |
...