[FOLIO-1845] Replace postgresql-embedded by testcontainers.org Created: 05/Mar/19  Updated: 14/Apr/21  Resolved: 29/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Julian Ladisch Assignee: Unassigned
Resolution: Done Votes: 0
Labels: R2, back-end, decision, platform-backlog, platform-core, testing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-2145 Upgrade to PostgreSQL 12 Closed
blocks FOLIO-2144 Upgrade to PostgreSQL 11 Closed
is blocked by OKAPI-825 Switch from postgresql-embedded to te... Closed
is blocked by CQLPG-100 Replace postgresql-embedded by testco... Closed
Duplicate
is duplicated by RMB-483 TestContainers instead of postgresql-... Closed
Relates
relates to OKAPI-971 Fail unit tests if docker is not avai... Closed
relates to FOLIO-1955 Create databases using und-x-icu coll... Open
relates to FOLIO-3118 SPIKE: testcontainers and Vagrant Closed
relates to MODINVSTOR-369 Unit tests may hang when using Embedd... Closed
relates to MODPWD-49 Discover tools/libraries to support P... Closed
relates to RMB-317 Error message about embedded Postgres... Closed
relates to MODCAT-194 New embedded Postgres release to crea... Closed
Requires
is required by RMB-789 Remove support of EmbeddedPostgres Closed
Sprint: CP: ready for planning, CP: Roadmap backlog
Development Team: Core: Platform

 Description   

We cannot upgrade from postgres 10 to postgres 11, 12 or 13 as long as our back-ends still use Postgresql-embedded for unit/integration tests because Postgresql-embedded only supports postgres 9 und 10.

Postgresql-embedded is is not being actively maintained anymore and recommends to switch to testcontainers.org, see announcement from April 2019: https://github.com/yandex-qatools/postgresql-embedded/commit/0895ad0ad96ff27613bd1e1a351fb329a76fde22

Vert.x has replaced postgresql-embedded by testcontainers.org for their unit tests eight months before in August 2018:
https://github.com/vert-x3/vertx-mysql-postgresql-client/commit/16c538947d0e99ba3a66e4c99ccb87c9a972ffbb

testcontainers.org starts and stops a postgresql docker container.

postgresql-embedded does NOT support Linux x86-64, Linux x86-32, Windows x86-32 for PostgreSQL >= 11.
References:
https://github.com/yandex-qatools/postgresql-embedded/commit/15685611972bacd8ba61dd7f11d4dbdcb3ba8dc1
https://www.enterprisedb.com/downloads/postgres-postgresql-downloads

Okapi has succesfully implemented postgres testcontainers.org containers for the SSL tests ( OKAPI-787 Closed , pull request), this works fine in Jenkins.

General Docker requirements of testcontainers.org: https://www.testcontainers.org/supported_docker_environment/
Windows Docker requirements of testcontainers.org: https://www.testcontainers.org/supported_docker_environment/windows/



 Comments   
Comment by John Malconian [ 09/May/19 ]

It should be fairly easy to start up and make a fresh postgres container available in FOLIO Jenkins for backend modules builds I would think.

Comment by Marc Johnson [ 09/May/19 ]

A long while ago, the CI used to provide an external PostgreSQL server.

I think it was used in the early precursor to mod-inventory to test database integration

Comment by John Malconian [ 09/May/19 ]

Indeed. I ended up removing it because of the proliferation of modules using embedded postrgesql. On the other hand using something like testcontainers.org may be easier for developers for running their unit tests locally.

Comment by Marc Johnson [ 09/May/19 ]

John Malconian Do our current backend CI builds run within a docker container, or on a native OS?

Comment by John Malconian [ 09/May/19 ]

They do and it is possible to invoke other containers within the main build container.

Comment by Marc Johnson [ 09/May/19 ]

John Malconian Oh, I didn't know it was possible to invoke containers from inside a container!

Comment by Julian Ladisch [ 24/Jun/19 ]

On my Windows laptop it takes 47 seconds to start postresql-embedded and 4 seconds to start the postgres docker container. Both times do not include the download time, the download is cached.
This is more than 11 times faster.

Comment by Marc Johnson [ 14/Oct/20 ]

I recently asked if all developers had access to Docker, and got the following response from Taras Spashchenko on Slack

The only tricky case that can be here is when backend developers are using windows workstations. but at least two options are possible here
run docker containers in the virtual box
install docker for windows and lose support for virtual box

I think we need to be mindful of the developers maintaining the modules access to Docker when we roll this out

Comment by Julian Ladisch [ 21/Oct/20 ]

https://github.com/MicrosoftDocs/WSL/issues/798 "WSL2 breaks VirtualBox"

Comment by Julian Ladisch [ 04/Dec/20 ]

RMB looks for command line options
https://github.com/folio-org/raml-module-builder#command-line-options
and environment variables
https://github.com/folio-org/raml-module-builder#environment-variables
and picks the PostgreSQL connection information from them if available.
If this is not available RMB starts embedded postgres.
This is used both in test code and main code.
Developers using Windows make use of this so that they can run tests against an existing running PostgreSQL instance ( https://www.postgresql.org/download/windows/ ) and avoid the 47 seconds startup time that embedded postgres needs.
We can keep these command line and environment options so that Windows developers can run most of the tests without the need for docker.

Comment by Adam Dickmeiss [ 04/Dec/20 ]

I don't think we should keep postgresql-embedded under no circumstances. Have two system: containters and real postgres. The embedded postgres takes time to boot on both Linux and Windows. Starting Docker containers also takes minutes - first time - an image is downloaded. That's a price, regardless of platform.

Doesn't Docker work on Windows?

As far for starting modules with "no postgres", we'd just use containers for that. So the "embedded postgres" functionality would be preserved.

Comment by John Malconian [ 08/Jan/21 ]

Please remember that any containers that are created in the CI build pipeline need to be cleaned up after use.

Comment by Marc Johnson [ 08/Jan/21 ]

John Malconian

Please remember that any containers that are created in the CI build pipeline need to be cleaned up after use.

What does that involve? Is it something that test containers doesn't do already?

Does the Okapi build, which uses test containers I believe, leave containers behind?

Comment by Marc Johnson [ 08/Jan/21 ]

Adam Dickmeiss

I don't think we should keep postgresql-embedded under no circumstances.

Agreed.

As far for starting modules with "no postgres", we'd just use containers for that. So the "embedded postgres" functionality would be preserved.

Are you proposing that if a module being started (outside of tests, more production like) does not receive external database configuration it would attempt to start a docker container instance?

I know this is what RAML Module Builder does with embedded PostgreSQL at the moment. Personally, I would prefer we didn't have that path at all and storage modules had to receive some form of external database configuration (which might refer to a container). I think tools like test containers or embedded PostgreSQL should only be used for assisting with automated tests and not be involved in production code.

Comment by John Malconian [ 08/Jan/21 ]

Marc Johnson - I have seen orphaned testcontainers containers left behind on the FOLIO CI build systems and have had to clean them up manually. There could be a few reasons that might account for this:

1. The build failed before the container was cleaned up.
2. Early experiments with testcontainers.
3. testcontainers artifacts require a missing, explicit cleanup.

I don't know enough yet about testcontainers to determine which of those cases are true. If they can be cleaned up programmatically or they are cleaned up automatically after use, that would be ideal. If not, I can always add a post build cleanup step to the CI pipelines that cleans up container artifacts whether the build fails or succeeds. I would just need guidance on how best to proceed.

Comment by Marc Johnson [ 11/Jan/21 ]

John Malconian

I have seen orphaned testcontainers containers left behind on the FOLIO CI build systems and have had to clean them up manually.

Thanks, that is useful to know. I think we should try to identify the causes of those and figure out how to avoid / mitigate them before we roll this out all RAML Module Builder based modules.

If they can be cleaned up programmatically or they are cleaned up automatically after use, that would be ideal. If not, I can always add a post build cleanup step to the CI pipelines that cleans up container artifacts whether the build fails or succeeds. I would just need guidance on how best to proceed.

Are you referring to cleaning up the container images or the running instances of those images?

Comment by Marc Johnson [ 11/Jan/21 ]

Julian Ladisch

This is used both in test code and main code.

I think this would be a good opportunity to not have this option for the production code.

We can keep these command line and environment options so that Windows developers can run most of the tests without the need for docker.

I agree with keeping these configuration options. As I understand it, they can refer to any instance of PostgreSQL. That would allow developers to use a hosted, local native or a docker based instance of PostgreSQL as they choose.

I think that is sufficient options without retaining the use of embedded PostgreSQL, especially as one of the motivations for this change is to stop relying on a unsupported mechanism that is an impediment to FOLIO officially supporting later PostgreSQL versions.

Comment by Marc Johnson [ 11/Jan/21 ]

Craig McNally Jakub Skoczen Is this a tooling decision?

Comment by Julian Ladisch [ 13/Jan/21 ]

John Malconian

I have seen orphaned testcontainers containers

testcontainers.org should remove all containers in all cases, before reporting success or failure.
It also uses https://github.com/testcontainers/moby-ryuk for cleanup.
However, if the java process is killed the container may remain.
We've bumped Okapi to the latest testcontainers.org version 6 days ago: https://github.com/folio-org/okapi/commit/0c42b51cddcef1576e7540a0b69de220cf7bd488
Have new orphaned containers been created since then?

Comment by Julian Ladisch [ 13/Jan/21 ]

On a Windows 10 Pro laptop I've installed Docker Desktop Community 3.0.4 with default settings, one of the defaults is that "Use the WSL 2 based engine" is enabled.

I use maven on Windows (not in WSL 2):

$ mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: C:\Java\apache-maven-3.6.3
Java version: 11, vendor: Oracle Corporation, runtime: C:\Java\jdk-11
Default locale: de_DE, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

The okapi/okapi-core unit tests that use testcontainers properly start Docker containers and succeed:

[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,079.447 s - in org.folio.okapi.ModuleTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.834 s - in org.folio.okapi.service.impl.PostgresHandleTest

DockerModuleHandleTest skips one test, though:

[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 20.302 s - in org.folio.okapi.service.impl.DockerModuleHandleTest
Comment by Marc Johnson [ 04/Feb/21 ]

Jakub Skoczen Adam Dickmeiss Julian Ladisch

Has the decision that FOLIO recommends testcontainers for testing been made?

It seems to me that it has given that both OKAPI-825 Closed and RMB-483 Closed have been moved to it (or were they special cases?)

Comment by Julian Ladisch [ 08/Feb/21 ]

Each module can use for testing whatever the CI pipeline supports.
Okapi has switched to testcontainers.
RMB has switched to testcontainers.
RMB based modules can use the testcontainers support that RMB ships with ( https://github.com/folio-org/raml-module-builder/tree/master/postgres-testing ) or use something else. By providing the PostgresTesterContainer class in the postgres-testing sub-module RMB in some way recommends testcontainers.

Comment by Hanna Hulevich [ 29/Mar/21 ]

This is done by RMB-789 Closed and other issues

Generated at Thu Feb 08 23:16:19 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.