[FOLIO-1845] Replace postgresql-embedded by testcontainers.org Created: 05/Mar/19 Updated: 14/Apr/21 Resolved: 29/Mar/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Julian Ladisch | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | R2, back-end, decision, platform-backlog, platform-core, testing | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | CP: ready for planning, CP: Roadmap backlog | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | Core: Platform | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
We cannot upgrade from postgres 10 to postgres 11, 12 or 13 as long as our back-ends still use Postgresql-embedded for unit/integration tests because Postgresql-embedded only supports postgres 9 und 10. Postgresql-embedded is is not being actively maintained anymore and recommends to switch to testcontainers.org, see announcement from April 2019: https://github.com/yandex-qatools/postgresql-embedded/commit/0895ad0ad96ff27613bd1e1a351fb329a76fde22 Vert.x has replaced postgresql-embedded by testcontainers.org for their unit tests eight months before in August 2018: testcontainers.org starts and stops a postgresql docker container. postgresql-embedded does NOT support Linux x86-64, Linux x86-32, Windows x86-32 for PostgreSQL >= 11. Okapi has succesfully implemented postgres testcontainers.org containers for the SSL tests (
General Docker requirements of testcontainers.org: https://www.testcontainers.org/supported_docker_environment/ |
| Comments |
| Comment by John Malconian [ 09/May/19 ] |
|
It should be fairly easy to start up and make a fresh postgres container available in FOLIO Jenkins for backend modules builds I would think. |
| Comment by Marc Johnson [ 09/May/19 ] |
|
A long while ago, the CI used to provide an external PostgreSQL server. I think it was used in the early precursor to mod-inventory to test database integration |
| Comment by John Malconian [ 09/May/19 ] |
|
Indeed. I ended up removing it because of the proliferation of modules using embedded postrgesql. On the other hand using something like testcontainers.org may be easier for developers for running their unit tests locally. |
| Comment by Marc Johnson [ 09/May/19 ] |
|
John Malconian Do our current backend CI builds run within a docker container, or on a native OS? |
| Comment by John Malconian [ 09/May/19 ] |
|
They do and it is possible to invoke other containers within the main build container. |
| Comment by Marc Johnson [ 09/May/19 ] |
|
John Malconian Oh, I didn't know it was possible to invoke containers from inside a container! |
| Comment by Julian Ladisch [ 24/Jun/19 ] |
|
On my Windows laptop it takes 47 seconds to start postresql-embedded and 4 seconds to start the postgres docker container. Both times do not include the download time, the download is cached. |
| Comment by Marc Johnson [ 14/Oct/20 ] |
|
I recently asked if all developers had access to Docker, and got the following response from Taras Spashchenko on Slack
I think we need to be mindful of the developers maintaining the modules access to Docker when we roll this out |
| Comment by Julian Ladisch [ 21/Oct/20 ] |
|
https://github.com/MicrosoftDocs/WSL/issues/798 "WSL2 breaks VirtualBox" |
| Comment by Julian Ladisch [ 04/Dec/20 ] |
|
RMB looks for command line options |
| Comment by Adam Dickmeiss [ 04/Dec/20 ] |
|
I don't think we should keep postgresql-embedded under no circumstances. Have two system: containters and real postgres. The embedded postgres takes time to boot on both Linux and Windows. Starting Docker containers also takes minutes - first time - an image is downloaded. That's a price, regardless of platform. Doesn't Docker work on Windows? As far for starting modules with "no postgres", we'd just use containers for that. So the "embedded postgres" functionality would be preserved. |
| Comment by John Malconian [ 08/Jan/21 ] |
|
Please remember that any containers that are created in the CI build pipeline need to be cleaned up after use. |
| Comment by Marc Johnson [ 08/Jan/21 ] |
What does that involve? Is it something that test containers doesn't do already? Does the Okapi build, which uses test containers I believe, leave containers behind? |
| Comment by Marc Johnson [ 08/Jan/21 ] |
Agreed.
Are you proposing that if a module being started (outside of tests, more production like) does not receive external database configuration it would attempt to start a docker container instance? I know this is what RAML Module Builder does with embedded PostgreSQL at the moment. Personally, I would prefer we didn't have that path at all and storage modules had to receive some form of external database configuration (which might refer to a container). I think tools like test containers or embedded PostgreSQL should only be used for assisting with automated tests and not be involved in production code. |
| Comment by John Malconian [ 08/Jan/21 ] |
|
Marc Johnson - I have seen orphaned testcontainers containers left behind on the FOLIO CI build systems and have had to clean them up manually. There could be a few reasons that might account for this: 1. The build failed before the container was cleaned up. I don't know enough yet about testcontainers to determine which of those cases are true. If they can be cleaned up programmatically or they are cleaned up automatically after use, that would be ideal. If not, I can always add a post build cleanup step to the CI pipelines that cleans up container artifacts whether the build fails or succeeds. I would just need guidance on how best to proceed. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
Thanks, that is useful to know. I think we should try to identify the causes of those and figure out how to avoid / mitigate them before we roll this out all RAML Module Builder based modules.
Are you referring to cleaning up the container images or the running instances of those images? |
| Comment by Marc Johnson [ 11/Jan/21 ] |
I think this would be a good opportunity to not have this option for the production code.
I agree with keeping these configuration options. As I understand it, they can refer to any instance of PostgreSQL. That would allow developers to use a hosted, local native or a docker based instance of PostgreSQL as they choose. I think that is sufficient options without retaining the use of embedded PostgreSQL, especially as one of the motivations for this change is to stop relying on a unsupported mechanism that is an impediment to FOLIO officially supporting later PostgreSQL versions. |
| Comment by Marc Johnson [ 11/Jan/21 ] |
|
Craig McNally Jakub Skoczen Is this a tooling decision? |
| Comment by Julian Ladisch [ 13/Jan/21 ] |
|
John Malconian
testcontainers.org should remove all containers in all cases, before reporting success or failure. |
| Comment by Julian Ladisch [ 13/Jan/21 ] |
|
On a Windows 10 Pro laptop I've installed Docker Desktop Community 3.0.4 with default settings, one of the defaults is that "Use the WSL 2 based engine" is enabled. I use maven on Windows (not in WSL 2): $ mvn -version Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: C:\Java\apache-maven-3.6.3 Java version: 11, vendor: Oracle Corporation, runtime: C:\Java\jdk-11 Default locale: de_DE, platform encoding: Cp1252 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" The okapi/okapi-core unit tests that use testcontainers properly start Docker containers and succeed: [INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,079.447 s - in org.folio.okapi.ModuleTest [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.834 s - in org.folio.okapi.service.impl.PostgresHandleTest DockerModuleHandleTest skips one test, though: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 20.302 s - in org.folio.okapi.service.impl.DockerModuleHandleTest |
| Comment by Marc Johnson [ 04/Feb/21 ] |
|
Jakub Skoczen Adam Dickmeiss Julian Ladisch Has the decision that FOLIO recommends testcontainers for testing been made? It seems to me that it has given that both
|
| Comment by Julian Ladisch [ 08/Feb/21 ] |
|
Each module can use for testing whatever the CI pipeline supports. |
| Comment by Hanna Hulevich [ 29/Mar/21 ] |
|
This is done by
|