[FOLIO-3206] build-platform-complete-snapshot job failure Created: 15/Jun/21 Updated: 17/Jun/21 Resolved: 17/Jun/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | John Malconian | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue links: |
|
||||||||||||
| Sprint: | DevOps Sprint 116 | ||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||
| Description |
|
Investigate new, repeatable build-platform-complete-snapshot job failure. Error message: fatal: [10.36.1.158]: FAILED! => {"changed": false, "connection": "close", "content": "POST request for mod-data-export-spring-1.0.5-SNAPSHOT.93 /_/tenant failed with 500: Liquibase error: Migration failed for change set classpath:db/changelog/changes/create-db.xml::MODEXPS-1@@create-job-table::prokhorovalexey:
Reason: liquibase.exception.DatabaseException: ERROR: function gen_random_uuid() does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 63 [Failed SQL: (0) CREATE TABLE diku_mod_data_export_spring.job (id UUID DEFAULT gen_random_uuid() NOT NULL, name VARCHAR(100), description TEXT, source VARCHAR(50), is_system_source BOOLEAN, type EXPORTTYPE NOT NULL, export_type_specific_parameters JSONB NOT NULL, status JOBSTATUS DEFAULT 'SCHEDULED', files JSONB, start_time TIMESTAMP WITHOUT TIME ZONE, end_time TIMESTAMP WITHOUT TIME ZONE, created_date TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW() NOT NULL, created_by_user_id UUID, created_by_username VARCHAR(50), updated_date TIMESTAMP WITHOUT TIME ZONE, updated_by_user_id UUID, updated_by_username VARCHAR(50), output_format VARCHAR(50), error_details TEXT, batch_status BATCHSTATUS, exit_status JSONB, CONSTRAINT pk_job PRIMARY KEY (id))]", "content_length": "1192", "content_type": "text/plain", "elapsed": 123, "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.158:9130/_/proxy/tenants/diku/install?deploy=true&tenantParameters=loadReference%3Dtrue%2CloadSample%3Dtrue", "vary": "origin"}
|
| Comments |
| Comment by John Malconian [ 15/Jun/21 ] |
|
I believe the error logged above is a bit of a red herring. The likely culprit appears to be mod-tags. Specifically, mod-tags-0.10.0-SNAPSHOT.61, related to this PR: https://github.com/folio-org/mod-tags/pull/51. I've pinned the snapshot builds (core and complete) to mod-tags-0.10.0-SNAPSHOT.60 and the builds complete successfully. Something is wrong with mod-tags-0.10.0-SNAPSHOT.61 tenant initialization. I'll attach logs to this issue as soon as I have them. |
| Comment by John Malconian [ 15/Jun/21 ] |
|
Deepening this mystery, however, is the fact the folio-testing builds work with mod-tags-0.10.0-SNAPSHOT.61. I have not pinned the module for folio-testing builds. |
| Comment by John Malconian [ 15/Jun/21 ] |
|
I've attached the log for mod-tags, but according to the log, there were no errors during tenant init. Will check other logs. |
| Comment by John Malconian [ 15/Jun/21 ] |
|
Attached log for mod-data-export-spring where the 'gen_random_uuid()' error occurs. The log provides more detail on this error. |
| Comment by John Malconian [ 15/Jun/21 ] |
|
So here's what we know.
|
| Comment by John Malconian [ 15/Jun/21 ] |
|
Also relevant: the build-platform-core-snapshot job fails in the same way as build-platform-complete-snapshot and is also successful when mod-tags is pinned to build 60. But the interesting thing is that platform-core does not contain the data-export-spring module. So whatever issue mod-tags build 61 introduced is impacting some other downstream module on that platform. |
| Comment by Pavlo Smahin [ 16/Jun/21 ] |
|
Hey John Malconian. It is really unexpected issue. I think this will be fixed with PR: https://github.com/folio-org/mod-tags/pull/52 |
| Comment by Marc Johnson [ 16/Jun/21 ] |
|
Thanks for making that change. Is that change intended to only affect newly build environments (as it changes an existing migration)? I wonder if this means that PostgreSQL either
|
| Comment by Pavlo Smahin [ 16/Jun/21 ] |
|
Marc Johnson, yes it affects only newly build envs. Postgres documentation has this notice: Remember that the extension itself is not considered to be within any schema: extensions have unqualified names that must be unique database-wide. But objects belonging to the extension can be within schemas. https://www.postgresql.org/docs/9.1/sql-createextension.html |
| Comment by Marc Johnson [ 16/Jun/21 ] |
|
Thanks Pavlo Smahin I think, based upon Pavlo Smahin comment, that the objects (the functions, e.g. gen_random_uuid()) in the extension are registered in a specific namespace. My hypothesis is that mod-tags does this first in snapshot and effectively puts the objects in the namespace for that module. However mod-data-export-spring expects that gen_random_uuid() to be in the public namespace and when it isn’t there, it fails as having not been found. This explains why the create extension migration doesn’t fail, because as far as PostgreSQL is concerned, it is already created so is skipped. |
| Comment by Ann-Marie Breaux (Inactive) [ 17/Jun/21 ] |
|
Hi Marc Johnson and Pavlo Smahin I'm just seeing this thread now. Do we think this will be resolved via
|
| Comment by Pavlo Smahin [ 17/Jun/21 ] |
|
Hi Ann-Marie Breaux. I assume that it was already fixed. But needed confirmation from John Malconian. John, is new builds are successes with the new version of mod-tags? |
| Comment by Marc Johnson [ 17/Jun/21 ] |
I think that change should resolve it (by being consistent with other modules). Unfortunately, a
cc: John Malconian |
| Comment by Ann-Marie Breaux (Inactive) [ 17/Jun/21 ] |
|
Pavlo Smahin Marc Johnson John Malconian Whatever can be done ASAP to get at least one of the envs back to life (testing, snapshot, or snapshot-load) will be most helpful. I think there are several teams trying to verify the last few issues before Juniper releases. |
| Comment by Marc Johnson [ 17/Jun/21 ] |
That needs folks to resolve
|