[FOLIO-3206] build-platform-complete-snapshot job failure Created: 15/Jun/21  Updated: 17/Jun/21  Resolved: 17/Jun/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File mod-data-export-spring.log.gz     File mod-tags.log.gz    
Issue links:
Relates
relates to MODTAG-52 Migrate application from RMB and Vert... Closed
relates to MODTAG-54 MODTAG(mod-tags) Release for R2 2021 Closed
Sprint: DevOps Sprint 116
Development Team: FOLIO DevOps

 Description   

Investigate new, repeatable build-platform-complete-snapshot job failure. Error message:

fatal: [10.36.1.158]: FAILED! => {"changed": false, "connection": "close", "content": "POST request for mod-data-export-spring-1.0.5-SNAPSHOT.93 /_/tenant failed with 500: Liquibase error: Migration failed for change set classpath:db/changelog/changes/create-db.xml::MODEXPS-1@@create-job-table::prokhorovalexey:
     Reason: liquibase.exception.DatabaseException: ERROR: function gen_random_uuid() does not exist
  Hint: No function matches the given name and argument types. You might need to add explicit type casts.
  Position: 63 [Failed SQL: (0) CREATE TABLE diku_mod_data_export_spring.job (id UUID DEFAULT gen_random_uuid() NOT NULL, name VARCHAR(100), description TEXT, source VARCHAR(50), is_system_source BOOLEAN, type EXPORTTYPE NOT NULL, export_type_specific_parameters JSONB NOT NULL, status JOBSTATUS DEFAULT 'SCHEDULED', files JSONB, start_time TIMESTAMP WITHOUT TIME ZONE, end_time TIMESTAMP WITHOUT TIME ZONE, created_date TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW() NOT NULL, created_by_user_id UUID, created_by_username VARCHAR(50), updated_date TIMESTAMP WITHOUT TIME ZONE, updated_by_user_id UUID, updated_by_username VARCHAR(50), output_format VARCHAR(50), error_details TEXT, batch_status BATCHSTATUS, exit_status JSONB, CONSTRAINT pk_job PRIMARY KEY (id))]", "content_length": "1192", "content_type": "text/plain", "elapsed": 123, "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.158:9130/_/proxy/tenants/diku/install?deploy=true&tenantParameters=loadReference%3Dtrue%2CloadSample%3Dtrue", "vary": "origin"}


 Comments   
Comment by John Malconian [ 15/Jun/21 ]

I believe the error logged above is a bit of a red herring. The likely culprit appears to be mod-tags. Specifically, mod-tags-0.10.0-SNAPSHOT.61, related to this PR: https://github.com/folio-org/mod-tags/pull/51. I've pinned the snapshot builds (core and complete) to mod-tags-0.10.0-SNAPSHOT.60 and the builds complete successfully. Something is wrong with mod-tags-0.10.0-SNAPSHOT.61 tenant initialization. I'll attach logs to this issue as soon as I have them.

Comment by John Malconian [ 15/Jun/21 ]

Deepening this mystery, however, is the fact the folio-testing builds work with mod-tags-0.10.0-SNAPSHOT.61. I have not pinned the module for folio-testing builds.

Comment by John Malconian [ 15/Jun/21 ]

I've attached the log for mod-tags, but according to the log, there were no errors during tenant init. Will check other logs.

Comment by John Malconian [ 15/Jun/21 ]

Attached log for mod-data-export-spring where the 'gen_random_uuid()' error occurs. The log provides more detail on this error.

Comment by John Malconian [ 15/Jun/21 ]

So here's what we know.

  • Something changed with the release of mod-tags-0.10.0-SNAPSHOT.61 (Spring-based) that somehow impacts mod-data-export-spring (Spring-based) during tenant initialization. The error does not occur in mod-data-export-spring when mod-tags is pinned to mod-tags-0.10.0-SNAPSHOT.60.
  • The error seems to occur only on snapshot builds (folio-snapshot*), but does not seem to occur on folio-testing builds. It could be the modules are initialized in a slightly different order in testing builds that doesn't cause the error. Snapshot builds and testing builds are performed a little differently. That's my theory at this point.
Comment by John Malconian [ 15/Jun/21 ]

Also relevant: the build-platform-core-snapshot job fails in the same way as build-platform-complete-snapshot and is also successful when mod-tags is pinned to build 60. But the interesting thing is that platform-core does not contain the data-export-spring module. So whatever issue mod-tags build 61 introduced is impacting some other downstream module on that platform.

Comment by Pavlo Smahin [ 16/Jun/21 ]

Hey John Malconian. It is really unexpected issue. I think this will be fixed with PR: https://github.com/folio-org/mod-tags/pull/52

Comment by Marc Johnson [ 16/Jun/21 ]

Pavlo Smahin

Thanks for making that change. Is that change intended to only affect newly build environments (as it changes an existing migration)?

John Malconian Pavlo Smahin

I wonder if this means that PostgreSQL either

  • only allows one instance of an extension on a server, irrespective of namespace and any subsequent registrations either fail or supersede the previous one?
  • a check for if an extension exists is ignorant of the namespace, so if it's installed in a local namespace prior to the public one, it won't get installed?
Comment by Pavlo Smahin [ 16/Jun/21 ]

Marc Johnson, yes it affects only newly build envs.

Postgres documentation has this notice:

Remember that the extension itself is not considered to be within any schema: extensions have unqualified names that must be unique database-wide. But objects belonging to the extension can be within schemas.

https://www.postgresql.org/docs/9.1/sql-createextension.html

Comment by Marc Johnson [ 16/Jun/21 ]

Thanks Pavlo Smahin

I think, based upon Pavlo Smahin comment, that the objects (the functions, e.g. gen_random_uuid()) in the extension are registered in a specific namespace.

My hypothesis is that mod-tags does this first in snapshot and effectively puts the objects in the namespace for that module. However mod-data-export-spring expects that gen_random_uuid() to be in the public namespace and when it isn’t there, it fails as having not been found.

This explains why the create extension migration doesn’t fail, because as far as PostgreSQL is concerned, it is already created so is skipped.

Comment by Ann-Marie Breaux (Inactive) [ 17/Jun/21 ]

Hi Marc Johnson and Pavlo Smahin I'm just seeing this thread now. Do we think this will be resolved via MODTAG-52 Closed , or is additional MODTAG work needed?

Comment by Pavlo Smahin [ 17/Jun/21 ]

Hi Ann-Marie Breaux. I assume that it was already fixed. But needed confirmation from John Malconian. John, is new builds are successes with the new version of mod-tags?

Comment by Marc Johnson [ 17/Jun/21 ]

Pavlo Smahin Ann-Marie Breaux

Do we think this will be resolved via MODTAG-52 Closed , or is additional MODTAG work needed?

I think that change should resolve it (by being consistent with other modules).

Unfortunately, a FOLIO-3209 Closed means that the environments cannot be built meaning we likely cannot confirm this until that is resolved.

cc: John Malconian

Comment by Ann-Marie Breaux (Inactive) [ 17/Jun/21 ]

Pavlo Smahin Marc Johnson John Malconian Whatever can be done ASAP to get at least one of the envs back to life (testing, snapshot, or snapshot-load) will be most helpful. I think there are several teams trying to verify the last few issues before Juniper releases.

Comment by Marc Johnson [ 17/Jun/21 ]

Ann-Marie Breaux

Whatever can be done ASAP to get at least one of the envs back to life (testing, snapshot, or snapshot-load) will be most helpful. I think there are several teams trying to verify the last few issues before Juniper releases.

That needs folks to resolve FOLIO-3209 Closed

Generated at Thu Feb 08 23:26:24 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.