[FOLIO-2738] Reference environments unavailable due to failure enabling mod-courses Created: 19/Aug/20  Updated: 19/Aug/20  Resolved: 19/Aug/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Marc Johnson Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File courses.log     File inventory-storage.log    
Issue links:
Blocks
is blocked by MODINVSTOR-558 /item-storage/items fails because "mo... Closed
Relates
relates to MODCR-41 Tenant initialization with sample dat... Draft
Sprint: DevOps: Sprint 95
Development Team: FOLIO DevOps

 Description   

Symptoms

Ann-Marie Breaux reported on Slack that various hosted reference environments are unavailable

folio-testing, snapshot, and snapshot-load all seem to be dead. Getting a 503 message

folio-snapshot-core is available.

Investigation

The testing-backend build failed with the following error:

fatal: [10.36.1.198]: FAILED! => {"changed": false, "connection": "close", "content": "POST request for mod-courses-1.0.6-SNAPSHOT.96 /_/tenant failed with 500: Error calling perform() POST http://10.36.1.198:9148/coursereserves/reserves returned status 400: Expected status code 200 for GET request to url http://10.36.1.198:9130/item-storage/items?query=barcode=4539876054383, got 400: ", "content_length": "301", "content_type": "text/plain", "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.198:9130/_/proxy/tenants/diku/install?deploy=true&tenantParameters=loadSample%3Dtrue%2CloadReference%3Dtrue"}

A similar error is present for the snapshot build.

The snapshot load build failed almost immediately. I believe this is a separate known issue, however there is little value in triggering this pipeline again as the issue is likely to be the same.

Hypothesis

The tenant activation API for mod-courses is failing which stops any of the environments where it is included from being built.

This might be caused by a request being made from mod-courses to mod-inventory-storage during this process.

Temporary Workarounds

  • If only platform core (rather than platform complete) modules are needed, then the folio-snapshot-core environment can be used.
  • The course reserves modules can be temporarily removed from platform complete until the issue is resolved.

Observations

It appears that the sample records in mod-courses depend upon the sample records in mod-inventory-storage at runtime and if they aren't present the tenant activation fails. In other modules, these kinds of dependencies (as awkward as they are) have been managed at design / compile time and mismatches result in an inconsistent system rather than failure to enable the module.



 Comments   
Comment by Marc Johnson [ 19/Aug/20 ]

Kurt Nordstrom

The module responsibility wiki page suggests you are the lead maintainer for mod-courses. Is this something you are best placed to investigate?

Comment by Ian Hardy [ 19/Aug/20 ]

I'll attach logs from courses and inventory storage. I suspect the issue is with inventory storage though. Mod courses fails trying to retrieve some sample data from /item-storage/items.

Comment by Jakub Skoczen [ 19/Aug/20 ]

Kurt Nordstrom Mike Taylor In addition to fixing mod-inventory-storage it might be a good idea to add some defensive coding in mod-courses.

Comment by Marc Johnson [ 19/Aug/20 ]

Ian Hardy

I suspect the issue is with inventory storage though. Mod courses fails trying to retrieve some sample data from /item-storage/items.

I believe Zak Burke recently encountered a different issue with mod-configuration when doing a similar query to the root of a collection (/configuration/entries).

curl -i  --location --request GET 'https://folio-snapshot-core-okapi.dev.folio.org:443/configurations/entries' --header 'x-okapi-tenant: diku' --header 'x-okapi-token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkaWt1X2FkbWluIiwidXNlcl9pZCI6IjFiOGNhMjlmLWNkNWYtNWE1NS1hODE2LTY3ODc3NjdmM2E5NSIsImlhdCI6MTU5Nzg0MTU3MCwidGVuYW50IjoiZGlrdSJ9.sv8J-hOi01Iz5JWwhlojeueZUnHwqCHCa5TA_nN2hLM'
HTTP/2 400 
date: Wed, 19 Aug 2020 13:29:31 GMT
content-type: text/plain
x-okapi-trace: GET mod-authtoken-2.6.0-SNAPSHOT.73 http://10.36.1.156:9154/configurations/entries : 202 2246us
x-okapi-trace: GET mod-configuration-5.5.0-SNAPSHOT.77 http://10.36.1.156:9131/configurations/entries : 400 1454us

Both mod-inventory-storage and mod-configuration were upgraded to the latest release of RAML Module Builder and Java 11 yesterday.

Jakub Skoczen Craig McNally Julian Ladisch Adam Dickmeiss Could the two issues be related, and that upgrade have caused these issues?

Comment by Kurt Nordstrom [ 19/Aug/20 ]

Jakub Skoczen Marc Johnson I have created https://folio-org.atlassian.net/browse/MODCR-41

Comment by Craig McNally [ 19/Aug/20 ]

Both mod-inventory-storage and mod-configuration were upgraded to the latest release of RAML Module Builder and Java 11 yesterday.
... Could the two issues be related, and that upgrade have caused these issues?

Seems likely to me, though I'm not even sure what exactly changed in RMB off the top of my head. If this isn't the case, it feels like quite the coincidence... the modules involved, the timing, etc...?

Comment by Marc Johnson [ 19/Aug/20 ]

Craig McNally

Seems likely to me, though I'm not even sure what exactly changed in RMB off the top of my head. If this isn't the case, it feels like quite the coincidence... the modules involved, the timing, etc...?

I think Julian Ladisch has already confirmed this is an issue with finding the module name (which also caused the aborted upgrade of mod-circulation due to the use of this mechanism in the pub-sub client)

Comment by Ian Hardy [ 19/Aug/20 ]

reverted the jdk-11 updates on mod-inventory-storage and mod-configuration so we could get a passing build on the reference envs. There are two new issues to address what went wrong with those updates.

https://folio-org.atlassian.net/browse/MODINVSTOR-558
https://folio-org.atlassian.net/browse/RMB-699

I'll close this for now since we can build instances again and let discussion continue on those new issues.

Generated at Thu Feb 08 23:22:53 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.