[FOLIO-2738] Reference environments unavailable due to failure enabling mod-courses Created: 19/Aug/20 Updated: 19/Aug/20 Resolved: 19/Aug/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Marc Johnson | Assignee: | Ian Hardy |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue links: |
|
||||||||||||||||
| Sprint: | DevOps: Sprint 95 | ||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||
| Description |
|
Symptoms Ann-Marie Breaux reported on Slack that various hosted reference environments are unavailable
folio-snapshot-core is available. Investigation The testing-backend build failed with the following error:
fatal: [10.36.1.198]: FAILED! => {"changed": false, "connection": "close", "content": "POST request for mod-courses-1.0.6-SNAPSHOT.96 /_/tenant failed with 500: Error calling perform() POST http://10.36.1.198:9148/coursereserves/reserves returned status 400: Expected status code 200 for GET request to url http://10.36.1.198:9130/item-storage/items?query=barcode=4539876054383, got 400: ", "content_length": "301", "content_type": "text/plain", "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.198:9130/_/proxy/tenants/diku/install?deploy=true&tenantParameters=loadSample%3Dtrue%2CloadReference%3Dtrue"}
A similar error is present for the snapshot build. The snapshot load build failed almost immediately. I believe this is a separate known issue, however there is little value in triggering this pipeline again as the issue is likely to be the same. Hypothesis The tenant activation API for mod-courses is failing which stops any of the environments where it is included from being built. This might be caused by a request being made from mod-courses to mod-inventory-storage during this process. Temporary Workarounds
Observations It appears that the sample records in mod-courses depend upon the sample records in mod-inventory-storage at runtime and if they aren't present the tenant activation fails. In other modules, these kinds of dependencies (as awkward as they are) have been managed at design / compile time and mismatches result in an inconsistent system rather than failure to enable the module. |
| Comments |
| Comment by Marc Johnson [ 19/Aug/20 ] |
|
The module responsibility wiki page suggests you are the lead maintainer for mod-courses. Is this something you are best placed to investigate? |
| Comment by Ian Hardy [ 19/Aug/20 ] |
|
I'll attach logs from courses and inventory storage. I suspect the issue is with inventory storage though. Mod courses fails trying to retrieve some sample data from /item-storage/items. |
| Comment by Jakub Skoczen [ 19/Aug/20 ] |
|
Kurt Nordstrom Mike Taylor In addition to fixing mod-inventory-storage it might be a good idea to add some defensive coding in mod-courses. |
| Comment by Marc Johnson [ 19/Aug/20 ] |
I believe Zak Burke recently encountered a different issue with mod-configuration when doing a similar query to the root of a collection (/configuration/entries). curl -i --location --request GET 'https://folio-snapshot-core-okapi.dev.folio.org:443/configurations/entries' --header 'x-okapi-tenant: diku' --header 'x-okapi-token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkaWt1X2FkbWluIiwidXNlcl9pZCI6IjFiOGNhMjlmLWNkNWYtNWE1NS1hODE2LTY3ODc3NjdmM2E5NSIsImlhdCI6MTU5Nzg0MTU3MCwidGVuYW50IjoiZGlrdSJ9.sv8J-hOi01Iz5JWwhlojeueZUnHwqCHCa5TA_nN2hLM' HTTP/2 400 date: Wed, 19 Aug 2020 13:29:31 GMT content-type: text/plain x-okapi-trace: GET mod-authtoken-2.6.0-SNAPSHOT.73 http://10.36.1.156:9154/configurations/entries : 202 2246us x-okapi-trace: GET mod-configuration-5.5.0-SNAPSHOT.77 http://10.36.1.156:9131/configurations/entries : 400 1454us Both mod-inventory-storage and mod-configuration were upgraded to the latest release of RAML Module Builder and Java 11 yesterday. Jakub Skoczen Craig McNally Julian Ladisch Adam Dickmeiss Could the two issues be related, and that upgrade have caused these issues? |
| Comment by Kurt Nordstrom [ 19/Aug/20 ] |
|
Jakub Skoczen Marc Johnson I have created https://folio-org.atlassian.net/browse/MODCR-41 |
| Comment by Craig McNally [ 19/Aug/20 ] |
Seems likely to me, though I'm not even sure what exactly changed in RMB off the top of my head. If this isn't the case, it feels like quite the coincidence... the modules involved, the timing, etc...? |
| Comment by Marc Johnson [ 19/Aug/20 ] |
I think Julian Ladisch has already confirmed this is an issue with finding the module name (which also caused the aborted upgrade of mod-circulation due to the use of this mechanism in the pub-sub client) |
| Comment by Ian Hardy [ 19/Aug/20 ] |
|
reverted the jdk-11 updates on mod-inventory-storage and mod-configuration so we could get a passing build on the reference envs. There are two new issues to address what went wrong with those updates. https://folio-org.atlassian.net/browse/MODINVSTOR-558 I'll close this for now since we can build instances again and let discussion continue on those new issues. |