[FOLIO-1624] Publishing and fetching docker artefacts fails Created: 24/Nov/18  Updated: 03/Jun/20  Resolved: 22/Jan/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: Marc Johnson Assignee: Marc Johnson
Resolution: Done Votes: 0
Labels: devops, platform-backlog, sprint52, sprint53, sprint54
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks CIRC-154 design and implement a "check-in-by-b... Closed
Sprint:
Development Team: Core: Platform

 Description   

The overnight environment builds (folio-testing-backend, folio-snapshot) appear to fail due to not being able to fetch docker artefacts. Whilst the modules referred to are different, the error messages seem to be similar. As an example:
Testing

TASK [folio-ansible/roles/okapi-tenant-deploy : Post install list for deployment and enabling] ***
fatal: [10.36.1.223]: FAILED! => {"changed": false, "connection": "close", "content": "http://localhost:4243/images/folioci/mod-configuration:5.0.2-SNAPSHOT.50/json HTTP error 404\n{\"message\":\"no such image: folioci/mod-configuration:5.0.2-SNAPSHOT.50: No such image: folioci/mod-configuration:5.0.2-SNAPSHOT.50\"}\n", "content_length": "226", "content_type": "text/plain", "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.223:9130/_/proxy/tenants/diku/install?deploy=true"}

Snapshot

failed: [10.36.1.185] (item=mod-feesfines-15.0.2-SNAPSHOT.18) => {"changed": false, "item": "mod-feesfines-15.0.2-SNAPSHOT.18", "msg": "Error pulling image folioci/mod-feesfines:15.0.2-SNAPSHOT.18 - 404 Client Error: Not Found (\"{\"message\":\"error parsing HTTP 404 response body: invalid character 'p' after top-level value: \\\"404 page not found\\\\n\\\"\"}\")"}
changed: [10.36.1.185] => (item=mod-vendors-1.0.2-SNAPSHOT.27)
changed: [10.36.1.185] => (item=mod-codex-inventory-1.4.0-SNAPSHOT.65)
FATAL: command execution failed

Publishing of docker images also appears to fail. For example the [mod-inventory-storage 13.1.0 release build] or mod-inventory-storage-master build:
Release build

d18b0764eff2: Waiting
unauthorized: authentication required
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // dir
[Pipeline] echo
script returned exit code 1

Master

e620de361ca9: Pushed
blob upload unknown
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // dir
[Pipeline] echo
script returned exit code 1


 Comments   
Comment by Marc Johnson [ 24/Nov/18 ]

I've disabled the stripes testing hourly build, in order to avoid continued failures.

Comment by Marc Johnson [ 24/Nov/18 ]

This appears to be resolved. The release and master builds have been re-run successfully. As has the folio-testing-backend environment build. Am rebuilding the folio-testing-backend VM and folio-snapshot environment.

Comment by Marc Johnson [ 24/Nov/18 ]

It would appear that I spoke too soon about this being fully resolved (or there is a different, related issue). The re-ran snapshot and testing-backend builds failed:

Snapshot environment

TASK [folio-ansible/roles/okapi-deploy-config : Pull Docker images] ************
changed: [10.36.1.139] => (item=mod-users-15.3.0-SNAPSHOT.56)
changed: [10.36.1.139] => (item=mod-permissions-5.4.0-SNAPSHOT.34)
changed: [10.36.1.139] => (item=mod-login-4.5.1-SNAPSHOT.32)
changed: [10.36.1.139] => (item=mod-inventory-storage-13.3.0-SNAPSHOT.203)
changed: [10.36.1.139] => (item=mod-password-validator-1.0.1-SNAPSHOT.12)
failed: [10.36.1.139] (item=mod-users-bl-4.0.4-SNAPSHOT.38) => {"changed": false, "item": "mod-users-bl-4.0.4-SNAPSHOT.38", "msg": "Error pulling image folioci/mod-users-bl:4.0.4-SNAPSHOT.38 - 404 Client Error: Not Found (\"{\"message\":\"error parsing HTTP 404 response body: invalid character 'p' after top-level value: \\\"404 page not found\\\\n\\\"\"}\")"}
failed: [10.36.1.139] (item=mod-authtoken-2.0.2-SNAPSHOT.35) => {"changed": false, "item": "mod-authtoken-2.0.2-SNAPSHOT.35", "msg": "Error pulling image folioci/mod-authtoken:2.0.2-SNAPSHOT.35 - 404 Client Error: Not Found (\"{\"message\":\"error parsing HTTP 404 response body: invalid character 'p' after top-level value: \\\"404 page not found\\\\n\\\"\"}\")"}
changed: [10.36.1.139] => (item=mod-configuration-5.0.2-SNAPSHOT.50)
failed: [10.36.1.139] (item=mod-calendar-1.0.5-SNAPSHOT.63) => {"changed": false, "item": "mod-calendar-1.0.5-SNAPSHOT.63", "msg": "Error pulling image folioci/mod-calendar:1.0.5-SNAPSHOT.63 - 404 Client Error: Not Found (\"{\"message\":\"error parsing HTTP 404 response body: invalid character 'p' after top-level value: \\\"404 page not found\\\\n\\\"\"}\")"}

Testing backend

    testing-backend: fatal: [default]: FAILED! => {"changed": false, "connection": "close", "content": "http://localhost:4243/images/folioci/mod-user-import:3.1.1-SNAPSHOT.31/json HTTP error 404\n{\"message\":\"no such image: folioci/mod-user-import:3.1.1-SNAPSHOT.31: No such image: folioci/mod-user-import:3.1.1-SNAPSHOT.31\"}\n", "content_length": "220", "content_type": "text/plain", "msg": "Status code was not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.0.2.15:9130/_/proxy/tenants/diku/install?deploy=true"}
Comment by Marc Johnson [ 26/Nov/18 ]

This seems intermittent, most of the overnight builds succeeded last night, except the testing-backend (a second run succeeded)

    testing-backend: TASK [okapi-tenant-deploy : Post install list for deployment and enabling] *****
    testing-backend: fatal: [default]: FAILED! => {"changed": false, "connection": "close", "content": "http://localhost:4243/images/folioci/mod-permissions:5.4.0-SNAPSHOT.34/json HTTP error 404\n{\"message\":\"no such image: folioci/mod-permissions:5.4.0-SNAPSHOT.34: No such image: folioci/mod-permissions:5.4.0-SNAPSHOT.34\"}\n", "content_length": "220", "content_type": "text/plain", "msg": "Status code was not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.0.2.15:9130/_/proxy/tenants/diku/install?deploy=true"}

Maybe it only affects some agents in our CI setup? Or maybe it is based upon when builds run?

Comment by Jakub Skoczen [ 22/Jan/19 ]

Marc Johnson is this still an open issue? I am closing – reopen if its not fixed.

Generated at Thu Feb 08 23:14:42 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.