[FOLIO-836] Intermittent failure on folio-testing-backend01 build Created: 15/Sep/17  Updated: 01/Apr/19  Resolved: 01/Apr/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P3
Reporter: Wayne Schneider Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: 30 minutes
Original estimate: Not Specified

Sprint:
Development Team: Core: Platform

 Description   

E.g.: https://jenkins-aws.indexdata.com/job/Automation/job/folio-testing-backend01/105/console

TASK [folio-ansible/roles/mod-auth-data : Enable mod-auth modules for tenant] ***
failed: [10.36.1.229] (item=(0, {u'id': u'mod-permissions-5.0.1-SNAPSHOT.4', u'name': u'permissions'})) => {"changed": false, "connection": "close", "content": "Request for mod-permissions-5.0.1-SNAPSHOT.4 /_/tenant failed with Connection refused: /10.36.1.229:9132", "content_length": "104", "content_type": "text/plain", "failed": true, "item": [0, {"id": "mod-permissions-5.0.1-SNAPSHOT.4", "name": "permissions"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"}
changed: [10.36.1.229] => (item=(1, {u'id': u'mod-authtoken-1.0.1-SNAPSHOT.2', u'name': u'authtoken'}))
failed: [10.36.1.229] (item=(2, {u'id': u'mod-login-4.0.1-SNAPSHOT.2', u'name': u'login'})) => {"changed": false, "connection": "close", "content": "Request for mod-login-4.0.1-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.229:9133", "content_length": "98", "content_type": "text/plain", "failed": true, "item": [2, {"id": "mod-login-4.0.1-SNAPSHOT.2", "name": "login"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"}
failed: [10.36.1.229] (item=(3, {u'id': u'mod-users-14.2.2-SNAPSHOT.2', u'name': u'users'})) => {"changed": false, "connection": "close", "content": "Request for mod-users-14.2.2-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.229:9135", "content_length": "99", "content_type": "text/plain", "failed": true, "item": [3, {"id": "mod-users-14.2.2-SNAPSHOT.2", "name": "users"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"}


 Comments   
Comment by Wayne Schneider [ 15/Sep/17 ]

Okapi seems to have trouble communicating with all its modules.

Comment by John Malconian [ 16/Sep/17 ]

Looks like the backend build failed again tonight. And again when I re-ran it. Here is the okapi log:

2017-09-16 02:10:45,789 INFO  ProxyContext         471312/proxy REQ 10.36.1.5:48014 okapi.supertenant P
OST /_/proxy/tenants/diku/modules okapi-2.0.0
2017-09-16 02:10:45,816 INFO  OkapiClient          471312/proxy;538377/tenant REQ okapiClient diku POST
 http://10.36.1.5:9132/_/tenant
2017-09-16 02:10:45,822 WARN  OkapiClient          471312/proxy;538377/tenant OkapiClient exception: io
.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9132: Connecti
on refused: /10.36.1.5:9132
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9132
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j
ava:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
2017-09-16 02:10:45,828 WARN  ProxyContext         471312/proxy Request for mod-permissions-5.0.1-SNAPS
HOT.4 /_/tenant failed with Connection refused: /10.36.1.5:9132
2017-09-16 02:10:45,828 INFO  ProxyContext         471312/proxy RES 500 okapi Request for mod-permissio
ns-5.0.1-SNAPSHOT.4 /_/tenant failed with Connection refused: /10.36.1.5:9132
2017-09-16 02:10:46,179 INFO  ProxyContext         042011/proxy REQ 10.36.1.5:48018 okapi.supertenant P
OST /_/proxy/tenants/diku/modules okapi-2.0.0
2017-09-16 02:10:46,229 INFO  ProxyContext         042011/proxy RES 201 okapi-2.0.0 /_/proxy/tenants/di
ku/modules
2017-09-16 02:10:46,542 INFO  ProxyContext         559806/proxy REQ 10.36.1.5:48020 okapi.supertenant P
OST /_/proxy/tenants/diku/modules okapi-2.0.0
2017-09-16 02:10:46,543 INFO  OkapiClient          559806/proxy;184482/tenant REQ okapiClient diku POST
 http://10.36.1.5:9133/_/tenant
2017-09-16 02:10:46,544 WARN  OkapiClient          559806/proxy;184482/tenant OkapiClient exception: io
.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9133: Connecti
on refused: /10.36.1.5:9133
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9133
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j
ava:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
2017-09-16 02:10:46,544 WARN  ProxyContext         559806/proxy Request for mod-login-4.0.1-SNAPSHOT.2 
/_/tenant failed with Connection refused: /10.36.1.5:9133
2017-09-16 02:10:46,544 INFO  ProxyContext         559806/proxy RES 500 okapi Request for mod-login-4.0
.1-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9133
2017-09-16 02:10:46,857 INFO  ProxyContext         200175/proxy REQ 10.36.1.5:48024 okapi.supertenant P
OST /_/proxy/tenants/diku/modules okapi-2.0.0
2017-09-16 02:10:46,861 INFO  OkapiClient          200175/proxy;653116/tenant REQ okapiClient diku POST
 http://10.36.1.5:9135/_/tenant
2017-09-16 02:10:46,861 WARN  OkapiClient          200175/proxy;653116/tenant OkapiClient exception: io
.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9135: Connecti
on refused: /10.36.1.5:9135
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9135
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j
ava:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
2017-09-16 02:10:46,862 WARN  ProxyContext         200175/proxy Request for mod-users-14.2.2-SNAPSHOT.2
 /_/tenant failed with Connection refused: /10.36.1.5:9135
2017-09-16 02:10:46,862 INFO  ProxyContext         200175/proxy RES 500 okapi Request for mod-users-14.
2.2-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9135
Comment by Wayne Schneider [ 16/Sep/17 ]

Added a longer wait time before attempting to enable mod-permissions. Feels like a stopgap, though.

Comment by Wayne Schneider [ 01/Apr/19 ]

Have not had this error in quite a long time, likely it was resolved long ago.

Generated at Thu Feb 08 23:08:41 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.