[FOLIO-836] Intermittent failure on folio-testing-backend01 build Created: 15/Sep/17 Updated: 01/Apr/19 Resolved: 01/Apr/19 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P3 |
| Reporter: | Wayne Schneider | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 30 minutes | ||
| Original estimate: | Not Specified | ||
| Sprint: | |
| Development Team: | Core: Platform |
| Description |
|
E.g.: https://jenkins-aws.indexdata.com/job/Automation/job/folio-testing-backend01/105/console TASK [folio-ansible/roles/mod-auth-data : Enable mod-auth modules for tenant] *** failed: [10.36.1.229] (item=(0, {u'id': u'mod-permissions-5.0.1-SNAPSHOT.4', u'name': u'permissions'})) => {"changed": false, "connection": "close", "content": "Request for mod-permissions-5.0.1-SNAPSHOT.4 /_/tenant failed with Connection refused: /10.36.1.229:9132", "content_length": "104", "content_type": "text/plain", "failed": true, "item": [0, {"id": "mod-permissions-5.0.1-SNAPSHOT.4", "name": "permissions"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"} changed: [10.36.1.229] => (item=(1, {u'id': u'mod-authtoken-1.0.1-SNAPSHOT.2', u'name': u'authtoken'})) failed: [10.36.1.229] (item=(2, {u'id': u'mod-login-4.0.1-SNAPSHOT.2', u'name': u'login'})) => {"changed": false, "connection": "close", "content": "Request for mod-login-4.0.1-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.229:9133", "content_length": "98", "content_type": "text/plain", "failed": true, "item": [2, {"id": "mod-login-4.0.1-SNAPSHOT.2", "name": "login"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"} failed: [10.36.1.229] (item=(3, {u'id': u'mod-users-14.2.2-SNAPSHOT.2', u'name': u'users'})) => {"changed": false, "connection": "close", "content": "Request for mod-users-14.2.2-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.229:9135", "content_length": "99", "content_type": "text/plain", "failed": true, "item": [3, {"id": "mod-users-14.2.2-SNAPSHOT.2", "name": "users"}], "msg": "Status code was not [201]: HTTP Error 500: Internal Server Error", "redirected": false, "status": 500, "url": "http://10.36.1.229:9130/_/proxy/tenants/diku/modules"} |
| Comments |
| Comment by Wayne Schneider [ 15/Sep/17 ] |
|
Okapi seems to have trouble communicating with all its modules. |
| Comment by John Malconian [ 16/Sep/17 ] |
|
Looks like the backend build failed again tonight. And again when I re-ran it. Here is the okapi log: 2017-09-16 02:10:45,789 INFO ProxyContext 471312/proxy REQ 10.36.1.5:48014 okapi.supertenant P OST /_/proxy/tenants/diku/modules okapi-2.0.0 2017-09-16 02:10:45,816 INFO OkapiClient 471312/proxy;538377/tenant REQ okapiClient diku POST http://10.36.1.5:9132/_/tenant 2017-09-16 02:10:45,822 WARN OkapiClient 471312/proxy;538377/tenant OkapiClient exception: io .netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9132: Connecti on refused: /10.36.1.5:9132 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9132 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j ava:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748) 2017-09-16 02:10:45,828 WARN ProxyContext 471312/proxy Request for mod-permissions-5.0.1-SNAPS HOT.4 /_/tenant failed with Connection refused: /10.36.1.5:9132 2017-09-16 02:10:45,828 INFO ProxyContext 471312/proxy RES 500 okapi Request for mod-permissio ns-5.0.1-SNAPSHOT.4 /_/tenant failed with Connection refused: /10.36.1.5:9132 2017-09-16 02:10:46,179 INFO ProxyContext 042011/proxy REQ 10.36.1.5:48018 okapi.supertenant P OST /_/proxy/tenants/diku/modules okapi-2.0.0 2017-09-16 02:10:46,229 INFO ProxyContext 042011/proxy RES 201 okapi-2.0.0 /_/proxy/tenants/di ku/modules 2017-09-16 02:10:46,542 INFO ProxyContext 559806/proxy REQ 10.36.1.5:48020 okapi.supertenant P OST /_/proxy/tenants/diku/modules okapi-2.0.0 2017-09-16 02:10:46,543 INFO OkapiClient 559806/proxy;184482/tenant REQ okapiClient diku POST http://10.36.1.5:9133/_/tenant 2017-09-16 02:10:46,544 WARN OkapiClient 559806/proxy;184482/tenant OkapiClient exception: io .netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9133: Connecti on refused: /10.36.1.5:9133 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9133 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j ava:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748) 2017-09-16 02:10:46,544 WARN ProxyContext 559806/proxy Request for mod-login-4.0.1-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9133 2017-09-16 02:10:46,544 INFO ProxyContext 559806/proxy RES 500 okapi Request for mod-login-4.0 .1-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9133 2017-09-16 02:10:46,857 INFO ProxyContext 200175/proxy REQ 10.36.1.5:48024 okapi.supertenant P OST /_/proxy/tenants/diku/modules okapi-2.0.0 2017-09-16 02:10:46,861 INFO OkapiClient 200175/proxy;653116/tenant REQ okapiClient diku POST http://10.36.1.5:9135/_/tenant 2017-09-16 02:10:46,861 WARN OkapiClient 200175/proxy;653116/tenant OkapiClient exception: io .netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9135: Connecti on refused: /10.36.1.5:9135 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.36.1.5:9135 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:347) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.j ava:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748) 2017-09-16 02:10:46,862 WARN ProxyContext 200175/proxy Request for mod-users-14.2.2-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9135 2017-09-16 02:10:46,862 INFO ProxyContext 200175/proxy RES 500 okapi Request for mod-users-14. 2.2-SNAPSHOT.2 /_/tenant failed with Connection refused: /10.36.1.5:9135 |
| Comment by Wayne Schneider [ 16/Sep/17 ] |
|
Added a longer wait time before attempting to enable mod-permissions. Feels like a stopgap, though. |
| Comment by Wayne Schneider [ 01/Apr/19 ] |
|
Have not had this error in quite a long time, likely it was resolved long ago. |