Issues

Select view

Select search mode

 
15 of 15

Upgrading tenant from Q4 to Q1, Okapi reports hazelcast errors

Cannot Reproduce

Description

When in-place upgrading a Folio Q4 instance with data loaded - to Q1, Okapi reports Hazelcast errors in its log output that may be related to why the tenant upgrade fails.

Okapi version was v2.38, running 3 in a cluster.

The final request to Okapi was:

curl -w '\n' -D - -X POST -H "Content-type: application/json" \ -d @upgrade.json \ $OKAPI_URL/_/proxy/tenants/$TENANT_ID/install?deploy=false\&preRelease=false\&tenantParameters=loadSample%3Dfalse%2CloadReference%3Dfalse

The IP address listed are the various Okapi cluster members assigned by K8s on the K8s private network, and their Hazelcast ports.

The errors are as follows:

15:01:44 WARN ? Connecting to server 10.42.18.7:5702 failed io.netty.channel.ConnectTimeoutException: connection timed out: /10.42.18.7:5702 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:580) [okapi-core-fat.jar:?] at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [okapi-core-fat.jar:?] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:139) [okapi-core-fat.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) [okapi-core-fat.jar:?] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [okapi-core-fat.jar:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [okapi-core-fat.jar:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [okapi-core-fat.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] 15:01:44 WARN ? Connecting to server 10.42.17.6:5702 failed io.netty.channel.ConnectTimeoutException: connection timed out: /10.42.17.6:5702 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:580) [okapi-core-fat.jar:?] at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [okapi-core-fat.jar:?] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:139) [okapi-core-fat.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) [okapi-core-fat.jar:?] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [okapi-core-fat.jar:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [okapi-core-fat.jar:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [okapi-core-fat.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] 15:01:44 WARN ? Connecting to server 10.42.16.38:5702 failed io.netty.channel.ConnectTimeoutException: connection timed out: /10.42.16.38:5702 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:580) [okapi-core-fat.jar:?] at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [okapi-core-fat.jar:?] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:139) [okapi-core-fat.jar:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) [okapi-core-fat.jar:?] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) [okapi-core-fat.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [okapi-core-fat.jar:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [okapi-core-fat.jar:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [okapi-core-fat.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
5:34:27 WARN jFactory$Log4jLogger [10.42.7.71]:5701 [folio-test] [3.12] Connection[id=4, /10.42.7.71:5701->/10.42.17.122:35958, qualifier=null, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=4, /10.42.7.71:5701->/10.42.17.122:35958, qualifier=null, endpoint=null, alive=true, type=NONE], thread=hz._hzInstance_1_folio-test.IO.thread-in-1 java.lang.IllegalStateException: REST API is not enabled. at com.hazelcast.nio.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:96) ~[okapi-core-fat.jar:?] at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:135) ~[okapi-core-fat.jar:?] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:369) [okapi-core-fat.jar:?] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:354) [okapi-core-fat.jar:?] at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:280) [okapi-core-fat.jar:?] at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:235) [okapi-core-fat.jar:?]

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Checklist

hide

TestRail: Results

Details

Assignee

Reporter

Priority

Sprint

Development Team

Core: Platform

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs
Created April 17, 2020 at 4:12 PM
Updated August 10, 2020 at 12:18 PM
Resolved August 10, 2020 at 12:18 PM

Activity

Show:

Oleksii Popov August 10, 2020 at 12:18 PM

Issues can not be reproduced.

Please raise a new bug in case of this issues appears.

jroot June 29, 2020 at 2:42 PM

I will try this on the Folio Q2-2020 instances we will spin up - using Okapi v3.

Adam Dickmeiss May 20, 2020 at 11:51 AM

Perhaps. Any chance you could try that? Have no idea whether it would help the situation since I can't reproduce.

jroot May 19, 2020 at 10:45 PM

should setting -Dhazelcast.rest.enabled=true be the default then?

Adam Dickmeiss May 19, 2020 at 12:57 PM

I have a hard time reproducing this. I tried install + upgrade on Okapi in clustered mode.. That works.. But I can Google, like everyone else.

https://stackoverflow.com/questions/37070845/hazelcast-rest-api

?

TestRail: Cases
TestRail: Runs

Flag notifications