okapi cluster not ensure module descriptor data consistency.

Description

cluster environment

node1

192.168.2.101

node2

192.168.2.102

node3

192.168.2.103

  • okapi 2.17.0

  • mod-users 14.1.1

scene

step 1. okapi start cluster mode on node1,node2,node3.

step 2. submit mod-users's ModuleDescriptor on node1.

step 3. deploy mod-users's instance on node1,node2,node3.

After the above process is completed, the cluster service is ok.

GET http://node1:9130/_/discovery/nodes

GET http://node1:9130/_/discovery/modules

But when all node shutdown,then first the node2'okapi start cluster mode.

node2 startup fail, got exception:

I found out the node2 startup fail that is due to module descriptor not persisted to node2 okapi's database, which cause [ Deploy failed: java.lang.Throwable: Module mod-users-14.1.1 not found ] exception.

submit mod-users's ModuleDescriptor on node2 is not persisted when cluster already have this module descriptor.

Expect cluster keep okapi data consistency.

CSP Request Details

None

CSP Rejection Details

None

Potential Workaround

None

Checklist

hide

TestRail: Results

Activity

Show:

jie ma (fluagen)March 25, 2019 at 7:13 AM

I created issue for the last question.

Adam DickmeissMarch 18, 2019 at 1:20 PM

I don't know what you mean by your last description. What exactly are you doing? Okapi forwarding to other Okapis??

jie ma (fluagen)March 13, 2019 at 9:18 AM

I change all Okapi's are connect to the same database, the above error has been fixed. Thank you for help.

But I found another problem.

offline node's module service that it can still be found, actually this module service instance have been shutdown when node's okapi shutdown.

The problem cause that cluster‘s service is unavailability sometimes, because okapi may random forwarding requests to instances of other nodes.

Adam DickmeissMarch 12, 2019 at 2:43 PM

Question: are all Okapi using the same Postgres instance?

Okapi assumes that all Okapi's are talking to the same database.. In most cases, only ONE Okapi instance will actually write to the Postgres storage.

If you are using a database instance "per" Okapi.. Then it's a total no-go. Not working. And never will.

Adam DickmeissMarch 12, 2019 at 2:25 PM

There have been a couple of fixes for Okapi since the version from August 2018 you are using. Can you please, repeat the test with Okapi 2.25.0 ?

It should be fully backwards compatible with 2.17.0, so nothing else in your env should need changing.

Won't Do

Details

Assignee

Reporter

Priority

Development Team

Core: Platform

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created March 12, 2019 at 8:36 AM
Updated January 3, 2020 at 2:30 PM
Resolved March 27, 2019 at 7:57 PM
TestRail: Cases
TestRail: Runs