[FOLIO-2353] Add kafka to vagrant boxes Created: 03/Oct/19 Updated: 03/Jun/20 Resolved: 09/Dec/19 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Oleksii Kuzminov | Assignee: | Ian Hardy |
| Resolution: | Done | Votes: | 0 |
| Labels: | data-import, kafka, platform-backlog, pub-sub | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue links: |
|
||||||||||||
| Sprint: | CP: sprint 77, CP: sprint 78 | ||||||||||||
| Story Points: | 2 | ||||||||||||
| Development Team: | Core: Platform | ||||||||||||
| Description |
|
| Comments |
| Comment by Oleksii Kuzminov [ 03/Oct/19 ] |
|
Wayne SchneiderJakub Skoczen |
| Comment by Wayne Schneider [ 03/Oct/19 ] |
|
Oleksii Kuzminov thanks – I've added this to the Platform: Core team backlog. |
| Comment by Ian Hardy [ 15/Nov/19 ] |
|
Will need to add task to folio-ansible that deploys kafka and zookeeper as docker images. Probably using docker-compose & ansible's docker_service module. |
| Comment by Ian Hardy [ 20/Nov/19 ] |
|
Oleksii Kuzminov will mod-pubsub be part of paltform-core or complete? Will affect which builds kafka goes into. |
| Comment by Oleksii Kuzminov [ 20/Nov/19 ] |
|
Ian Hardy I hope yes, PubSub will be a core module. So we need to add kafka for all builds |
| Comment by Ian Hardy [ 20/Nov/19 ] |
|
Thanks, I'll add it to all the Vagrant builds and wait on the reference environments until pubsub is ready to be integrated. |
| Comment by Ian Hardy [ 20/Nov/19 ] |
|
Oleksii Kuzminov kafka/zookeeper should be available in the Vagrant images after the next build of those images tonight. Both are on their standard ports 9092 for kafka and 2181 for zk. Let me know if there are any issues. |
| Comment by Kateryna Senchenko [ 26/Nov/19 ] |
|
Hi Ian Hardy, |
| Comment by Ian Hardy [ 26/Nov/19 ] |
|
Hi Kateryna Senchenko thanks for testing. I failed to set the proper restart policy for the kafka container, I've updated the config now(https://github.com/folio-org/folio-ansible/pull/295/files). The only vagrant box that has been built since I made that change is "testing". That one is looking good to me now. Lets check snapshot-backend-core again after the next rebuild. |
| Comment by Kateryna Senchenko [ 27/Nov/19 ] |
|
Hi Ian Hardy, thank you, I checked kafka on folio/snapshot-backend-core today, it is up and running on vagrant box startup. However, when mod-pubsub starts and KafkaAdminClient and Producer are instantiated successfully, I receive the following error trying to create topics: [kafka-admin-client-thread | adminclient-1] ERROR KafkaThread$1 [104927eqId] Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': java.lang.IllegalStateException: No entry found for connection 1 at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:330) ~[mod-pubsub-server-fat.jar:?] at org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:134) ~[mod-pubsub-server-fat.jar:?] at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:921) ~[mod-pubsub-server-fat.jar:?] at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287) ~[mod-pubsub-server-fat.jar:?] at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:898) ~[mod-pubsub-server-fat.jar:?] at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1113) ~[mod-pubsub-server-fat.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Looks like some configs should be adjusted, not sure on which side. Do you have any ideas? |
| Comment by Ian Hardy [ 02/Dec/19 ] |
|
This is a networking problem on my end. I add an alias for the kafka container which isn't being updated when the container restarts (as it does at some point during the production of the vagrant box). I'll find a fix and get back to you. |
| Comment by Kateryna Senchenko [ 02/Dec/19 ] |
|
Sounds good, thank you Ian Hardy |
| Comment by Ian Hardy [ 04/Dec/19 ] |
|
Kateryna Senchenko I've updated the kafka config on the vagrant boxes. Could you run "vagrant box update snapshot-core" and try again? Im testing using these instructions (starting at step 3): https://kafka.apache.org/quickstart Let me know if pubsub is working now. |
| Comment by Kateryna Senchenko [ 04/Dec/19 ] |
|
Thanks, will do |
| Comment by Ian Hardy [ 05/Dec/19 ] |
|
Adding some notes from our conversation earlier today: When you get a chance, let me know what version fo the snapshot-core vagrant box you're running. The following command will tell you if you're logged in to it: cat /vagrant/.vagrant/machines/snapshot-core/virtualbox/box_meta Deploying mod-pubsub-1.1.0-SNAPSHOT.34 OK on the snapshot-core box, I don't see any particular kafka related errors, and it appears to be finding kafka and setting up the producer like you mentioned. Not sure how to make pubsub try to create a topic, let me know how i can replicate any errors you're still seeing. I do see the following error on the pubsub log (which I'm assuming is not related to the ability to communicate w/kafka) 15:40:51.131 [vert.x-eventloop-thread-0] INFO LogUtil [23693eqId] 10.0.2.15:44666 POST /_/tenant null HTTP_1_1 200 5 7691 tid=diku OK 15:40:51.758 [vert.x-eventloop-thread-0] ERROR SecurityManagerImpl [24320eqId] Failed to add permission inventory.all for pub-sub user. Received status code 400 The complete log is attached. pubsub.log |
| Comment by Kateryna Senchenko [ 06/Dec/19 ] |
|
Hi Ian Hardy, 1. POST to host:port/pubsub/event-types with body
{
"eventType": "RECEIVED_SAMPLE_ENTITY",
"eventTTL": 30
}
2. POST to host:port/pubsub/event-types/declare/publisher
{
"moduleId": "mod-sample-publisher",
"eventDescriptors": [
{
"eventType": "RECEIVED_SAMPLE_ENTITY",
"eventTTL": 30
}]
}
|
| Comment by Kateryna Senchenko [ 09/Dec/19 ] |
|
Hi Ian Hardy, |
| Comment by Ian Hardy [ 09/Dec/19 ] |
|
Kateryna Senchenko, Thanks for testing and the details on creating topics with pubsub, that was very helpful. |