[FOLIO-2353] Add kafka to vagrant boxes Created: 03/Oct/19  Updated: 03/Jun/20  Resolved: 09/Dec/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Oleksii Kuzminov Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: data-import, kafka, platform-backlog, pub-sub
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File pubsub.log    
Issue links:
Relates
relates to FOLIO-2386 Add mod-pubsub to hosted envs and vag... Closed
relates to MODPUBSUB-61 Ensure pub-sub runs correctly on kafk... Closed
Sprint: CP: sprint 77, CP: sprint 78
Story Points: 2
Development Team: Core: Platform

 Description   
  • Kafka should be available inside vagrant box with the default configuration
  • Kafka should startups with vagrant box


 Comments   
Comment by Oleksii Kuzminov [ 03/Oct/19 ]

Wayne SchneiderJakub Skoczen
Hello! According to the implementation of mod-pubsub we need to have a working instance of kafka inside the vagrant box. I don't know who is the right person for it, so, please take a look on this task. Thank you!

Comment by Wayne Schneider [ 03/Oct/19 ]

Oleksii Kuzminov thanks – I've added this to the Platform: Core team backlog.

Comment by Ian Hardy [ 15/Nov/19 ]

Will need to add task to folio-ansible that deploys kafka and zookeeper as docker images. Probably using docker-compose & ansible's docker_service module.

Comment by Ian Hardy [ 20/Nov/19 ]

Oleksii Kuzminov will mod-pubsub be part of paltform-core or complete? Will affect which builds kafka goes into.

Comment by Oleksii Kuzminov [ 20/Nov/19 ]

Ian Hardy I hope yes, PubSub will be a core module. So we need to add kafka for all builds

Comment by Ian Hardy [ 20/Nov/19 ]

Thanks, I'll add it to all the Vagrant builds and wait on the reference environments until pubsub is ready to be integrated.

Comment by Ian Hardy [ 20/Nov/19 ]

Oleksii Kuzminov kafka/zookeeper should be available in the Vagrant images after the next build of those images tonight. Both are on their standard ports 9092 for kafka and 2181 for zk. Let me know if there are any issues.

Comment by Kateryna Senchenko [ 26/Nov/19 ]

Hi Ian Hardy,
I downloaded folio/snapshot-backend-core vagrant box, it has kafka and zookeeper images in it, but kafka is not starting on 'vagrant up'. I couldn't just run the kafka image because it required some additional config params, and anyway we need kafka up and running for pub-sub module to start successfully. Could you please look into it?

Comment by Ian Hardy [ 26/Nov/19 ]

Hi Kateryna Senchenko thanks for testing. I failed to set the proper restart policy for the kafka container, I've updated the config now(https://github.com/folio-org/folio-ansible/pull/295/files). The only vagrant box that has been built since I made that change is "testing". That one is looking good to me now. Lets check snapshot-backend-core again after the next rebuild.

Comment by Kateryna Senchenko [ 27/Nov/19 ]

Hi Ian Hardy, thank you, I checked kafka on folio/snapshot-backend-core today, it is up and running on vagrant box startup. However, when mod-pubsub starts and KafkaAdminClient and Producer are instantiated successfully, I receive the following error trying to create topics:

[kafka-admin-client-thread | adminclient-1] ERROR KafkaThread$1        [104927eqId] Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':
java.lang.IllegalStateException: No entry found for connection 1
        at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:330) ~[mod-pubsub-server-fat.jar:?]
        at org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:134) ~[mod-pubsub-server-fat.jar:?]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:921) ~[mod-pubsub-server-fat.jar:?]
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287) ~[mod-pubsub-server-fat.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:898) ~[mod-pubsub-server-fat.jar:?]
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1113) ~[mod-pubsub-server-fat.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

Looks like some configs should be adjusted, not sure on which side. Do you have any ideas?

Comment by Ian Hardy [ 02/Dec/19 ]

This is a networking problem on my end. I add an alias for the kafka container which isn't being updated when the container restarts (as it does at some point during the production of the vagrant box). I'll find a fix and get back to you.

Comment by Kateryna Senchenko [ 02/Dec/19 ]

Sounds good, thank you Ian Hardy

Comment by Ian Hardy [ 04/Dec/19 ]

Kateryna Senchenko I've updated the kafka config on the vagrant boxes. Could you run "vagrant box update snapshot-core" and try again? Im testing using these instructions (starting at step 3): https://kafka.apache.org/quickstart

Let me know if pubsub is working now.

Comment by Kateryna Senchenko [ 04/Dec/19 ]

Thanks, will do

Comment by Ian Hardy [ 05/Dec/19 ]

Adding some notes from our conversation earlier today:

When you get a chance, let me know what version fo the snapshot-core vagrant box you're running. The following command will tell you if you're logged in to it:

cat /vagrant/.vagrant/machines/snapshot-core/virtualbox/box_meta

Deploying mod-pubsub-1.1.0-SNAPSHOT.34 OK on the snapshot-core box, I don't see any particular kafka related errors, and it appears to be finding kafka and setting up the producer like you mentioned. Not sure how to make pubsub try to create a topic, let me know how i can replicate any errors you're still seeing. I do see the following error on the pubsub log (which I'm assuming is not related to the ability to communicate w/kafka)

15:40:51.131 [vert.x-eventloop-thread-0] INFO  LogUtil              [23693eqId] 10.0.2.15:44666 POST /_/tenant null HTTP_1_1 200 5 7691 tid=diku OK
15:40:51.758 [vert.x-eventloop-thread-0] ERROR SecurityManagerImpl  [24320eqId] Failed to add permission inventory.all for pub-sub user. Received status code 400

The complete log is attached. pubsub.log

Comment by Kateryna Senchenko [ 06/Dec/19 ]

Hi Ian Hardy,
I'm running v1.0.0-20191206.3281 of snapshot-core. My log looks exactly the same (error with permission is unrelated and does not affect connection with kafka in any way).
To trigger topics creation you should send 2 requests (see examples below). But if you just wait for a minute or so, you'll see the connection error log that I attached in previous comments.

1. POST to host:port/pubsub/event-types with body

{
  "eventType": "RECEIVED_SAMPLE_ENTITY",
  "eventTTL": 30
}

2. POST to host:port/pubsub/event-types/declare/publisher

{
  "moduleId": "mod-sample-publisher",
  "eventDescriptors": [
    {
       "eventType": "RECEIVED_SAMPLE_ENTITY",
       "eventTTL": 30
    }]
}
Comment by Kateryna Senchenko [ 09/Dec/19 ]

Hi Ian Hardy,
Looks like this commit https://github.com/folio-org/folio-ansible/pull/306 did the trick - pub-sub works correctly on folio/snapshot-backend-core, which was rebuilt since you merged it. I'm still getting the same error on folio/snapshot-core, but it was built earlier, so I expect it to be resolved after the rebuild. I'll check it tomorrow and will get back to you. Thanks.

Comment by Ian Hardy [ 09/Dec/19 ]

Kateryna Senchenko, Thanks for testing and the details on creating topics with pubsub, that was very helpful.

Generated at Thu Feb 08 23:20:02 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.