[FOLIO-3245] Support multiple Kafka brokers connection Created: 20/Jul/21  Updated: 22/Jul/21

Status: Open
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: TBD
Reporter: Hongwei Ji Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint:
Development Team: None

 Description   

Production Kafka environment typically have multiple brokers. Clients usually learn about these from the bootstrap server they first connect to.

Current FOLIO modules only support KAFKA_HOST and KAFKA_PORT env vars. That means only one Kafka address can be specified.

It would be nice to introduce KAFKA_BOOTSTRAP_SERVERS where a list of bootstrap servers can be specified, for example: b-1.domain.com:9092,b-2.domain.com:9092,b-3.domain.com:9092 That the client can then use to discover the full set of brokers.



 Comments   
Comment by Marc Johnson [ 21/Jul/21 ]

Hongwei Ji Thank you for raising this.

It would be nice to introduce KAFKA_URL where a list of brokers can be specified, for example: b-1.domain.com:9092,b-2.domain.com:9092,b-3.domain.com:9092

I think it's a useful change.

I think we need a place to document the expectations for all modules related to Kafka integration and to communicate changes to all of the maintainers of modules that already use Kafka.

Is this environment variable intended to replace the others?

I'm not very familiar with Kafka terminology. Some quick research suggests that the clients should be provided with a set of bootstrap servers which the client then uses to determine the brokers (even though these may be the same). Is this list intended to be the bootstrap servers?

Comment by Hongwei Ji [ 21/Jul/21 ]

Marc Johnson thanks for commenting on this one

Is this environment variable intended to replace the others

Technically it can/should replace the old ones. For backwards compatibility, maybe keep the old ones for a while?

Is this list intended to be the bootstrap servers?

You are exactly right!

Comment by Marc Johnson [ 21/Jul/21 ]

Hongwei Ji

Technically it can/should replace the old ones. For backwards compatibility, maybe keep the old ones for a while?

I think we likely need the environment to provide both for a while as modules transition. I think modules supporting both at the same time needs more consideration, e.g. which take precedence if both are supplied?

You are exactly right!

I suggest we name the environment variable to explicitly reflect that, maybe KAFKA_BOOTSTRAP_SERVERS?

Comment by Hongwei Ji [ 21/Jul/21 ]

Marc Johnson

I think modules supporting both at the same time needs more consideration, e.g. which take precedence if both are supplied?

Agreed. I would assume the new one will take precedence. We need to document that to make it clear though.

I suggest we name the environment variable to explicitly reflect that, maybe KAFKA_BOOTSTRAP_SERVERS?

{{KAFKA_BOOTSTRAP_SERVERS }}sounds good to me. Please feel free to update the description to use the new name. Thanks.

Generated at Thu Feb 08 23:26:41 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.