[FOLIO-3245] Support multiple Kafka brokers connection Created: 20/Jul/21 Updated: 22/Jul/21 |
|
| Status: | Open |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | New Feature | Priority: | TBD |
| Reporter: | Hongwei Ji | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Sprint: | |
| Development Team: | None |
| Description |
|
Production Kafka environment typically have multiple brokers. Clients usually learn about these from the bootstrap server they first connect to. Current FOLIO modules only support KAFKA_HOST and KAFKA_PORT env vars. That means only one Kafka address can be specified. It would be nice to introduce KAFKA_BOOTSTRAP_SERVERS where a list of bootstrap servers can be specified, for example: b-1.domain.com:9092,b-2.domain.com:9092,b-3.domain.com:9092 That the client can then use to discover the full set of brokers. |
| Comments |
| Comment by Marc Johnson [ 21/Jul/21 ] |
|
Hongwei Ji Thank you for raising this.
I think it's a useful change. I think we need a place to document the expectations for all modules related to Kafka integration and to communicate changes to all of the maintainers of modules that already use Kafka. Is this environment variable intended to replace the others? I'm not very familiar with Kafka terminology. Some quick research suggests that the clients should be provided with a set of bootstrap servers which the client then uses to determine the brokers (even though these may be the same). Is this list intended to be the bootstrap servers? |
| Comment by Hongwei Ji [ 21/Jul/21 ] |
|
Marc Johnson thanks for commenting on this one
Technically it can/should replace the old ones. For backwards compatibility, maybe keep the old ones for a while?
You are exactly right! |
| Comment by Marc Johnson [ 21/Jul/21 ] |
I think we likely need the environment to provide both for a while as modules transition. I think modules supporting both at the same time needs more consideration, e.g. which take precedence if both are supplied?
I suggest we name the environment variable to explicitly reflect that, maybe KAFKA_BOOTSTRAP_SERVERS? |
| Comment by Hongwei Ji [ 21/Jul/21 ] |
Agreed. I would assume the new one will take precedence. We need to document that to make it clear though.
{{KAFKA_BOOTSTRAP_SERVERS }}sounds good to me. Please feel free to update the description to use the new name. Thanks. |