Naming Convention

Kafka Topic

...

ENV and tenant id setting should be used in topic naming convention. This will separate data of different customers to different Kafka topics. In addition to that it allows a Kafka instance to be shared by multiple environments that have the same tenant id. 

...

Code Block
bugfest.fs09000000.inventory.instance

Example of topic naming for Data Import process

Code Block
ibf3.Default.fs09000000.DI_RAW_RECORDS_CHUNK_PARSED

We should also consider cases with many producers and the only consumer, the best example of which is mod-audit - there can be many modules pushing their events for audit, and only the mod-audit consuming them. We can follow exactly the same convention as stated above, e.g. bugfest.fs09000000.mod-inventory-storage.audit, or can specify not producer module name but consumer module name instead, e.g. bugfest.fs09000000.mod-audit. (this might be more confusing).

Consumer group

While the proposed topic naming convention is based on the producer module name, consumer group naming is based on the consumer module name. This will allow 1+ modules to work with one topic without interfering with each other. Example:

cg.search
cg.innreach
cg.remotestorage

Topic partitioning

In order not to have problems with consistency, that can occur due to race condition in case of concurrent writes to database Kafka topics should have appropriate partition_key. It could be id of the record or some another value, that allow to segregate events between consumer instances.

...