| Table of Contents |
|---|
Kafka Topic Naming Convention
ENV and tenant id setting should be used in topic naming convention. This will separate data of different customers to different Kafka topics. In addition to that it allows a Kafka instance to be shared by multiple environments that have the same tenant id.
...
- Environment name (from ENV environment variable)
- Tenant id (should be the second, because it is convenient to use wildcard in ACL for Kafka users)
- Producer module name ("storage" postfix should be omitted)
RA: I'd propose to explicitly specify full module name here to avoid any ambiguation, e.g. bugfest.fs09000000.mod-inventory-storage.instance (mod- as a mandatory part of any module name can be potentially skipped if all would agree on that)
- Domain entity name in singular form (if it is not domain event, the name of process should be used or just event name)
...
We should also consider cases with many producers and the only consumer, the best example of which is mod-audit - there can be a single topic many modules push their event inpushing their events for audit, and only the mod-audit consumes events from it to log them for further auditingconsuming them. We can follow exactly the same convention as stated above, with the only difference - e.g. bugfest.fs09000000.mod-inventory-storage.audit, or can specify not producer module name but consumer module name instead, e.g.
...
bugfest.fs09000000.mod-audit. (this might be more confusing).
Topic partitioning
In order not to have problems with consistency, that can occur due to race condition in case of concurrent writes to database Kafka topics should have appropriate partition_key. It could be id of the record or some another value, that allow to segregate events between consumer instances.
...
The event payload should have the following structure:
| Code Block |
|---|
{ |
...
"old": {...}, // the entity state before update or delete |
...
"new": {...}, // the entity state update or create |
...
"type": "UPDATE|DELETE|CREATE|DELETE_ALL", // type of the event |
...
"tenant": "diku" // tenant id } |
X-Okapi-Url and X-Okapi-Tenant headers could be set from the request to the Kafka message.
RA: I'm also thinking about versioning of such schema, just to cover potential extension in it and avoid version compatibility issues.
Domain events for delete all APIs
...
- Partition key:
00000000-0000-0000-0000-000000000000 - Event payload:
{
...
"type":
...
"DELETE_ALL",
...
"tenant":
...
"<the
...
tenant
...
name>"
...
}
Open items and questions
Below is the list of other items to be clarified
- Retention policy recommendations - should be recommend cross-platform standard for that?
- Obsolete topic deletion - should exist a mechanism of automated deletion for obsolete topics? How to determine obsolescence?