Naming Convention

Kafka Topic

...

ENV and tenant id setting should be used in topic naming convention. This will separate data of different customers to different Kafka topics. In addition to that it allows a Kafka instance to be shared by multiple environments that have the same tenant id. 

...

Code Block
bugfest.fs09000000.inventory.instance

Example of topic naming for Data Import process

Code Block
ibf3.Default.fs09000000.DI_RAW_RECORDS_CHUNK_PARSED

We should also consider cases with many producers and the only consumer, the best example of which is mod-audit - there can be many modules pushing their events for audit, and only the mod-audit consuming them. We can follow exactly the same convention as stated above, e.g. bugfest.fs09000000.mod-inventory-storage.audit, or can specify not producer module name but consumer module name instead, e.g. bugfest.fs09000000.mod-audit. (this might be more confusing).

Consumer group

While the proposed topic naming convention is based on the producer module name, consumer group naming is based on the consumer module name. This will allow 1+ modules to work with one topic without interfering with each other. Example:

cg.search
cg.innreach
cg.remotestorage

Topic partitioning

In order not to have problems with consistency, that can occur due to race condition in case of concurrent writes to database Kafka topics should have appropriate partition_key. It could be id of the record or some another value, that allow to segregate events between consumer instances.

...

X-Okapi-Url and X-Okapi-Tenant headers could be set from the request to the Kafka message.

RA: I'm also thinking about versioning of such schema, just to cover potential extension in it and avoid version compatibility issues.

Domain events for delete all APIs

...

{ "type": "DELETE_ALL", "tenant": "<the tenant name>" }

Other items and aspects

Event schema versioning

Q: Does it make sense to include explicit version field into domain event schemas?

A: Potentially, this would make sense. Though now the basic schema is extremely straightforward so that there's a very small chance for changes. Also, adding new fields later should not be a breaking change.

Retention policy

Q: Retention policy recommendations - should particular cross-platform standard for that be recommended?

...