Apache Kafka usage in inventory-storage



OutcomeApache Kafka will be used directly in inventory-storage as a special case 
Created date




This decision has been migrated to the Technical Council's Decision Log as part of a consolidation effort.  See:   DR-000006 - Apache Kafka Usage in Inventory Storage


Apache Kafka should be used directly in inventory-storage for implementation of domain event pattern, which is needed for both remote-storage integration and new search module. This direct usage of Apache Kafka should be considered as a special case, as it was for data-import. Also it is special case, that storage module use more than one data storage. For search module the performance is significant. Kafka partitioning provide system with out-of-the-box mechanism for scaling out modules.

It will be one fault tolerant solution for sending notification to other applications, when data is changed in inventory-storage. This notifications allow us to make business logic segregation and avoid implementation of not related business logic in inventory module:

  • Remote-storage module will start accession process based on these notifications.
  • Search will need to reindex data based on this notifications, so system cannot live with missing even one update notification, because the data would be inconsistent in such case.
  • In future the same notifications may be consumed by other modules (e.g. possible implementation of INN-Reach)

Action items

  • Implement sending of Kafka messages (domain events) from inventory storage, when instance/holding/item is changed. There should separate topics for each type of entities. The id of the changed entity and difference (between old/new) should be provided in the body of the message.