DR-000001 - Use Kafka as the distributed event streaming platform

RFC - N/A

Other Documents

• Search Proposal
• Pub-Sub RFC
• Data Import Proposal
• Subsequent Documentation
  • Kafka development guidelines
  • /wiki/spaces/~Vasily/pages/3571867

Contributors

• Marc Johnson 

Approvers

• Craig McNally 
• Zak Burke 
• Marc Johnson 
• Tod Olson 
• Jeremy Huff 
• Chulin Meng 
• Raman Auramau 
• Steffen Köhler 

Decision

Kafka is an official infrastructure component of the FOLIO platform

Background/Context

As this decision is being documented after the fact, this section describes the discovered history of the decision rather than the background to the decision point.

Kafka was first introduced to support a publish subscribe mechanism in FOLIO. At this point, Kafka was intended to be private to this mechanism to allow for substitution with other tooling without broad impact.

Kafka became general use when Data Import was changed to use it directly in order to improve reliability and a centralized search mechanism was introduced to provide a better inventory search experience.

None of these decisions outline why Kafka was chosen over other tool options.

Assumptions

Unknown

Constraints

Unknown

Implications

• Pros
  • Alternative integration approach to HTTP APIs
• Cons
  • Additional infrastructure component required for hosting providers to operate
  • Dependency on third party tooling (much like PostgreSQL and Elastic Search)
  • Increased operational complexity e.g. monitoring