DR-000004 - Cross application data sync proposal

Owned by Raman Auramau
Last updated: May 06, 2024 by Jenn Colt
Submitted Date

 

Approved Date

 

StatusACCEPTED
ImpactMEDIUM

RFC - n/a

Other Related Resources

Contributors

Approvers

Background/Context

The cross-app data synchronization working group was formed to analyze known data consistency issues between individual parts (modules, applications) of FOLIO and propose a solution to address all (or part) of these issues. The working group developed and documented the proposed approach (see links above) and also presented it to PC and TC. This ADR was created at the request of TC and is intended to review and decide on the recommendation of the proposed approach for use on the FOLIO platform.

Briefly, the proposed approach is to apply the domain event pattern, generate events when data changes, publish such events to the notification channel (Kafka), and process such events on the consumer side.

Assumptions

  • It is possible to identify a Single Source of Truth (i.e. an application owning domain data) in a particular use case
  • Retention policy will always be long enough for messages to be consumed by a receiving module
  • Initial data synchronization is performed during the data migration as a part of a feature release
  • The hosting provider is responsible for providing a reliable and stable Kafka service, including data backup to avoid losing events

More on Assumptions, Limitations, Challenges section and Assumptions and Challenges slides of the TC Updates & Outcome presentation.

Constraints

  • Based on an existing tech stack
  • A unified approach for addressing data consistency issues (specifically for Copies of data to be updated, Notifications on deletion, Notification on data update issues) for the FOLIO platform

Decision

  • FOLIO will use Kafka and domain events as documented above to address cross-application data consistency issues of the indicated categories.

  • The use of the Outbox and Inbox patterns in the context of the solution described above is out of scope of this decision.  The Technical Council may choose to revisit this at a later date.

Implications

  • Pros
    • Without adding explicit dependencies between modules
    • Moves FOLIO towards using asynchronous, event driven solutions
  • Cons
    • Under certain conditions, eventual consistency may not meet the expectations or requirements of specific use cases
    • Potential issues with the consistent state after failures