Page Comparison

Technical Council Sub Groups Updates

Previous Notes:

• Presentation (15m-20m) & Q/A (10m): Data sync updates and outcome (TC)
• Looking for solutions that are generally applicable and preserve loose coupling between modules
• Changes in source of truth are to be done atomically, and then notifications sent to consumers
• Details in proposal FOLIO Cross-application Data Sync Proposal
• Proposal is already in use in inventory, changes in mod-inventory-storage  send notifications to be consumed by mod-search 
• Q&A:
  • The message passing via Kafka introduces dependencies between modules. Do we have a mechanism for documenting / declaring the dependencies. Discussed, but intentionally out-of-scope for the group.
  • Are there guidelines on how retention period may change? Trade-offs, retention periods may be different based on use case, no one approach for all cases. Deliberately does not allow for replay.
  • Full-body vs. diff message: there are message sizes, and Kafka works better for small events. 10-100KB cover current known use cases.
  • Regarding outbox pattern, was the Debezium idea evaluated? No, just came up in brainstorming.
    • Debezium is used in LDP/Metadb. Debezium/WAL solutions do generate A LOT of events, which heavily impacts Cloud hosting costs.
  • Are there performance issues with Outbox pattern? Did not do specific testing, existing uses do not have numbers, but also did not state concerns about performance. Outbox will introduce some delay in notifying consumer. Different use cases have their own acceptable delays.
  • Do we need to propose a mechanism for synchronizing data in cases where events get lost.
  • What are next steps? This process pre-dates ADR/RFC process.
    • Action: group will create and ADR and ask for TC approval
    • POC & performance testing? existing work in inventory could be considered POC, but do not use Outbox pattern.
    • Can discuss next steps after TC approval
  • Group was starting from current state of FOLIO and current use cases, was working from a more constrained view than green field.
  • Current reindexing approach undermines some of the is proposal, relies upon consumer knowing the source of truth. 

Today:

• ADR-000004 - Cross application data sync proposal
• Next steps:
  • PoC - Do we need this?  What is the scope?  Timeline?  Who will do this?  etc.