[UXPROD-3764] Switching from PubSub to Direct Kafka approach in Circulation Created: 01/Aug/22  Updated: 08/Feb/24

Status: Draft
Project: UX Product
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: TBD
Reporter: Denis Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
is defined by MODPUBSUB-244 Spike: test Open
is defined by CIRC-1897 intermittently on restart mod-circula... Closed
is defined by MODPATBLK-160 Automated maximum fee/fine blocks not... Closed
Requires
requires CIRC-1570 [Spike] Estimate mod-pubsub deprecation Closed
requires CIRC-1829 [Spike] Investigate Transactional out... Closed
Development Team: None
PO Rank: 0
Rank: Cornell (Full Sum 2021): R2

 Description   

Current situation or problem:
There are known PubSub issues based on production experience with PubSub in mod-circulation, mod-feesfines, and mod-patron-blocks, as well as results from mod-pubsub performance testing (it should be noted that this testing was conducted some time ago, and apparently there is no more recent data). PubSub issues are notoriously time-consuming and hard to investigate. Mostly because they are usually invisible to the end user. When an event or a series of events can't reach the intended subscriber, libraries rarely notice this immediately, but rather when data inconsistencies caused by undelivered events manifest themselves elsewhere.

Benefits:

  • Guaranteed delivery provided by Kafka allows addressing reliability concern
  • Improved data consistency since Kafka does not deliver newer messages until older ones are acknowledged
  • Better performance by eliminating the overhead of multiple HTTP calls per event dispatch
  • Enabling good HA since every new Event Consumer instance connects Kafka within a consumer group, so that the load is distributed evenly
  • Improved manageability because of easier investigation capabilities, less data inconsistency, and following fail-fast approach
  • the Pull mechanism provided by the Direct Kafka (as implemented in Data Import) - this implementation places the consumer code in mod-inventory and it will pull message from Kafka when it has capacity

In scope
Refactoring of the following modules:

  1. mod-circulation
  2. mod-feesfines
  3. mod-patron-blocks
  4. mod-audit
  5. mod-remote-storage

Out of scope

Use case(s)

Proposed solution/stories

Required efforts can be divided into two groups:
#switching to folio-kafka-wrapper, reusing its capabilities and independently implementing missing functionality (in terms of creating and configuring topics, for example); a small spike-story will help to better understand the size of this group
#transferring listed above modules to the event-approach (a simpler and more understandable activity, because the essence of the process is the same everywhere)

Links to additional info

Questions


Generated at Fri Feb 09 00:34:37 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.