|
Current situation or problem:
There are known PubSub issues based on production experience with PubSub in mod-circulation, mod-feesfines, and mod-patron-blocks, as well as results from mod-pubsub performance testing (it should be noted that this testing was conducted some time ago, and apparently there is no more recent data). PubSub issues are notoriously time-consuming and hard to investigate. Mostly because they are usually invisible to the end user. When an event or a series of events can't reach the intended subscriber, libraries rarely notice this immediately, but rather when data inconsistencies caused by undelivered events manifest themselves elsewhere.
Benefits:
- Guaranteed delivery provided by Kafka allows addressing reliability concern
- Improved data consistency since Kafka does not deliver newer messages until older ones are acknowledged
- Better performance by eliminating the overhead of multiple HTTP calls per event dispatch
- Enabling good HA since every new Event Consumer instance connects Kafka within a consumer group, so that the load is distributed evenly
- Improved manageability because of easier investigation capabilities, less data inconsistency, and following fail-fast approach
- the Pull mechanism provided by the Direct Kafka (as implemented in Data Import) - this implementation places the consumer code in mod-inventory and it will pull message from Kafka when it has capacity
In scope
Refactoring of the following modules:
- mod-circulation
- mod-feesfines
- mod-patron-blocks
- mod-audit
- mod-remote-storage
Out of scope
Use case(s)
Proposed solution/stories
Required efforts can be divided into two groups:
#switching to folio-kafka-wrapper, reusing its capabilities and independently implementing missing functionality (in terms of creating and configuring topics, for example); a small spike-story will help to better understand the size of this group
#transferring listed above modules to the event-approach (a simpler and more understandable activity, because the essence of the process is the same everywhere)
Links to additional info
Questions
|