...

Assumption: Based on the information received from the FSE and Vega teams, their experience in analyzing the issues described and the available statistics, it is reasonable to assume that these issues are caused by the PubSub issues described above.

Consequences of the Push mechanism while Data Import

The existing PubSub is a Push mechanism. Source Record Manager would place large numbers of messages (one per record) into the queue during a large import job. Mod-pubsub would then push these into the callback function provided by mod-inventory. There was no means for mod-inventory to say “enough already”, it would get overloaded and crash. This was discussed with Folijet previously, and no viable solution was found.

The proposed scheme of modules interaction through Direct Kafka

...

• Guaranteed delivery provided by Kafka allows addressing reliability concern
• Improved data consistency since Kafka does not deliver newer messages until older ones are acknowledged
• Better performance by eliminating the overhead of multiple HTTP calls per event dispatch
• Enabling good HA since every new Event Consumer instance connects Kafka within a consumer group, so that the load is distributed evenly
• Improved manageability because of easier investigation capabilities, less data inconsistency, and following fail-fast approach
• the Pull mechanism provided by the Direct Kafka (as implemented in Data Import) - this implementation places the consumer code in mod-inventor,y and it will pull message from Kafka when it has capacity

Limitations, Risks and Assumptions

...

Module name	Owning team	Is it a Producer or Consumer of events?
mod-circulation	Vega	Producer (in a number of flows), Consumer (for events from mod-feesfines)
mod-circulation	Vega
mod-feesfines	Vega	Producer
mod-patron-blocks	Vega	Consumer
mod-audit	Firebird	Consumer
mod-remote-storage	Firebird	Consumer

...