Skip to end of banner
Go to start of banner

PubSub deprecation plan

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Overview

mod-pubsub deprecation will be implemented in two phases:

  1. Transport. mod-pubsub will be replaced with pure Kafka as a transport. This should be implemented in all affected modules at the same time and become a part of a single FOLIO release to ensure seamless transition from one transport to another.
  2. Atomicity. Additionally to changing transport, a pattern like "Transactional outbox/inbox" can be implemented to ensure atomicity of two operations - writing into the DB and sending a message to Kafka.

Skipping Phase 2 DOES NOT mean creating a regression because currently we don't have anything to ensure atomicity of DB writes and PubSub publishing operations. However, we believe that the majority of issues with the PubSub event delivery is being caused by the PubSub itself. This means that completing Phase 1 should greatly improve the reliability of "Automated patron blocks". Phase 2 will improve it even further, but it requires more effort. For example, some message-sending functionality will need to be moved from modules that don't have a DB (like mod-circulation) to modules that do (like mod-circulation-storage).

Affected modules and event types

mod-circulation-storage

Publications

Subscriptions

LOG_RECORD


mod-circulation

Publications

Subscriptions

ITEM_CHECKED_OUT
ITEM_CHECKED_IN
ITEM_DECLARED_LOST
ITEM_AGED_TO_LOST
ITEM_CLAIMED_RETURNED
LOAN_DUE_DATE_CHANGED
LOAN_CLOSED
LOG_RECORD

LOAN_RELATED_FEE_FINE_CLOSED
FEE_FINE_BALANCE_CHANGED

mod-patron-blocks

Publications

Subscriptions

? ITEM_CHECKED_OUT
? ITEM_CHECKED_IN
? ITEM_DECLARED_LOST
? LOAN_DUE_DATE_CHANGED
? FEE_FINE_BALANCE_CHANGED

FEE_FINE_BALANCE_CHANGED
ITEM_CHECKED_OUT
ITEM_CHECKED_IN
ITEM_DECLARED_LOST
ITEM_AGED_TO_LOST
ITEM_CLAIMED_RETURNED
LOAN_DUE_DATE_CHANGED
LOAN_CLOSED

mod-feesfines

Publications

Subscriptions

FEE_FINE_BALANCE_CHANGED
LOAN_RELATED_FEE_FINE_CLOSED
LOG_RECORD


mod-remote-storage

Publications

Subscriptions

LOG_RECORD

? LOG_RECORD

mod-audit

Publications

Subscriptions

? LOG_RECORD

LOG_RECORD


"?" means that, most likely, it was added due to a PubSub (fixed) bug that required a module to subscribe to the same event types that it produces. Because of that, developers used to make publications and subscription lists identical.

Phase 1 implementation plan

For each of the affected modules:

  • Add Kafka default setting to the ModuleDescriptor.
  • Create Kafka topics using KafkaAdminClientService (refer to mod-inventory-storage). Most likely, this should be a part of the _tenant API.
  • Create services that would allow the module to act as a Kafka producer or consumer (or both, whichever is required). Make sure that they are testable, this would make it easy to rewrite tests that depend on the PubSub event matching.
  • Remove all PubSub dependencies (mod-pubsub-client).
  • Remove the MessagingDescriptor file.
  • Remove endpoints that handle PubSub events from the ModuleDescriptor (that's a breaking change!).
  • Remove all the code that registers or unregisters a module as a publisher or a subscriber of PubSub event types. Most likely, this code will be a part of the _tenant API. Replace it with the code that creates Kafka topics using KafkaAdminClientService. 
  • Replace the code that publishes PubSub events with Kafka message producing.
  • Replace the code that consumes PubSub events with Kafka message consuming.

Phase 2 implementation plan

TBD

Technical details and examples

folio-kafka-wrapper will be used for all Kafka interactions.

Example of Kafka default settings in the ModuleDescriptor:      

{ "name": "DB_MAXPOOLSIZE", "value": "5" },
{ "name": "KAFKA_HOST", "value": "kafka" },
{ "name": "KAFKA_PORT", "value": "9092" },
{ "name": "REPLICATION_FACTOR", "value": "1" },
{ "name": "ENV", "value": "folio" }

Example of defining topics and specifying the number of partitions:

public enum CirculationStorageKafkaTopic implements KafkaTopic {
  REQUEST("request", 10),
  LOAN("loan", 10),
  CHECK_IN("check-in", 10);
…
  @Override
  public String moduleName() {
    return "circulation";
  }
…
  @Override
  public int numPartitions() {
    return partitions;
  }
}

Example of topics creation:

.compose(r -> new KafkaAdminClientService(vertxContext.owner())
.createKafkaTopics(CirculationStorageKafkaTopic.values(), tenantId))
  • No labels