Overview

While working on the Data Import test using AWS MSK, Kafka messages we getting blocked in Kafka topic and we had to clear out messages from all DI topics. Messages can get stuck in a topic for many reasons such as consumers not having enough resources to consume all topics, the consumer goes down or AWS MSK Broker becomes unstable. Depending on the number of tenants per FOLIO instance, there can be thousands of topics per FOLIO instance. Instead of manually clearing out all messages from each of these topics, we decided to automate it to make this process faster.

...

Before running the scripts, try the below following options first

• restart all DI modules
• Wait until log.retention.minutes duration specified in MSK config has passedpassed 

Scripts

To clear out messages from the selected list of Kafka topics, there are 3 steps:

...