Motivation
While working on the Data Import test using AWS MSK, Kafka messages we getting blocked in Kafka topic and we had to clear out messages from all DI topics. Messages can get stuck in a topic for many reasons such as consumers not having enough resources to consume all topics, the consumer goes down or AWS MSK Broker becomes unstable. Depending on the number of tenants per FOLIO instance, there can be thousands of topics per FOLIO instance. Instead of manually clearing out all messages from each of these topics, we decided to automate it to make this process faster.
To clear out messages from Kafka topic, there are 3 steps:
- Get the list of topics to clear out messages
- Download the latest Kafka release and extract it - https://kafka.apache.org/quickstart
- Navigate to the bin directory kafka_2.12-2.7.0/bin/ - we will be using kafka-topics.sh and kafka-configs.sh scripts out of the box
- Get list of topics and write to file topics.txt
./kafka-topics.sh --describe --zookeeper <zookeeper-plaintext-connection-url> --topic "imtc.Default.fs00001020.*" | grep Configs | awk '{printf "%s\n", $2}' > topics.txt
- Alter topic by adding custom config by setting retention time to 1 second or less
- Revert topic by deleting the custom config