Overview
While working on the Data Import test using AWS MSK, Kafka messages we getting blocked in Kafka topic and we had to clear out messages from all DI topics. Messages can get stuck in a topic for many reasons such as consumers not having enough resources to consume all topics, the consumer goes down or AWS MSK Broker becomes unstable. Depending on the number of tenants per FOLIO instance, there can be thousands of topics per FOLIO instance. Instead of manually clearing out all messages from each of these topics, we decided to automate it to make this process faster.
...
- restart all DI modules
- Wait until log.retention.minutes duration specified in MSK config has passedĀ
Scripts
To clear out messages from the selected list of Kafka topics, there are 3 steps:
...
Code Block | ||||
---|---|---|---|---|
| ||||
./kafka-topics.sh --describe --zookeeper <zookeeper-plaintext-connection-url> --topic "imtc.Default.tenant001.DI_COMPLETED" Topic: imtc.Default.tenant001.DI_COMPLETED PartitionCount: 1 ReplicationFactor: 2 Configs: Topic: imtc.Default.tenant001.DI_COMPLETED Partition: 0 Leader: 1 Replicas: 1,2 Isr: 2,1 |
Scripts for recreation topics with different number of partitions
Step1
Delete topicsĀ
./delete_topics <file>
Code Block | ||
---|---|---|
| ||
#!/bin/bash -e val=$1 awk 'BEGIN{print ARGC,ARGV[1]} { system("./kafka-topics.sh --zookeeper z-1.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-2.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-3.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181 --delete --topic "$1" ") }' "$val" |
example of using
Step 2
Create topics with new parameters (number of partitions - 2)
...