Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

While working on the Data Import test using AWS MSK, Kafka messages we getting blocked in Kafka topic and we had to clear out messages from all DI topics. Messages can get stuck in a topic for many reasons such as consumers not having enough resources to consume all topics, the consumer goes down or AWS MSK Broker becomes unstable. Depending on the number of tenants per FOLIO instance, there can be thousands of topics per FOLIO instance. Instead of manually clearing out all messages from each of these topics, we decided to automate it to make this process faster.

...

  • restart all DI modules
  • Wait until log.retention.minutes duration specified in MSK config has passedĀ 

Scripts

To clear out messages from the selected list of Kafka topics, there are 3 steps:

...

Code Block
languagebash
themeMidnight
./kafka-topics.sh --describe --zookeeper <zookeeper-plaintext-connection-url> --topic "imtc.Default.tenant001.DI_COMPLETED"
Topic: imtc.Default.tenant001.DI_COMPLETED PartitionCount: 1 ReplicationFactor: 2 Configs:
Topic: imtc.Default.tenant001.DI_COMPLETED Partition: 0 Leader: 1 Replicas: 1,2 Isr: 2,1

Skip to end of metadata


Scripts for recreation topics with different number of partitions

Step1

Delete topicsĀ 

./delete_topics <file>

Code Block
languagebash
#!/bin/bash -e
val=$1
awk 'BEGIN{print ARGC,ARGV[1]} { system("./kafka-topics.sh --zookeeper z-1.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-2.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181,z-3.kafka280.xwt0mn.c17.kafka.us-east-1.amazonaws.com:2181 --delete --topic "$1"  ") }' "$val" 


example of using


Step 2

Create topics with new parameters (number of partitions - 2)

...