Kafka recreating topics

Owned by Roman_Fedynyshyn
Last updated: Jun 26, 2024
3 min read

 

Overview

The purpose of this document is to highlight the way PTF is recreating Kafka topics using JMeter and Kafdrop.

Summary

  1. Have a Kafdrop pointed to current cluster and another Kafdrop on new cluster

  2. Manually copy all topic names, partition count, replication factor to CSV attached to JMeter

  3. Aim JMeter on new Kafdrop and run it. It will recreate all the topics using Kafdrops' API.

Detailed steps

1. Deploy Kafdrop

Using http://10.23.33.46:8080/job/fse-deploy-kafdrop/ you can deploy Kafdrop (Web app that giving comfortable UI and API for Kafka) if you don’t have one yet.

pic.1

image-20240626-073246.png

pic.2

image-20240626-073426.png

2. Extract Kafdrop data

From pic.2 view, with the simple selecting and copying from UI you can get the list of Kafka topics with number of the partitions. (take into account that data copied from Kafdrop may require additional manual edit). But mainly here we’re interested in two coma separated values topic name (like ompt.Default.fs00001137.DI_COMPLETED) and partition

In result you should have something like this saved in topics.csv

ompt.Default.fs00001137.DI_COMPLETED,2,100%,0,No ompt.Default.fs00001137.DI_ERROR,2,100%,0,No ompt.Default.fs00001137.DI_INITIALIZATION_STARTED,2,100%,0,No ompt.Default.fs00001137.DI_INVENTORY_HOLDING_CREATED,2,100%,0,No

Edit /jmeter-supported-data/topics.csv file with data like above

2. Run JMeter script to recreate Kafka topics

*JMeter is simple API automation tool used by PTF for various purposes (e.g. API automation, load generation, etc.)

pic.3

Here replace [kafdropHostorIP] with valid Kafdrop hostname or IP, replace [kafdropPort] with valid port to Kafdrop. (replicationFactor is defaulted to “2“, it’s changeable according to number of Kafka brokers).

pic.4

This JMeter script will loot through .csv file with data and will stop as soon as records ends.

pic.5

JMeter will do POST ${kafdropHostorIP}:${kafdropPort}/topic with parameters topic, replicationFactor and partitions.

Conclusions

This approach may be recreated with another API automation tool (postman, even curl). FSE may want to adapt it for their own purposes.

This is the simple way to recreate lot of topics on new Kafka cluster using Kafdrop API + JMeter.

PTF has not fully automated it yet so there’s some manual steps (like getting a topics name and partition count).