Overview
The purpose of this document is to highlight the way PTF is recreating Kafka topics using JMeter and Kafdrop.
Summary
Have a Kafdrop pointed to current cluster and another Kafdrop on new cluster
Manually copy all topic names, partition count, replication factor to CSV attached to JMeter
Aim JMeter on new Kafdrop and run it. It will recreate all the topics using Kafdrops' API.
Detailed steps
1. Deploy Kafdrop
Using http://10.23.33.46:8080/job/fse-deploy-kafdrop/ you can deploy Kafdrop (Web app that giving comfortable UI and API for Kafka) if you don’t have one yet.
pic.1
pic.2
2. Extract Kafdrop data
From pic.2 view, with the simple selecting and copying from UI you can get the list of Kafka topics with number of the partitions. (take into account that data copied from Kafdrop may require additional manual edit). But mainly here we’re interested in two coma separated values topic name (like ompt.Default.fs00001137.DI_COMPLETED) and partition
In result you should have something like this saved in topics.csv
ompt.Default.fs00001137.DI_COMPLETED,2,100%,0,No ompt.Default.fs00001137.DI_ERROR,2,100%,0,No ompt.Default.fs00001137.DI_INITIALIZATION_STARTED,2,100%,0,No ompt.Default.fs00001137.DI_INVENTORY_HOLDING_CREATED,2,100%,0,No
Edit /jmeter-supported-data/topics.csv file with data like above
2. Run JMeter script to recreate Kafka topics
*JMeter is simple API automation tool used by PTF for various purposes (e.g. API automation, load generation, etc.)
pic.3
Here replace [kafdropHostorIP] with valid Kafdrop hostname or IP, replace [kafdropPort] with valid port to Kafdrop. (replicationFactor is defaulted to “2“, it’s changeable according to number of Kafka brokers).
pic.4
This JMeter script will loot through .csv file with data and will stop as soon as records ends.
pic.5
JMeter will do POST ${kafdropHostorIP}:${kafdropPort}/topic with parameters topic, replicationFactor and partitions.