Managing Apache Kafka Topics
Instaclustr has several different tools that you can use to manage Topics in Apache Kafka. These include a CLI tool, the Instaclustr API, and the Instaclustr Terraform Provider.
Table of Contents
Kafka Topic Management via CLI
Introduction to Kafka Topic Management
Kafka-topics is a tool that can be used to manage Kafka topics using a connection to a Kafka broker.
The core “actions” supported by kafka-topics include:
- list – list the topics available on the cluster
- create – create a topic
- describe – provide details of one or more topics
- alter – change the properties of one or more topics
- delete – delete one or more topics
Installing kafka-topics
Kafka-topics tool is available as a part of Apache Kafka. To install:
- Download and unpack a copy of Kafka, matching the version that your cluster is running.
- Go to the bin directory of the Kafka installation
> cd kafka-x.x.x-src/bin - Run kafka-topics.sh
> ./kafka-topics.sh
This should display help text for kafka-topics.sh:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Create, delete, describe, or change a topic. Option Description ------ ----------- --alter Alter the number of partitions, replica assignment, and/or configuration for the topic. --bootstrap-server <String: host> REQUIRED: The connection string for the Kafka connection in the form host:port. --config <String: name=value> A topic configuration override for the topic being created or altered.The following is a list of valid configurations: cleanup.policy .... .... |
Using kafka-topics
Once you have downloaded Apache Kafka, you can use kafka-topics to manage topics within your cluster.
To enable kafka-topics to connect to your cluster you will always need to provide two options to the tool:
Option | Description |
‑‑bootstrap-server | provides the details of one of the nodes (brokers) in your Kafka cluster in the form host:port. These details are available on the Instaclustr Console in the Connection Info page. |
‑‑command-config | a path to the properties which contains details of your cluster username and password. These details are also available on the Instaclustr Console in the Connection Info page |
A minimal properties file requires your username and password in the following format:
1 2 3 4 5 |
security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="123abc123abc123abc123abc123abc123abc123abc123abc123abc123abc"; |
(carefully note the placement of the backslashes and semicolon at the end of the lines)
If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:
1 2 3 4 5 6 7 8 |
ssl.truststore.location=truststore.jks ssl.truststore.password=instaclustr ssl.protocol=TLS security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="361d4871ff1a5ef58deaf3b887b489029faee9690e62c549078a1f51f18f755"; |
You can find out more about using certificates with Kafka clients including where to find them here.
All usage of the kafka-topics.sh command will need to specify these details. For example, to list the topics on your cluster, run:
1 |
> ./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --list |
The following sections describe each of the actions supported by the tool. You can get help at any time at the command line by running ./kafka-topics.sh with no arguments.
Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092. To learn more about private networks, please refer to our documentation about Private Network Clusters.
List
The List action provides a listing of the topics that are available on the cluster. To use the List action, include the ‑‑list option
Option | Description |
‑‑list | the list action |
Example:
1 2 3 4 5 |
> ./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --list device-metrics-topic device-log-topic device-activation-topic customer-change-topic |
Create
The “Create” action is used to create a new topic.
For simple topic creation
For simple topic creation: use the Create action. You must provide the following details:
Option | Description |
‑‑create | the create action |
‑‑topic <topic-name> | the name of the topic your wish to create |
‑‑partitions <num-of-partitions> | the number of partitions you want the topic to have |
‑‑replication-factor <replication-factor> | the replication factor for the topic. may not be greater than the number of brokers |
‑‑config <key>=<value> (Optional) | a custom config value to set for this cluster. This can be omitted to use default configuration settings. You can specify more than one config to be changed using multiple ‑‑config options |
Example:
The following example will create a topic called “mytopic” with 9 partitions and a replication factor of 3.
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --create --topic mytopic --partitions 9 --replication-factor 3 |
Kafka will automatically assign partitions to available brokers.
Creating topics with a replica-assignment
If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the ‑‑replica-assignment option.
Describe
The “describe” action is used to provide a detailed description of a topic. You need to provide the following details
Option | Description |
‑‑describe | the creation action |
‑‑topic <TopicName> | the name of the topic to be altered. If this is not specified, all topics will be described. A (java-style) regular expression can also be specified to select multiple topics. Regular expressions must be enclosed in double quotations. |
Only configurations which are not set to defaults will be displayed. This includes values inherited from broker default configs or values which have been set for the topic.
For more details see the description of the output format here:
Example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
./kafka-topics.sh --bootstrap-server 192.168.1.45:9092 --command-config mycluster.properties --describe --topic customer Topic:customer PartitionCount:12 ReplicationFactor:3 Configs:min.insync.replicas=2 Topic: customer Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: customer Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Topic: customer Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: customer Partition: 3 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3 Topic: customer Partition: 4 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: customer Partition: 5 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Topic: customer Partition: 6 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: customer Partition: 7 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Topic: customer Partition: 8 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: customer Partition: 9 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3 Topic: customer Partition: 10 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: customer Partition: 11 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 |
Alter
The “alter” action is used to set custom configurations on the topic. You can configure additional custom configurations on a property using the ‑‑alter action.
To set a configuration on a topic you must specify the following options:
option | description |
‑‑alter | the alter action |
‑‑topic <topic-name> | the name of the topic to be altered. If this is not specified, all topics will be modified. A (java-style) regular expression can also be specified to select multiple topics. Regular expressions must be enclosed in double quotations. |
‑‑config <key>=<value> | a config value to change. You can specify more than one config to be changed using multiple ‑‑config options |
‑‑delete-config <key> | a config value to remove. You may specifiy more than one config to be removed using multiple ‑‑delete-config options |
Examples
The following command will alter the retention.ms configuration on the topic called “mytopic“.
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --alter --topic mytopic --config retention.ms=50000000 |
The following command will alter two values (retention.ms and compression.type) on the topic called “mytopic”
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --alter --topic mytopic --config retention.ms=50000000 --config compression.type=gzip |
The following command will set retention.ms to 50000000 on all topics starting with the string “test”
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --alter --topic "test.*" --config retention.ms=50000000 |
The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker.
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --alter --topic mytopic --config min.insync.replicas=2 |
Deleting a configuration
To remove a custom configuration from a topic, use ‑‑alter combined with the ‑‑delete-config option. The ‑‑delete-config option should specify the name of the configuration to remove. Removing a custom configuration will cause the value to revert to the default value.
Example:
The following command will remove the retention.ms configuration, reverting it to the default value.
1 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --alter --topic mytopic --delete-config retention.ms |
Delete (Topic Deletion)
The Delete action is used to a delete a topic. To delete a topic you must specify the following options:
option | description |
‑‑delete | the delete action |
‑‑topic <topic-name> | the name of the topic to be altered.
A (java-style) regular expression can also be specified to select multiple topics. Regular expressions must be enclosed in double quotations. |
Examples
The following command will delete the topic called “mytopic“.
1 2 3 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --delete --topic mytopic Topic mytopic is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. |
The following command will delete the all topics starting with “dev”.
1 2 3 4 5 |
./kafka-topics.sh --bootstrap-server 192.168.1.1:9092 --command-config mycluster.properties --delete --topic "dev.*" Topic dev-customer is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. Topic dev-device is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. |
Kafka Topic Management via Instaclustr API
Instaclustr provides Topic Management for Kafka clusters as part of the Cluster Management API to help you with managing topics. The API supports the following Topic Management functions:
For information on the usage of the Instaclustr’s Cluster Management API please refer to the API documentation.
Kafka Topic Management via Instaclustr Terraform Provider
Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at
1 |
https://github.com/instaclustr/terraform-provider-instaclustr |
The documentation for the ACL resource is located in docs/resources/kafka_topic.md and the ACL data source is located in docs/data-sources/kafka_topic.md .