Managing Apache Kafka Topics

Instaclustr has several different tools that you can use to manage Topics in Apache Kafka. These include a CLI tool, the Instaclustr API, and the Instaclustr Terraform Provider.

Table of Contents

Kafka Topic Management via CLI

Introduction to Kafka Topic Management

Kafka-topics is a tool that can be used to manage Kafka topics using a connection to a Kafka broker.

The core “actions” supported by kafka-topics include:

  • list – list the topics available on the cluster
  • create – create a topic
  • describe – provide details of one or more topics
  • alter – change the properties of one or more topics
  • delete – delete one or more topics

Installing kafka-topics

Kafka-topics tool is available as a part of Apache Kafka.  To install:

  1. Download and unpack a copy of Kafka, matching the version that your cluster is running.
  2. Go to the bin directory of the Kafka installation
    > cd kafka-x.x.x-src/bin
  3. Run kafka-topics.sh
    > ./kafka-topics.sh

This should display help text for kafka-topics.sh:

Using kafka-topics

Once you have downloaded Apache Kafka, you can use kafka-topics to manage topics within your cluster.

To enable kafka-topics to connect to your cluster you will always need to provide two options to the tool:

Option Description
‑‑bootstrap-server provides the details of one of the nodes (brokers) in your Kafka cluster in the form host:port.  These details are available on the Instaclustr Console in the Connection Info page.
‑‑command-config a path to the properties which contains details of your cluster username and password.  These details are also available on the Instaclustr Console in the Connection Info page

A minimal properties file requires your username and password in the following format:

(carefully note the placement of the backslashes and semicolon at the end of the lines)

If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:

You can find out more about using certificates with Kafka clients including where to find them here.

All usage of the kafka-topics.sh command will need to specify these details.  For example, to list the topics on your cluster, run:

The following sections describe each of the actions supported by the tool.  You can get help at any time at the command line by running ./kafka-topics.sh with no arguments.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092. To learn more about private networks, please refer to our documentation about Private Network Clusters.

List

The List action provides a listing of the topics that are available on the cluster.  To use the List action, include the ‑‑list option

Option Description
‑‑list the list action

Example:

Create

The “Create” action is used to create a new topic.

For simple topic creation

For simple topic creation: use the Create action.  You must provide the following details:

Option Description
‑‑create the create action
‑‑topic <topic-name> the name of the topic your wish to create
‑‑partitions <num-of-partitions> the number of partitions you want the topic to have
‑‑replication-factor <replication-factor> the replication factor for the topic.  may not be greater than the number of brokers
‑‑config <key>=<value> (Optional) a custom config value to set for this cluster. This can be omitted to use default configuration settings. You can specify more than one config to be changed using multiple ‑‑config options

Example:

The following example will create a topic called “mytopic” with 9 partitions and a replication factor of 3.

Kafka will automatically assign partitions to available brokers.

Creating topics with a replica-assignment

If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the ‑‑replica-assignment option.

Describe

The “describe” action is used to provide a detailed description of a topic.  You need to provide the following details

Option Description
‑‑describe the creation action
‑‑topic <TopicName> the name of the topic to be altered.  If this is not specified, all topics will be described.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Only configurations which are not set to defaults will be displayed.  This includes values inherited from broker default configs or values which have been set for the topic.

For more details see the description of the output format here:

Example:

Alter

The “alter” action is used to set custom configurations on the topic.  You can configure additional custom configurations on a property using the ‑‑alter action.

To set a configuration on a topic you must specify the following options:

option description
‑‑alter the alter action
‑‑topic <topic-name> the name of the topic to be altered.  If this is not specified, all topics will be modified.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.
‑‑config <key>=<value> a config value to change.  You can specify more than one config to be changed using multiple ‑‑config options
‑‑delete-config <key> a config value to remove.  You may specifiy more than one config to be removed using multiple ‑‑delete-config options

Examples

The following command will alter the retention.ms configuration on the topic called “mytopic“.

The following command will alter two values (retention.ms and compression.type) on the topic called “mytopic

The following command will set retention.ms to 50000000 on all topics starting with the string “test

The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker.

Deleting a configuration

To remove a custom configuration from a topic, use ‑‑alter combined with the ‑‑delete-config option.  The ‑‑delete-config option should specify the name of the configuration to remove.  Removing a custom configuration will cause the value to revert to the default value.

Example:

The following command will remove the retention.ms configuration, reverting it to the default value.

Delete (Topic Deletion)

The Delete action is used to a delete a topic.  To delete a topic you must specify the following options:

option description
‑‑delete the delete action
‑‑topic <topic-name> the name of the topic to be altered.

A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Examples

The following command will delete the topic called “mytopic“.

The following command will delete the all topics starting with “dev”.

Kafka Topic Management via Instaclustr API

Instaclustr provides Topic Management for Kafka clusters as part of the Cluster Management API to help you with managing topics. The API supports the following Topic Management functions:

For information on the usage of the Instaclustr’s Cluster Management API please refer to the API documentation.

Kafka Topic Management via Instaclustr Terraform Provider

Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at

The documentation for the ACL resource is located in docs/resources/kafka_topic.md and the ACL data source is located in docs/data-sources/kafka_topic.md .

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes