Instaclustr has several different tools that you can use to manage Topics in Apache Kafka. These include a CLI tool, the Instaclustr API, and the Instaclustr Terraform Provider.

Table of Contents

Kafka Topic Management via CLI

Introduction to Kafka Topic Management

Kafka-topics is a tool that can be used to manage Kafka topics using a connection to a Kafka broker.

The core “actions” supported by kafka-topics include:

• list – list the topics available on the cluster
• create – create a topic
• describe – provide details of one or more topics
• alter – change the properties of one or more topics
• delete – delete one or more topics

Installing kafka-topics

Kafka-topics tool is available as a part of Apache Kafka.  To install:

1. Download and unpack a copy of Kafka, matching the version that your cluster is running.
2. Go to the bin directory of the Kafka installation
   > cd kafka-x.x.x-src/bin
3. Run kafka-topics.sh
   > ./kafka-topics.sh

This should display help text for kafka-topics.sh:

Using kafka-topics

Once you have downloaded Apache Kafka, you can use kafka-topics to manage topics within your cluster.

To enable kafka-topics to connect to your cluster you will always need to provide two options to the tool:

A minimal properties file requires your username and password in the following format:

(carefully note the placement of the backslashes and semicolon at the end of the lines)

If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:

You can find out more about using certificates with Kafka clients including where to find them here.

All usage of the kafka-topics.sh command will need to specify these details.  For example, to list the topics on your cluster, run:

The following sections describe each of the actions supported by the tool.  You can get help at any time at the command line by running ./kafka-topics.sh with no arguments.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092. To learn more about private networks, please refer to our documentation about Private Network Clusters.

List

The List action provides a listing of the topics that are available on the cluster.  To use the List action, include the ‑‑list option

Example:

Create

The “Create” action is used to create a new topic.

For simple topic creation

For simple topic creation: use the Create action.  You must provide the following details:

Example:

The following example will create a topic called “mytopic” with 9 partitions and a replication factor of 3.

Kafka will automatically assign partitions to available brokers.

Creating topics with a replica-assignment

If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the ‑‑replica-assignment option.

Describe

The “describe” action is used to provide a detailed description of a topic.  You need to provide the following details

Only configurations which are not set to defaults will be displayed.  This includes values inherited from broker default configs or values which have been set for the topic.

For more details see the description of the output format here:

Example:

Alter

The “alter” action is used to set custom configurations on the topic.  You can configure additional custom configurations on a property using the ‑‑alter action.

To set a configuration on a topic you must specify the following options:

Examples

The following command will alter the retention.ms configuration on the topic called “mytopic“.

The following command will alter two values (retention.ms and compression.type) on the topic called “mytopic”

The following command will set retention.ms to 50000000 on all topics starting with the string “test”

The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker.

Deleting a configuration

To remove a custom configuration from a topic, use ‑‑alter combined with the ‑‑delete-config option.  The ‑‑delete-config option should specify the name of the configuration to remove.  Removing a custom configuration will cause the value to revert to the default value.

Example:

The following command will remove the retention.ms configuration, reverting it to the default value.

Delete (Topic Deletion)

The Delete action is used to a delete a topic.  To delete a topic you must specify the following options:

Examples

The following command will delete the topic called “mytopic“.

The following command will delete the all topics starting with “dev”.

Kafka Topic Management via Instaclustr API

Instaclustr provides Topic Management for Kafka clusters as part of the Cluster Management API to help you with managing topics. The API supports the following Topic Management functions:

• List Topics
• Create a Topic
• Delete a Topic
• Get Topic details
• Update Topic configuration

For information on the usage of the Instaclustr’s Cluster Management API please refer to the API documentation.

Kafka Topic Management via Instaclustr Terraform Provider

Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at

The documentation for the ACL resource is located in docs/resources/kafka_topic.md and the ACL data source is located in docs/data-sources/kafka_topic.md .