Topic Management

Menu

Introduction

Ic-kafka-topics is a tool developed by Instaclustr that can be used to manage Kafka topics using a connection to a Kafka broker.  Ic-kafka-topics is based on the standard kakfa-topics tool, but unlike kafka-topics, it does not require a zookeeper connection to work.

The core “actions” supported by ic-kafka-topics include:

  • list – list the topics available on the cluster
  • create – create a topic
  • describe – provide details of one or more topics
  • alter – change the properties of one or more topics
  • delete – delete one or more topics

Installing ic-kafka-topics

Ic-kafka-topics is available as a tarfile, downloadable from the Instaclustr Console cluster connection page.  To install:

  1. Download the tarfile from the Instaclustr console
  2. Unpack the tarfile:
    tar -xf kafka_2.11-1.1.0.tgz
  3. Go to the bin directory of the kafka installation
    > cd kafka_2.11-1.1.0/bin
  4. Run ic-kafka-topics.sh
    > ./ic-kafka-topics.sh

This should display help text for ic-kafka-topic.sh:

Using ic-kafka-topics

Once you have downloaded and installed ic-kafka-topics, you can use it to manage topics with your cluster.

To enable ic-kafka-topics to connect to your cluster you will always need to provide two options to the tool:

OptionDescription
--bootstrap-serverprovides the details of one of the nodes (brokers) in your Kafka cluster in the form host:port.  These details are available on the Instaclustr console in the cluster connections page.
--properties-filea path to a file which contains details of your cluster username and password.  These details are also available from the customer support site.

A minimal properties file requires your userid and password in the following format:

(carefully note the placement of the backslashes and semicolon at the end of the lines)

If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:

You can find out more about using certificates with Kafka clients including where to find them here.

All usage of the ic-kafka-topics command will specify these details.  For example, to list the topics on your cluster, run:

The following sections describe each of the actions supported by the tool.  You can get help at any time at the command line by running ic-kafka-topics.sh with no arguments.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

List

The List action provides a listing of the topics that are available on the cluster.  To use the List action, include the --list option

OptionDescription
--listthe list action

Example:

Create

The Create action is used to create a new topic.

For simple topic creation

For simple topic creation: use the Create action.  You must provide the following details:

OptionDescription
--createthe create action
--topic <topic-name>the name of the topic your wish to create
--partitions <num-of-partitions>the number of partitions you want the topic to have
--replication-factor <replication-factor>the replication factor for the topic.  may not be greater than the number of brokers

Example:

The following example will create a topic called “mytopic” with 9 partitions and replication factor of 3.

Kafka will automatically assign partitions to available brokers.

Creating topics with a replica-assignment

If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the --replica-assignment option.

Creating topics with a custom configuration

It is not possible to create topics with a custom configuration in a single step. If you require custom topic configurations you will need to --create the topic and then use --alter to apply the custom configurations.

Describe

The “describe” action is used to provide a detailed description of a topic.  You need to provide the following details

OptionDescription
--describethe creation action
--topic <TopicName>the name of the topic to be altered.  If this is not specified, all topics will be described.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Only configurations which are not set to defaults will be displayed.  This includes values which inherited from broker default configs, or values which have been set for the topic.

For more details see the description of the output format for the standard kafka-topics tool here:

Example:

Alter

The “alter” action is used to set custom configurations on the topic.  You can configure additional custom configurations on a property using the --alter action.

To set a configuration on a topic you must specify the following options:

optiondescription
--alterthe alter action
--topic <topic-name>the name of the topic to be altered.  If this is not specified, all topics will be modified.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.
--config <key>=<value>a config value to change.  You can specify more than one config to be changed using multiple --config options
--delete-config <key>a config value to remove.  You may specifiy more than one config to be removed using multiple --delete-config options

Examples

The following command will alter the retention.ms configuration on the topic called “mytopic“.

The following command will alter two values (retention.ms and compression.type) on the topic called “mytopic

The following command will set retention.ms to 50000000 on all topics starting with the string “test

The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker.

Deleting a configuration

To remove a custom configuration from a topic, use --alter combined with the --delete-config option.  The --delete-config option should specify the name of the configuration to remove.  Removing a custom configuration will cause the value to revert to the default value.

Example:

The following command will remove the retention.ms configuration, reverting it to the default value.

Delete (Topic Deletion)

The Delete action is used to a delete a topic.  To delete a topic you must specify the following options:

optiondescription
--deletethe delete action
--topic <topic-name>the name of the topic to be altered.

A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Examples

The following command will delete the topic called “mytopic“.

The following command will delete the all topics starting with “dev”.

FREE TRIAL

Spin up a cluster in less
than 5 minutes.
(No credit card required)

Sign Up Now
Close

Site by Swell Design Group