What Is Kafka Management?

Kafka management involves the oversight, configuration, and maintenance of Apache Kafka clusters to ensure they run efficiently and reliably. It includes tasks like setting up clusters, managing topics and brokers, monitoring system health, optimizing performance, and enforcing security policies. Effective Kafka management ensures that data streams can be processed and transmitted smoothly across distributed systems, meeting performance, scalability, and fault tolerance requirements.

Given Kafka’s role as a distributed event streaming platform, proper management is critical for high availability, data consistency, and efficient resource utilization. Administrators often rely on tools and best practices to handle these tasks and troubleshoot issues as they arise.

Key Areas in Kafka Management

Cluster Setup and Configuration

Cluster setup and configuration is the foundational aspect of Kafka management. It involves designing the architecture of the Kafka cluster, setting up nodes (brokers), and configuring essential components like ZooKeeper (or its replacement, KRaft, in newer Kafka versions). The configuration should be tailored to ensure fault tolerance, scalability, and resource efficiency.

Key configuration parameters include replication factors, partition counts, and log retention policies. Administrators also need to configure network settings, such as listeners and advertised listeners, which allow brokers to communicate internally and externally.

Topic Management

Topic management in Kafka involves creating, deleting, and configuring topics, which are the fundamental data structures where messages are stored and processed. Each topic can be divided into partitions, allowing parallel processing of messages and high throughput.

Administrators manage topics by defining the number of partitions and replication factors based on use cases, ensuring a balance between fault tolerance and performance. They also monitor topic-level metrics like partition size and message lag to optimize performance. Policies like log retention and compaction can be configured at the topic level to control how long data is kept or how it’s compacted to save space.

Broker Management

Broker management focuses on maintaining the individual Kafka brokers, which handle data storage, processing, and transmission. This involves ensuring that brokers are running correctly, monitoring resource usage (CPU, memory, disk), and detecting anomalies like broker failures or unbalanced data distribution.

Effective broker management includes balancing partitions across brokers to avoid overloading specific nodes, upgrading Kafka versions without downtime, and ensuring that brokers are correctly connected to ZooKeeper or KRaft for leader election and metadata storage. Regular maintenance tasks, such as disk cleanup and log segment management, are also essential for smooth broker operation.

Performance Tuning

Performance tuning in Kafka management aims to optimize throughput, latency, and resource efficiency. Key areas for tuning include adjusting broker configurations, optimizing partition and replication settings, and fine-tuning producer and consumer performance.

Administrators focus on optimizing parameters like batch sizes, request timeouts, and memory buffers to increase data throughput. Network settings, such as compression algorithms and socket buffers, can be fine-tuned for optimal performance under varying loads. It’s also essential to monitor Kafka metrics using tools like Prometheus and Grafana to identify bottlenecks and take corrective action.

Security Management

Security management ensures that Kafka clusters are protected against unauthorized access, data breaches, and malicious attacks. This involves configuring authentication, authorization, and encryption mechanisms.

Administrators implement security protocols such as SSL/TLS for encrypted communication between clients and brokers and configure Kerberos or SASL for client authentication. Kafka’s access control lists (ACLs) are used to restrict client permissions, ensuring that only authorized users can produce or consume messages from specific topics. Security audits and regular updates are also critical to address vulnerabilities and maintain a secure Kafka environment.

3 Tools to Make Kafka Topic Management Easier

Instaclustr provides a managed Kafka service that helps clients reduce the workload required for Kafka management. We provide several tools you can use to manage Topics in Apache Kafka. These include a CLI tool, the Instaclustr API, and the Instaclustr Terraform Provider.

1. Kafka Topic Management via CLI

Kafka comes with its own topic management tools that can be downloaded from here. For our offering of Kafka 2.6.1 and newer, you can use Kafka-topics.sh tool that they have provided. However, for our older offerings, you will need to use ic-Kafka-topics.

Ic-Kafka-topics is a tool developed by Instaclustr that can be used to manage Kafka topics using a connection to a Kafka broker. Ic-Kafka-topics is based on the standard Kafka-topics tool, but unlike Kafka-topics, it does not require a zookeeper connection to work.

The core “actions” supported by ic-Kafka-topics include:

  • list – list the topics available on the cluster
  • create – create a topic
  • describe – provide details of one or more topics
  • alter – change the properties of one or more topics
  • delete – delete one or more topics

Installing ic-kafka-topics

Ic-Kafka-topics is available as a tarfile, downloadable from the Instaclustr Console cluster connection page.  To install:

  • Download the tarfile from the Instaclustr console
  • Unpack the tarfile:

  • Go to the bin directory of the Kafka installation

  • Run ic-kafka-topics.sh

This should display help text for ic-Kafka-topic.sh:

Using ic-kafka-topics

Once you have downloaded and installed ic-Kafka-topics, you can use it to manage topics with your cluster.

To enable ic-Kafka-topics to connect to your cluster you will always need to provide two options to the tool:

Option Description
‑‑bootstrap-server provides the details of one of the nodes (brokers) in your Kafka cluster in the form host:port. These details are available on the Instaclustr console in the cluster connections page.
‑‑properties-file a path to a file which contains details of your cluster username and password. These details are also available from the customer support site.

A minimal properties file requires your userid and password in the following format:

(carefully note the placement of the backslashes and semicolon at the end of the lines)

If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:

You can find out more about using certificates with Kafka clients including where to find them here.

All usage of the ic-Kafka-topics command will specify these details.  For example, to list the topics on your cluster, run:

The following sections describe each of the actions supported by the tool.  You can get help at any time at the command line by running ic-Kafka-topics.sh with no arguments.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Listing topics

The List action provides a listing of the topics that are available on the cluster. To use the List action, include the ‑‑list option.

Example:

Simple topic creation

For simple topic creation: use the Create action.  You must provide the following details:

Option Description
‑‑create the create action
‑‑topic <topic-name> the name of the topic your wish to create
‑‑partitions <num-of-partitions> the number of partitions you want the topic to have
‑‑replication-factor <replication-factor> the replication factor for the topic.  may not be greater than the number of brokers
‑‑config <key>=<value> (Optional) a custom config value to set for this cluster. This can be omitted to use default configuration settings. You can specify more than one config to be changed using multiple ‑‑config options

Example:

The following example will create a topic called mytopic with 9 partitions and a replication factor of 3.

Kafka will automatically assign partitions to available brokers.

Creating topics with a replica-assignment

If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the ‑‑replica-assignment option.

Describing topics

The “describe” action is used to provide a detailed description of a topic.  You need to provide the following details:

Option Description
‑‑describe the creation action
‑‑topic <TopicName> the name of the topic to be altered.  If this is not specified, all topics will be described.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Only configurations which are not set to defaults will be displayed.  This includes values inherited from broker default configs or values which have been set for the topic.

For more details see the description of the output format for the standard Kafka-topics tool here:

Example:

The alter action

The “alter” action is used to set custom configurations on the topic.  You can configure additional custom configurations on a property using the ‑‑alter action.

To set a configuration on a topic you must specify the following options:

option description
‑‑alter the alter action
‑‑topic <topic-name> the name of the topic to be altered.  If this is not specified, all topics will be modified. A (java-style) regular expression can also be specified to select multiple topics. Regular expressions must be enclosed in double quotations.
‑‑config <key>=<value> a config value to change.  You can specify more than one config to be changed using multiple ‑‑config options
‑‑delete-config <key> a config value to remove. You may specifiy more than one config to be removed using multiple ‑‑delete-config options

Examples:

The following command will alter the retention.ms configuration on the topic called mytopic:

The following command will alter two values (retention.ms and compression.type) on the topic called mytopic:

The following command will set retention.ms to 50000000 on all topics starting with the string test:

The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker:

Deleting a configuration

To remove a custom configuration from a topic, use ‑‑alter combined with the ‑‑delete-config option. The ‑‑delete-config option should specify the name of the configuration to remove. Removing a custom configuration will cause the value to revert to the default value.

Example:

The following command will remove the retention.ms configuration, reverting it to the default value:

Delete (Topic Deletion)

The Delete action is used to a delete a topic.  To delete a topic you must specify the following options:

option description
‑‑delete the delete action
‑‑topic <topic-name> the name of the topic to be altered.
A (java-style) regular expression can also be specified to select multiple topics. Regular expressions must be enclosed in double quotations.

Examples

The following command will delete the topic called mytopic:

The following command will delete the all topics starting with dev:

2. Kafka Topic Management via Instaclustr API

Instaclustr provides Topic Management API for Kafka clusters to help you with managing topics. The Topic Management API supports the following functions:

  • List Topics
  • Create a Topic
  • Delete a Topic
  • Get Topic information (number of partitions and replication factor)
  • Get Topic configuration
  • Modify Topic configuration

These examples show how to use the features of the Instaclustr Topic Management API.

For each endpoint listed below, all requests must include basic authentication details:

List Topics

To retrieve a list of topics currently available in the Kafka cluster, make a GET request to:

On a successful request, the API should respond with a 200 status code and a JSON containing the list of topics. E.g.:

Create a Topic

To create a new topic to the Kafka cluster make a POST request to:

A request body should be included, specifying the topic name, number of partitions, and replication factor. E.g.:

Number of partitions and the replication factor should be non-zero integers. The replication factor is limited by the number of available brokers in the clusters. Be aware of creating a topic with too many partitions, as it can slow down the cluster.

On successful topic creation, the API should respond with a 201 status code and a short message. E.g.:

Once this message is received, the topic should be immediately available in the Kafka cluster.

Delete a Topic

To delete an existing topic from the Kafka cluster make a DELETE request to:

No request body is required. The topic to be deleted is specified in the URL.

When the topic is successfully deleted, the API should respond with a 200 status code and a short message. E.g.:

Get Topic Information

To retrieve the topic information of a particular topic available in the Kafka cluster, make a GET request to:

The API should respond with a 200 status code and a JSON containing the information of the requested topic. E.g.:

Get Topic Configuration

To retrieve the topic configuration of a particular topic available in the Kafka cluster, make a GET request to:

On a successful request, the API should respond with a 200 status code and a JSON containing the configuration of the requested topic. E.g.:

Modify a Topic Configuration

To modify the configuration of a topic in the Kafka cluster, make a PUT request to:

A request body should be included, specifying the configuration that you want to change. E.g.:

We do not need to provide all of the available keys in the request body, only the keys of interest are required in the body.

For available configuration keys and values, consult https://kafka.apache.org/23/documentation.html#topicconfigs.

Alternatively, available configuration keys can also be inferred from the output of the get topic configuration endpoint. Note if you are using Kafka version that is newer than 2.3: Currently, we can’t change the message.format.version to a version that is newer than 2.3. Please use the Kafka CLI tools (kafka-configs.sh) to modify this configuration.

The API should respond with a 200 status code and a short message. E.g.:

3. Kafka Topic Management via Instaclustr Terraform Provider

Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at https://github.com/instaclustr/terraform-provider-instaclustr.

The documentation for the ACL resource is located in docs/resources/kafka_topic.md and the ACL data source is located in docs/data-sources/kafka_topic.md.

 

Instaclustr for Apache Kafka: Simplifying Managed Kafka Services

Instaclustr for Apache Kafka is revolutionizing the way organizations handle high-volume, real-time data streams. As a leader in managed open-source data technologies, our comprehensive platform makes deploying, managing, and scaling Apache Kafka clusters simple – so you can focus on crafting robust, real-time data streaming applications.

Managing Kafka clusters can be a complex task requiring expertise in infrastructure provisioning, configuration, and monitoring. Instaclustr eliminates these challenges by providing a fully managed service, taking care of the underlying infrastructure. Our solution includes automated provisioning and scaling, a user-friendly interface, and API for managing Kafka topics, partitions, and consumer groups.

Instaclustr for Apache Kafka is not just about simplicity. It’s about securing your data, too. Our platform offers advanced security features including encryption at rest and in transit, authentication, and authorization mechanisms. Plus, with our multi-region replication capabilities, your data streaming will continue uninterrupted even in the face of infrastructure failures. With comprehensive monitoring and alerting capabilities, you’ll get real-time metrics and proactive alerts to quickly identify and address any issues, ensuring optimal performance and minimizing downtime.

Ready to simplify your Apache Kafka management and security? Learn more about our Instaclustr for Apache Kafka solution and let’s start streamlining your data streaming processes today.

For more information: