Kafka Topics Explained: How They Work, Config & Best Practices

What Is an Apache Kafka Topic?

In Apache Kafka, topics are the fundamental, named logical channels used to organize and store streaming data. Producers write events to topics, and consumers read from them. Topics function as immutable, append-only logs, meaning historical data is preserved rather than deleted upon reading.

Command line operations:

You can manage topics using Kafka’s built-in CLI tools (like kafka-topics.sh or confluent kafka topic):

Create: kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

List: kafka-topics.sh --list --bootstrap-server localhost:9092

Describe: kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Delete: kafka-topics.sh --delete --topic my-topic --bootstrap-server localhost:9092

How Kafka Topics Work

Step 1: Producers Write Messages to Topics

In Kafka, a producer is any application or service that sends data to a Kafka topic. Producers publish messages to topics by sending records, which contain both a key and a value, to the Kafka cluster. The producer can specify the target topic and, optionally, the partition within that topic. This process is asynchronous, allowing producers to continue sending data without waiting for the messages to be processed, which supports high throughput and low latency.

Producers can balance load and ensure ordering by specifying how messages are distributed across partitions within a topic. If a key is provided, Kafka uses it to determine the target partition, which ensures that all messages with the same key are written to the same partition and maintain order. Without a key, Kafka distributes messages round-robin across available partitions. 

Step 2: Consumers Read From Topics

Consumers are applications or services that read messages from Kafka topics. Each consumer subscribes to one or more topics and pulls data from the partitions assigned to it. Kafka consumers can be grouped into consumer groups, where each group coordinates to ensure that each partition is read by only one consumer within the group, allowing for parallel processing and scalability.

Kafka tracks the offset, or position, of each consumer in the topic, so consumers can pick up where they left off if they disconnect or restart. This offset management allows for reliable message processing and the ability to replay messages if needed. Consumers can read messages in real time as they arrive or process historical data by specifying the offset from which to start reading, making Kafka suitable for a variety of streaming and batch workloads.

Step 3: Kafka Topics Are Stored as Logs

Internally, Kafka stores each topic as a log, which is an append-only, ordered sequence of records. Each partition within a topic is implemented as its own log file, where new messages are appended to the end. This design enables fast writes and efficient sequential reads, which are critical for high-throughput data streaming applications.

Kafka’s log-based storage model also underpins features like message retention and replay. Messages are not deleted once consumed; instead, they remain in the log for a configurable retention period or until storage limits are reached. This approach allows consumers to re-read data, enables fault tolerance, and supports use cases like auditing or data backfilling. The log structure makes Kafka a durable and reliable platform for managing large volumes of streaming data.

Kafka Topics vs. Partitions

A Kafka topic is a logical grouping of messages, while partitions are the physical units of storage and parallelism within a topic. Each topic consists of one or more partitions, and each partition is an ordered, immutable sequence of messages. 

Partitions enable Kafka to scale horizontally by distributing data and load across multiple brokers in the cluster. Partitions also provide the foundation for parallelism in both data ingestion and consumption. By splitting a topic into multiple partitions, Kafka allows multiple producers and consumers to write and read data concurrently. This partition-based architecture increases throughput and enables Kafka to ensure ordering guarantees within each partition, while still allowing high scalability and fault tolerance through replication.

Kafka Topic Configuration

Common Kafka Topic Settings

Kafka topics can be configured at creation time or updated later with topic-level configuration overrides. If no per-topic configuration is provided, Kafka uses the corresponding broker-level default. These settings control how the topic stores records, retains data, handles replication, and enforces durability guarantees.

Common topic settings include:

• The number of partitions
• Replication factor
• Retention settings
• Cleanup policy
• Compression type
• Maximum message size
• Minimum in-sync replicas

Partitions determine how topic data is distributed and processed in parallel, while the replication factor defines how many copies of each partition are maintained across brokers for availability and fault tolerance. Settings such as retention.ms and retention.bytes control how long or how much data Kafka keeps, while cleanup.policy determines whether Kafka deletes old log segments, compacts records by key, or applies both behaviors.

Durability-related settings are also important for production topics. For example, min.insync.replicas specifies the minimum number of in-sync replicas that must acknowledge a write when producers use acks=all. When combined with an appropriate replication factor, this helps prevent successful writes from being acknowledged unless enough replicas have persisted the data. Other settings, such as compression.type and max.message.bytes, influence storage efficiency and the size of records that can be written to a topic.

Topic Retention

Topic retention defines how long Kafka keeps records in a topic before they are eligible for deletion. Unlike traditional messaging systems, Kafka does not delete messages immediately after consumers read them. Instead, records remain available for the configured retention period or until configured storage limits are reached, allowing consumers to reprocess historical data, recover after downtime, or support auditing and backfilling use cases.

Kafka supports time-based and size-based retention: 

• The retention.ms setting controls the maximum amount of time Kafka retains log data before discarding old log segments when the topic uses the delete cleanup policy. 
• The default retention.ms value is seven days, while setting it to -1 removes the time limit. 
• The retention.bytes setting controls the maximum size a partition log can grow before Kafka discards old segments to free space. 
• Because retention.bytes is applied at the partition level, the total possible topic storage depends on the number of partitions.

Retention is performed on log segments rather than individual records. Kafka rolls log data into segment files, and old segments become eligible for deletion once the configured retention time or size conditions are met. This means retention settings are central to balancing storage cost, replay requirements, and consumer recovery windows.

Log Compaction

Log compaction is a Kafka cleanup policy that retains the latest value for each record key instead of simply deleting records based only on age or log size. When cleanup.policy is set to compact, Kafka’s log cleaner removes older records with the same key while preserving the most recent record for that key. This makes compacted topics useful for datasets where the latest state matters, such as:

• User profiles
• Account balances
• Configuration data
• Changelog topics

In a compacted topic, Kafka does not guarantee that every historical update will be retained forever. Instead, it ensures that the latest known value for each key is retained after compaction has run. This allows consumers to rebuild the current state of a keyed dataset by reading the topic from the beginning, without needing every intermediate update.

Kafka also supports combining delete and compact policies by setting cleanup.policy to delete,compact. In this configuration, Kafka can both compact records by key and delete old segments according to retention settings. Additional compaction-related settings, such as min.cleanable.dirty.ratio, min.compaction.lag.ms, max.compaction.lag.ms, and delete.retention.ms, influence when records become eligible for compaction and how long tombstone records are retained for deletes in compacted topics.

How to Create a Kafka Topic

Kafka topics can be created, listed, described, and deleted using Kafka’s command-line tools. The most common tool for topic administration is the kafka-topics.sh utility, which is included with the Apache Kafka distribution. This utility allows administrators and developers to manage topics in a Kafka cluster by connecting to one or more brokers through the --bootstrap-server option.

1. To create a topic, the user specifies the Kafka bootstrap server, the topic name, the number of partitions, and the replication factor. For example, the following command creates a topic named my-topic with three partitions and a replication factor of one:
   
   bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --create \ --topic application-events \ --partitions 3 \ --replication-factor 1

   1 2 3 4 5 6

   bin/kafka-topics.sh \   --bootstrap-server localhost:9092 \   --create \   --topic application-events \   --partitions 3 \   --replication-factor 1

2. To list all topics in a Kafka cluster, use the --list option:
   
   bin/kafka-topics.sh \ --bootstrap-server kafka-broker:9092 \ --list

   1 2 3

   bin/kafka-topics.sh \   --bootstrap-server kafka-broker:9092 \   --list

3. To describe a topic and view details such as its partitions, replicas, and in-sync replicas, use the --describe option:
   
   bin/kafka-topics.sh \ --bootstrap-server kafka-broker:9092 \ --describe \ --topic audit-events

   1 2 3 4

   bin/kafka-topics.sh \ --bootstrap-server kafka-broker:9092 \ --describe \ --topic audit-events

4. To delete a topic, use the --delete option with the topic name:
   
   bin/kafka-topics.sh \ --bootstrap-server kafka-broker:9092 \ --delete \ --topic audit-events

   1 2 3 4

   bin/kafka-topics.sh \   --bootstrap-server kafka-broker:9092 \   --delete \   --topic audit-events

 The --bootstrap-server option tells the command which Kafka broker to connect to, while --topic defines the topic name. The --partitions option determines how many partitions the topic will have, which affects parallelism and throughput. The --replication-factor option determines how many copies of each partition Kafka stores across brokers, improving availability and fault tolerance.

In production environments, topics are typically created deliberately with carefully chosen partition counts, replication factors, retention settings, and cleanup policies. While Kafka can be configured to automatically create topics when clients reference a topic that does not exist, explicitly creating topics is usually preferred because it gives teams more control over durability, scalability, and storage behavior.

Kafka Topics Best Practices

Here are some important practices to consider when using topics in Kafka.

1. Naming Conventions

Consistent and descriptive naming conventions help organize Kafka topics and prevent confusion in multi-team or complex environments. Names should reflect the purpose and content of the topic, such as user-signups, order-events, or inventory-updates. Avoid using generic or ambiguous names, as they can make it difficult to trace data flows or troubleshoot issues.

It’s common to use naming patterns that include environment, team, or version information (e.g., prod.user-signups.v2). This approach:

• Simplifies topic management
• Enables automation
• Reduces the risk of naming conflicts

Documenting naming conventions and enforcing them through automation or review processes further improves clarity and maintainability across the organization.

2. One Event Type Per Topic

Storing only one event type per topic is a recommended practice in Kafka architecture. This design:

• Simplifies schema management
• Reduces the risk of consumer errors
• Makes it easier to evolve event structures over time 

Mixing multiple event types in a single topic can complicate data parsing and increase the likelihood of processing mistakes. Dedicated topics for each event type also support independent scaling, retention, and security policies. For example, a topic handling sensitive user data can have different access controls and retention settings than a topic for system logs. Keeping event types separated helps maintain data quality and simplifies downstream processing.

3. Plan Partition Counts

Careful planning of partition counts is essential for achieving optimal performance and scalability in Kafka. The number of partitions determines how much parallelism is available for both producers and consumers. Too few partitions can create bottlenecks, while too many can increase management complexity and resource usage.

Partition count should reflect:

• Anticipated throughput
• Consumer group size
• Future growth

It’s often easier to increase partitions later than to decrease them, but changing partition counts can affect ordering guarantees and consumer assignments. Administrators should assess workload requirements, hardware capabilities, and scaling plans when deciding on partition counts.

4. Configure Replication for Fault Tolerance

Replication is critical for ensuring data durability and availability in Kafka. Each partition can be replicated across multiple brokers, with one leader and several followers. If a broker fails, another replica can take over, preventing data loss and minimizing downtime. The replication factor should be set based on:

• Desired fault tolerance
• Available resources

A replication factor of at least three is common in production environments to tolerate multiple failures. However, higher replication increases storage and network usage. Administrators should balance fault tolerance with infrastructure costs and monitor replica health to ensure that partitions remain fully replicated and available at all times.

5. Set Retention Based on the Topic’s Use Case

Retention settings should be tailored to how the topic is used rather than relying on default values. Some topics only need a few hours or days of data because records are processed immediately and rarely replayed. Other topics may require weeks or months of retention to support analytics, auditing, disaster recovery, or the ability to rebuild downstream systems from historical events. Retention policies should align with:

• Business requirements
• Storage capacity
• Recovery objectives

When configuring retention, consider both retention.ms and retention.bytes. Long retention periods increase storage requirements, especially for high-volume topics, while short retention periods may prevent consumers from recovering after extended outages. For state-oriented topics, log compaction may be more appropriate than long-term retention because it preserves the latest value for each key while reducing storage consumption. Regularly reviewing retention settings helps ensure they continue to match application needs as workloads evolve.

6. Monitor Consumer Lag and Partition Health

Consumer lag measures the difference between the latest offset in a partition and the offset a consumer group has processed. Persistent or growing lag can indicate that consumers are unable to keep up with incoming data because of insufficient processing capacity, application errors, slow downstream systems, or uneven partition distribution. Monitoring lag helps teams detect bottlenecks early and prevent delays in data processing.

Partition health should also be monitored to ensure topic reliability and performance. Key indicators include:

• Under-replicated partitions
• Offline partitions
• Leader election events
• Replica synchronization status

A healthy topic should have all replicas in sync and no partitions without an available leader. Monitoring these metrics allows administrators to identify broker failures, replication issues, or capacity constraints before they affect application availability or data durability.

Related content: Read our guide to 13 Kafka best practices to run Kafka like the pros.

Manage Kafka Topics at Scale with Instaclustr for Apache Kafka

Designing topics, tuning partitions, and enforcing retention and replication policies becomes far more demanding once Kafka runs in production. Instaclustr for Apache Kafka is a fully managed, hosted Kafka service that removes the operational burden of running Kafka in-house, delivering production-ready, fully supported clusters in the cloud or on-prem so your teams can focus on designing topics and building applications rather than maintaining infrastructure.

Key capabilities of Instaclustr for Apache Kafka:

• Rapid cluster provisioning: Spin up production-ready Kafka clusters in minutes, using the Instaclustr Console, API, or Terraform provider, without needing deep Kafka expertise.
• Dynamic scaling: Adapt to fluctuating workloads with dynamic scaling options that handle bursting demand across on-prem, cloud, and hybrid environments.
• Proactive monitoring: Use built-in monitoring tools to keep topics, partitions, and brokers operating at peak efficiency and to surface issues before they affect availability.
• Enterprise-grade support: Get 24/7 support from a team of seasoned Kafka professionals so your streaming environment is always covered.
• Security and compliance: Rely on built-in security features and compliance with industry standards, including SOC 2, ISO 27001, ISO 27018, PCI-DSS, and HIPAA.
• Industry-leading SLAs: Enterprise deployments with dedicated Apache ZooKeeper or KRaft nodes are backed by a 99.999% SLA, with transparent, predictable pricing and no hidden costs.

Ready to run Kafka without the operational overhead? Learn more about Instaclustr’s managed Apache Kafka service and start your free trial.