Apache Kafka is a distributed streaming platform that offers four key APIs: the Producer API, Consumer API, Streams API, and Connector API with features such as redundant storage of massive data volumes and a message bus capable of throughput reaching millions of messages each second. These capabilities and more make Kafka a solution that’s tailor-made for processing streaming data from real-time applications.
Table of Contents
- What is Apache Kafka®?
- Apache Kafka Architecture—Component Overview
- Kafka Cluster Architecture
- Basic Concepts
- Kafka Architecture—Component Relationship Examples
- Resourcing Consumers and Producers
1. What Is Apache Kafka?
Kafka is essentially a commit log with a simplistic data structure. The Kafka Producer API, Consumer API, Streams API, and Connect API can be used to manage the platform, and the Kafka cluster architecture is made up of Brokers, Consumers, Producers, and ZooKeeper.
Apache Kafka’s Architecture
Despite its name’s suggestion of Kafkaesque complexity, Apache Kafka’s architecture actually delivers an easier to understand approach to application messaging than many of the alternatives. Kafka is essentially a commit log with a very simplistic data structure. It just happens to be an exceptionally fault-tolerant and horizontally scalable one.
Apache Kafka’s Commit Log
The Kafka commit log provides a persistent ordered data structure. Records cannot be directly deleted or modified, only appended onto the log. The order of items in Kafka logs is guaranteed. The Kafka cluster creates and updates a partitioned commit log for each topic that exists. All messages sent to the same partition are stored in the order that they arrive. Because of this, the sequence of the records within this commit log structure is ordered and immutable. Kafka also assigns each record a unique sequential ID known as an “offset,” which is used to retrieve data.
Kafka addresses common issues with distributed systems by providing set ordering and deterministic processing. Because Kafka stores message data on-disk and in an ordered manner, it benefits from sequential disk reads. Considering the high resource cost of disk seeks, the fact that firstly Kafka processes reads and writes at a consistent pace, and secondly reads and writes happen simultaneously without getting in each other’s way, combine to deliver tremendous performance advantages.
With Kafka, horizontal scaling is easy. This means that Kafka can achieve the same high performance when dealing with any sort of task you throw at it, from the small to the massive.
2. Apache Kafka Architecture—Component Overview
Kafka architecture is made up of topics, producers, consumers, consumer groups, clusters, brokers, partitions, replicas, leaders, and followers.
The following diagram offers a simplified look at the interrelations between these components.
Kafka API Architecture
Apache Kafka offers four key APIst: the Producer API, Consumer API, Streams API, and Connector API.
Let’s take a brief look at how each of them can be used to enhance the capabilities of applications:
The Kafka Producer API enables an application to publish a stream of records to one or more Kafka topics.
The Kafka Consumer API enables an application to subscribe to one or more Kafka topics. It also makes it possible for the application to process streams of records that are produced to those topics.
The Kafka Streams API allows an application to process data in Kafka using a streams processing paradigm. With this API, an application can consume input streams from one or more topics, process them with streams operations, and produce output streams and send them to one or more topics. In this way, the Streams API makes it possible to transform input streams into output streams.
The Kafka Connector API connects applications or data systems to Kafka topics. This provides options for building and managing the running of producers and consumers, and achieving reusable connections among these solutions. For instance, a connector could capture all updates to a database and ensure those changes are made available within a Kafka topic.
3. Kafka Cluster Architecture
Now let’s take a closer look at some of Kafka’s main architectural components:
Typically, multiple brokers work in concert to form the Kafka cluster and achieve load balancing and reliable redundancy and failover. Brokers utilize Apache ZooKeeper® for management and coordination of the cluster. Each broker instance is capable of handling read and write quantities reaching to the hundreds of thousands each second (and terabytes of messages) without any impact on performance. Each broker has a unique ID, and can be responsible for partitions of one or more topic logs.
Kafka brokers also leverage ZooKeeper for leader elections, in which a broker is elected to lead the dealing with client requests for an individual partition of a topic. Connecting to any broker will bootstrap a client to the full Kafka cluster. To achieve reliable failover, a minimum of 3 brokers should be utilized—with greater numbers of brokers comes increased reliability.
Apache ZooKeeper Architecture
Kafka brokers use ZooKeeper to manage and coordinate the Kafka cluster. ZooKeeper notifies all nodes when the topology of the Kafka cluster changes, including when brokers and topics are added or removed.
For example, ZooKeeper informs the cluster if a new broker joins the cluster, or when a broker experiences a failure. ZooKeeper also enables leadership elections among brokers and topic partition pairs, helping determine which broker will be the leader for a particular partition (and server read and write operations from producers and consumers), and which brokers hold replicas of that same data. When ZooKeeper notifies the cluster of broker changes, they immediately begin to coordinate with each other and elect any new partition leaders that are required. This protects against the event that a broker is suddenly absent.
A Kafka producer serves as a data source that optimizes, writes, and publishes messages to one or more Kafka topics. Kafka producers also serialize, compress, and load balance data among brokers through partitioning.
Consumers read data by reading messages from the topics to which they subscribe. Consumers will belong to a consumer group. Each consumer within a particular consumer group will have responsibility for reading a subset of the partitions of each topic that it is subscribed to.
4. Basic Concepts
The following concepts are the foundation to understanding Kafka architecture:
A Kafka topic defines a channel through which data is streamed. Producers publish messages to topics, and consumers read messages from the topic they subscribe to.
Topics organize and structure messages, with particular types of messages published to particular topics. Topics are identified by unique names within a Kafka cluster, and there is no limit on the number of topics that can be created.
Within the Kafka cluster, topics are divided into partitions, and the partitions are replicated across brokers. From each partition, multiple consumers can read from a topic in parallel. It’s also possible to have producers add a key to a message—all messages with the same key will go to the same partition.
While messages are added and stored within partitions in sequence, messages without keys are written to partitions in a round robin fashion. By leveraging keys, you can guarantee the order of processing for messages in Kafka that share the same key. This is a particularly useful feature for applications that require total control over records. There is no limit on the number of Kafka partitions that can be created (subject to the processing capacity of a cluster).
Want Answers to Questions Like
“What impact does increasing partitions have on throughput?”
“Is there an optimal number of partitions for a cluster to maximize write throughput?”
Topic Replication Factor
Topic replication is essential to designing resilient and highly available Kafka deployments.
When a broker goes down, topic replicas on other brokers will remain available to ensure that data remains available and that the Kafka deployment avoids failures and downtime. The replication factor that is set defines how many copies of a topic are maintained across the Kafka cluster. It is defined at the topic level, and takes place at the partition level.
For example, a replication factor of 2 will maintain 2 copies of a topic for every partition. As mentioned above, a certain broker serves as the elected leader for each partition, and other brokers keep a replica to be utilized if necessary. Logically, the replication factor cannot be greater than the total number of brokers available in the cluster. A replica that is up to date with the leader of a partition is said to be an In-Sync Replica (ISR).
Kafka sends messages from partitions of a topic to consumers in the consumer group. At the time it is read, each partition is read by only a single consumer within the group. A consumer group has a unique group-id, and can run multiple processes or instances at once. Multiple consumer groups can each have one consumer read from a single partition. If the quantity of consumers within a group is greater than the number of partitions, some consumers will be inactive.
Kafka Internal Architecture in Brief
Assembling the components detailed above, Kafka producers write to topics, while Kafka consumers read from topics. Topics represent commit log data structures stored on disk. Kafka adds records written by producers to the ends of those topic commit logs. Topic logs are also made up of multiple partitions, straddling multiple files and potentially multiple cluster nodes. Consumers can use offsets to read from certain locations within topic logs. Consumer groups each remember the offset that represents the place they last read from a topic.
Partitions of topic logs are distributed across cluster nodes, or brokers, to achieve horizontal scalability and high performance. Kafka architecture can be leveraged to improve upon these goals, simply by utilizing additional consumers as needed in a consumer group to access topic log partitions replicated across nodes. This enables Apache Kafka to provide greater failover and reliability while at the same time increasing processing speed.