# Apache Kafka tutorial: Get started with Kafka in 5 simple steps Apache Kafka tutorial: Get started with Kafka in 5 simple steps =============================================================== Apache Kafka is an open source, distributed event-streaming platform for building high-performance, real-time data pipelines and resilient, decoupled microservices. Unlike traditional message queues that delete data on receipt, Kafka stores it in an immutable, distributed commit log. ![https://www.instaclustr.com/wp-content/uploads/Screenshot-2025-08-01-at-1.20.29 PM.png](https://www.instaclustr.com/wp-content/uploads/Screenshot-2025-08-01-at-1.20.29%E2%80%AFPM.png) Scaling seamlessly with Apache Kafka: Insights for enterprise leaders Plan and build your architecture for efficiency and performance> [ Download now ](/resources/scaling-seamlessly-with-apache-kafka-insights-for-enterprise-leaders/) ![https://www.instaclustr.com/wp-content/uploads/Screenshot-2025-08-01-at-1.20.29 PM.png]() Scaling seamlessly with Apache Kafka: Insights for enterprise leaders Plan and build your architecture for efficiency and performance> [ Download now ](/resources/scaling-seamlessly-with-apache-kafka-insights-for-enterprise-leaders/) - [ What is Apache Kafka? ](#sec-0) - [ Benefits of Apache Kafka ](#sec-1) - [ Key use cases of Apache Kafka ](#sec-2) - [ Tips from the expert ](#sec-3) - [ Tutorial for beginners: Getting started with Apache Kafka ](#sec-4) - [ Simplifying Apache Kafka management with Instaclustr ](#sec-5) What is Apache Kafka? Benefits of Apache Kafka Key use cases of Apache Kafka Tips from the expert Tutorial for beginners: Getting started with Apache Kafka Simplifying Apache Kafka management with Instaclustr What is Apache Kafka? --------------------- Apache Kafka is an open-source, distributed event-streaming platform optimized for building high-performance, real-time data pipelines and resilient, decoupled microservices. Unlike traditional message queues that delete data immediately upon receipt, Kafka preserves data inside an immutable, distributed commit log, acting as a highly reliable single source of truth for your data streaming architectures. Originally developed by LinkedIn and donated to the Apache Software Foundation, Kafka handles real-time data feeds with high throughput, low latency, and fault tolerance. It is integral to modern data pipelines, supporting use cases such as log aggregation, real-time analytics, and event sourcing. **Core concepts to know:** - **Events:** a record of something that happened, such as a payment transaction or a user click, stored as a key-value pair with a timestamp. - **Producers:** client applications that publish (write) events into Kafka topics. - **Consumers:** client applications that subscribe to (read and process) those events. - **Topics:** categories or feed names where events are organized and durably stored on disk. - **Partitions:** subdivisions of topics spread across servers, letting data be processed in parallel for scalability. - **Brokers:** the individual servers that make up a Kafka cluster. **Tooling to grow into:** - **Kafka Connect:** import and export bulk data between external systems and Kafka topics without writing custom code. - **Kafka Streams:** a native client library for real-time event operations, stateful joins, and aggregations. - **Client libraries:** build producers and consumers in Java, Python, .NET, Go, and other languages. *This is part of a series of articles about [Apache Kafka](https://www.instaclustr.com/education/apache-kafka/)* Benefits of Apache Kafka ------------------------ Apache Kafka offers benefits that make it a key component of modern data infrastructure. First, it provides high throughput and low latency, enabling real-time data processing. Kafka handles millions of messages per second, making it suitable for applications requiring rapid data ingestion and processing. Its distributed nature ensures high availability, as data is replicated across multiple nodes, preventing single points of failure and ensuring system resilience. Kafka’s scalability is another advantage. It can efficiently scale horizontally by adding more brokers to the cluster, accommodating increased data volumes and processing demands. The platform also supports diverse data integration capabilities, allowing integration with various data sources and sinks, including databases, cloud storage, and big data frameworks like Hadoop and Spark. Additionally, Kafka’s strong durability guarantees ensure that data is safely stored and retrievable even in the event of system failures, making it reliable for critical applications. ***Learn more in our detailed guide to [Kafka management](https://www.instaclustr.com/support/documentation/kafka/using-kafka/topic-management/)*** Key use cases of Apache Kafka ----------------------------- Apache Kafka’s versatility and robustness make it ideal for a variety of use cases across different industries. Here are some of the key use cases: - **Real-time analytics:** Organizations leverage Kafka for real-time data analytics to gain immediate insights. By streaming data from various sources into a Kafka cluster, companies can process and analyze the data in real time, enabling quick decision-making. This is particularly useful in monitoring systems, fraud detection, and online recommendations. - **Log aggregation:** Kafka serves as a centralized platform for collecting logs from multiple systems. By consolidating logs into a single Kafka cluster, organizations can simplify log processing, making it easier to search, analyze, and store log data. This approach enhances system monitoring and debugging capabilities. - **Event sourcing:** In event-driven architectures, Kafka is used to capture and store every change to an application’s state as an immutable event. This allows for a complete replay of events to reconstruct past states, ensuring data consistency and reliability. It’s widely used in financial systems, order management, and customer behavior tracking. - **Data integration:** Kafka acts as a data pipeline to facilitate data integration across different systems. It can ingest data from various sources, such as databases, applications, and sensors, and then stream it to downstream systems for further processing and storage. This is crucial for building scalable and efficient data architectures. - **Stream processing:** Kafka, combined with stream processing frameworks like Apache Flink, Apache Storm, and Kafka Streams, enables real-time processing of data streams. This capability is essential for applications that require continuous processing and analysis of incoming data, such as sensor data processing, anomaly detection, and real-time dashboards. - **Messaging:** Kafka’s messaging system supports asynchronous communication between microservices. It can handle large volumes of messages with high throughput and low latency, ensuring reliable and scalable message delivery. This makes Kafka an ideal choice for microservices architectures, enhancing their reliability and performance. ### Related content: Read our guide to Kafka management [ Read now ](/support/documentation/kafka/using-kafka/topic-management/) Tips from the expert -------------------- ![Andrew Mills]()Andrew Mills Senior Solution Architect Andrew Mills is an industry leader with extensive experience in open source data solutions and a proven track record in integrating and managing Apache Kafka and other event-driven architectures. In my experience, here are tips that can help you better utilize Apache Kafka: 1. **Partition your topics effectively:** Distribute partitions across brokers to balance the load. Consider key-based partitioning to ensure related messages are processed together, which is crucial for maintaining data integrity in some applications. 2. **Implement monitoring and alerting:** Set up robust monitoring using tools like Prometheus and Grafana. Track key metrics such as producer latency, consumer lag, and broker health to proactively address performance and reliability issues. 3. **Secure your Kafka cluster:** Use TLS for encrypting data in transit and implement SASL for authentication. Properly configure ACLs to control access and ensure that only authorized applications and individuals can produce or consume messages. 4. **Optimize batch sizes and linger time:** Adjusting the batch size and linger time for producers can significantly impact throughput and latency. Larger batch sizes improve throughput, while increasing linger time can help collect more records per batch. Tutorial for beginners: Getting started with Apache Kafka --------------------------------------------------------- This tutorial is adapted from the official Kafka documentation. ### Step 1: Get Kafka To start with Apache Kafka, download the latest Kafka release from the Kafka download page and extract it using the following commands: wget https://dlcdn.apache.org/kafka/4.3.0/kafka\_2.13-4.3.0.tgz $ tar -xzf kafka\_2.13-4.3.0.tgz $ cd kafka\_2.13-4.3.0 1 2 3 wget https://dlcdn.apache.org/kafka/4.3.0/kafka\_2.13-4.3.0.tgz $ tar -xzf kafka\_2.13-4.3.0.tgz $ cd kafka\_2.13-4.3.0 ### Step 2: Initialize Kafka To set up your Kafka environment, ensure your system has Java 17 or above installed (Kafka 4.0 and later require Java 17 for brokers, Connect, and tools) or above installed. As of Kafka 4.0, clusters run exclusively in KRaft (Kafka Raft) mode for metadata management, and ZooKeeper has been removed entirely. You can start Kafka using the downloaded local scripts or a Docker image. **Starting Kafka with KRaft:** Generate a Cluster UUID: $ export KAFKA\_CLUSTER\_ID="$(bin/kafka-storage.sh random-uuid)" 1 $ export KAFKA\_CLUSTER\_ID="$(bin/kafka-storage.sh random-uuid)" Format the log directories: $ bin/kafka-storage.sh format --<span style="font-weight: 400;" data-mce-style="font-weight: 400;">standalone</span> -t $KAFKA\_CLUSTER\_ID -c config/server.properties 1 $ bin/kafka-storage.sh format --<span style="font-weight: 400;" data-mce-style="font-weight: 400;">standalone</span> -t $KAFKA\_CLUSTER\_ID -c config/server.properties Start the Kafka server: $ bin/kafka-server-start.sh config/server.properties 1 $ bin/kafka-server-start.sh config/server.properties **Starting Kafka using a Docker image:** Pull the Kafka Docker image: $ docker pull apache/kafka:4.3.0 1 $ docker pull apache/kafka:4.3.0 Run the Kafka Docker container: $ docker run -p 9092:9092 apache/kafka:4.3.0 1 $ docker run -p 9092:9092 apache/kafka:4.3.0 ### Step 3: Create a topic and write events Before writing events to Kafka, create a topic to store them. In a new terminal session, run: $ bin/kafka-topics.sh --create --topic my-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 1 2 $ bin/kafka-topics.sh --create --topic my-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 To view details of the created topic: $ bin/kafka-topics.sh --describe --topic my-events --bootstrap-server localhost:9092 1 $ bin/kafka-topics.sh --describe --topic my-events --bootstrap-server localhost:9092 The output should look something like: ![Kafka tutorial screenshot]() Use the console producer client to write events to your topic: $ bin/kafka-console-producer.sh --topic my-events --bootstrap-server localhost:9092 1 $ bin/kafka-console-producer.sh --topic my-events --bootstrap-server localhost:9092 Type your messages, each resulting in an event: Example of one event Example of another event 1 2 Example of one event Example of another event Stop the producer client with `Ctrl-C`. ### Step 4: Read the Events Open a new terminal session and use the console consumer client to read the events: $ bin/kafka-console-consumer.sh --topic my-events --from-beginning --bootstrap-server localhost:9092 1 $ bin/kafka-console-consumer.sh --topic my-events --from-beginning --bootstrap-server localhost:9092 You should see: Example of one event Example of another event 1 2 Example of one event Example of another event Stop the producer client with `Ctrl-C`. ### Step 5: Use Kafka Connect to import/export data and process with Kafka streams Kafka Connect allows continuous data integration from external systems. First, ensure you have `connect-file-.jar` in the plugin path (you can find this file inside your Kafka directory, libs subfolder). Edit the `config/connect-standalone.properties` file as follows: echo "plugin.path=libs/connect-file-4.3.0.jar" >> config/connect-standalone.properties 1 echo "plugin.path=libs/connect-file-4.3.0.jar" >> config/connect-standalone.properties Create seed data: echo -e "example\\nbar" > myfile.txt 1 echo -e "example\\nbar" > myfile.txt Start the connectors in standalone mode: bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties 1 2 3 bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties The output should look something like: ![Kafka tutorial screenshot]() Verify the data in the output file using this command: more myfile.sink.txt 1 more myfile.sink.txt ![Kafka tutorial screenshot]() You can also use the console consumer to verify: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning 1 2 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning ![Kafka tutorial screenshot]() Add more data to `myfile.txt`: echo Another line >> myfile.txt 1 echo Another line >> myfile.txt ![Kafka tutorial screenshot]() Let’s implement a simple filtering algorithm that processes data stored in Kafka. This example will filter out any lines containing the word “filter” and write the rest to an output topic. **Note:** You will need to create a complete application using Maven/Gradle in order to run the code snippet. StreamsBuilder builder = new StreamsBuilder(); KStream<String, String> textLines = builder.stream("my-events"); // Filter out lines containing the word "filter" KStream<String, String> filteredLines = textLines.filter( (key, value) -> !value.toLowerCase().contains("filter") ); // Send the filtered data to a new topic filteredLines.to("output-topic", Produced.with(Serdes.String(), Serdes.String())); KafkaStreams streams = new KafkaStreams(builder.build(), getStreamsConfig()); streams.start(); 1 2 3 4 5 6 7 8 9 10 11 12 13 StreamsBuilder builder = new StreamsBuilder(); KStream<String, String> textLines = builder.stream("my-events"); // Filter out lines containing the word "filter" KStream<String, String> filteredLines = textLines.filter( (key, value) -> !value.toLowerCase().contains("filter") ); // Send the filtered data to a new topic filteredLines.to("output-topic", Produced.with(Serdes.String(), Serdes.String())); KafkaStreams streams = new KafkaStreams(builder.build(), getStreamsConfig()); streams.start(); In this example: - We create a `StreamsBuilder` instance. - We read from the`my-events` topic using `builder.stream("my-events")`. - We apply a filter to remove any lines containing the word “filter”. - The filtered lines are written to the `output-topic`. To test above functionality you will need to open three terminals: java -jar target/kafka-streams-example-1.0-SNAPSHOT.jar 1 java -jar target/kafka-streams-example-1.0-SNAPSHOT.jar ![Kafka tutorial screenshot]() On terminal number 2, start the producer using the following command: bin/kafka-console-producer.sh --topic my-events --bootstrap-server localhost:9092 1 2 bin/kafka-console-producer.sh --topic my-events --bootstrap-server localhost:9092 ![Kafka tutorial screenshot]() On terminal number 3, start the consumer with the topic ‘output-topic’ this is where our Java code forwards the filtered messages: bin/kafka-console-consumer.sh --topic output-topic --from-beginning --bootstrap-server localhost:9092 1 2 bin/kafka-console-consumer.sh --topic output-topic --from-beginning --bootstrap-server localhost:9092 ![Kafka tutorial screenshot]() You will notice that any line that contains the word “filter” are removed from the output-topic as shown in the above screenshot. Beyond the command line, you can build production producers and consumers using the official Kafka client libraries, which are available for Java, Python, .NET, Go, and other languages. Kafka Streams adds a native Java and Scala library for stateful, real-time transformations such as joins and aggregations directly on your topics. **Learn more about [Kafka architecture](https://www.instaclustr.com/education/apache-kafka/apache-kafka-architecture-a-complete-guide-2026/)** Simplifying Apache Kafka management with Instaclustr ---------------------------------------------------- [Apache Kafka](https://www.instaclustr.com/education/apache-kafka/) has emerged as a popular distributed streaming platform for building real-time data pipelines and streaming applications. However, managing and operating Kafka clusters can be complex and resource-intensive. This is where Instaclustr comes into play. Instaclustr simplifies Apache Kafka management and helps organizations focus on their core business objectives. Within few clicks or API calls, you can provision Kafka clusters on your preferred cloud provider or on-premises infrastructure. This convenience extends to real-time monitoring, where you get immediate insights into your cluster metrics, such as throughput, latency, and partition lag. Furthermore, Instaclustr shoulders the responsibility of keeping your Kafka clusters updated, handling automated maintenance, and applying patches as required. - **Simple Deployment:** Instaclustr provides a seamless experience for deploying Apache Kafka clusters on your desired platform. - **Real-Time Monitoring:** Comprehensive monitoring capabilities provide insights into cluster metrics to ensure smooth operations. - **Maintenance Ease:** Instaclustr takes care of automated maintenance, upgrades, and ensures your clusters run on the latest stable versions. Not only does Instaclustr simplify management complexities, but it also helps you enhance data security. Encryption at rest and in transit, access controls, authentication mechanisms, and assistance in meeting compliance requirements are all part of the robust security features Instaclustr offers. Additionally, it ensures data resilience through built-in disaster recovery and high availability features, minimizing the risk of data loss. In essence, Instaclustr offers a comprehensive managed service for Apache Kafka, enabling businesses to harness the power of Kafka without the hassle of infrastructure management. Reduce costs, enhance security, and streamline operations with Instaclustr. **Free download: Apache Kafka insights e-Book [\[Access now\]](/resources/scaling-seamlessly-with-apache-kafka-insights-for-enterprise-leaders/)** For more information visit our [Instaclustr for Apache Kafka page](https://www.instaclustr.com/platform/managed-apache-kafka/) or check out the following blogs: - [Designing Apache Kafka® Applications With Data Privacy Regulation Compliance](https://www.instaclustr.com/blog/designing-apache-kafka-applications-with-data-privacy-regulation-compliance/) - [Instaclustr for Apache Kafka® Introduces Support for Tiered Storage in Public Preview](https://www.instaclustr.com/blog/instaclustr-for-apache-kafka-introduces-support-for-tiered-storage-in-public-preview/) - [Custom Subject Alternative Names now available on Instaclustr for Apache Kafka](https://www.instaclustr.com/blog/custom-subject-alternative-names/) ### Related content [Apache Kafka on AWS: Features, pricing, tutorial and best practices](https://www.instaclustr.com/education/apache-kafka/apache-kafka-on-aws-features-pricing-tutorial-and-best-practices/) [Apache Kafka cluster: Key components and building your first cluster](https://www.instaclustr.com/education/apache-kafka/apache-kafka-cluster-key-components-and-building-your-first-cluster/) [13 Kafka Best Practices: Run Kafka Like the Pros](https://www.instaclustr.com/education/apache-kafka/13-kafka-best-practices-run-kafka-like-the-pros/) [Apache Kafka® API](https://www.instaclustr.com/education/apache-kafka/apache-kafka-api/) [Apache Kafka® Connect: The basics and a quick tutorial](https://www.instaclustr.com/education/apache-kafka/apache-kafka-connect-the-basics-and-a-quick-tutorial/) [Apache Kafka® architecture: A complete guide \[2026\]](https://www.instaclustr.com/education/apache-kafka/apache-kafka-architecture-a-complete-guide-2026/) [Apache Kafka® broker: Key components, tutorial, and best practices](https://www.instaclustr.com/education/apache-kafka/apache-kafka-broker-key-components-tutorial-and-best-practices/) [Apache Kafka® integration: Challenges, solutions and best practices](https://www.instaclustr.com/education/apache-kafka/apache-kafka-integration-challenges-solutions-and-best-practices/) [Apache Kafka® versions: Managing upgrades and tips for success](https://www.instaclustr.com/education/apache-kafka/apache-kafka-versions-managing-upgrades-and-tips-for-success/) [Apache Kafka® vs ActiveMQ: 5 key differences and how to choose](https://www.instaclustr.com/education/apache-kafka/apache-kafka-vs-activemq-5-key-differences-and-how-to-choose/) [Apache Kafka®: 4 use cases and 4 real-life examples](https://www.instaclustr.com/education/apache-kafka/kafka-4-use-cases-and-4-real-life-examples/) [Apache Kafka®: Overview and concepts](https://www.instaclustr.com/education/apache-kafka/apache-kafka-overview-and-concepts/) [Best Managed Apache Kafka Services: Top 6 Options in 2026](https://www.instaclustr.com/education/apache-kafka/best-managed-apache-kafka-services-top-5-options-in-2025/) [Best Managed Apache Kafka Tools: Top 7 Services in 2026](https://www.instaclustr.com/education/apache-kafka/best-managed-apache-kafka-tools-top-5-services-in-2026/) [Deploying Apache Kafka® with Docker: A practical guide](https://www.instaclustr.com/education/apache-kafka/deploying-apache-kafka-with-docker-a-practical-guide/) [Hosted Apache Kafka®: Features to look for and 4 key considerations](https://www.instaclustr.com/education/apache-kafka/hosted-apache-kafka-features-to-look-for-and-4-key-considerations/) [Kafka performance: 7 critical best practices in 2026](https://www.instaclustr.com/education/apache-kafka/kafka-performance-7-critical-best-practices-in-2026/) [Apache Kafka tutorial: Get started with Kafka in 5 simple steps](https://www.instaclustr.com/education/apache-kafka/apache-kafka-tutorial-get-started-with-kafka-in-5-simple-steps/) [Best Kafka as a service platforms: Top 5 in 2026](https://www.instaclustr.com/education/apache-kafka/best-kafka-as-a-service-platforms-top-5-in-2026/) [Best hosted Apache Kafka services: Top 9 providers in 2026](https://www.instaclustr.com/education/apache-kafka/best-hosted-apache-kafka-services-top-6-providers-in-2026/) [Best managed Kafka solutions: Top 10 services in 2026](https://www.instaclustr.com/education/apache-kafka/best-managed-kafka-solutions-top-10-services-in-2026/) ### Related content [ What is vector similarity search? Pros, cons, and 5 tips for success Vector similarity search is an information retrieval technique that matches data on semantic meaning rather than exact keyword ... ](https://www.instaclustr.com/education/vector-database/what-is-vector-similarity-search-pros-cons-and-5-tips-for-success/) [ What are managed database services and 7 key capabilities A managed database service (MDS) allows organizations to outsource the maintenance and management of database systems to a third-... ](https://www.instaclustr.com/education/data-architecture/what-are-managed-database-services-and-7-key-capabilities/) [ Vector search vs semantic search: 4 key differences and how to choose Vector search finds items in a dataset using vectors. Semantic search boosts accuracy by grasping searcher intent and term context... ](https://www.instaclustr.com/education/vector-database/vector-search-vs-semantic-search-4-key-differences-and-how-to-choose/) Optimize your Kafka cluster performance Today ------------------------------------------------- Whether you're deploying Kafka for the first time, optimizing throughput, or troubleshooting production concerns, we're here every step of the way. [ Book free consultation ](/contact-us/)