Apache Kafka® 4.0 has arrived, marking a milestone in streaming technology. Packed with new capabilities, this release includes architectural improvements like the long-awaited migration from ZooKeeper to KRaft, enhanced consumer group protocols, and updated Java requirements. Here’s a breakdown of key Apache Kafka 4.0 features every developer should know.

How did we get here? A brief Kafka history

It’s very exciting that Apache Kafka® turned “4.0” recently (March 18, 2025)—Happy Birthday Apache Kafka!

Looking at the history of Kafka releases I realized that I’ve been using Kafka since version “1.0” (“November 1, 2017). Over the years, Kafka has added many features and architectural changes and has grown significantly, from 400,000 Lines of Code (LOC) in 1.0 to 1.4M LOC in 4.0. Since then, I’ve learned various Kafka APIs, built demonstration applications, and specialized in Kafka performance and scalability.

In 2018, I built my first Kafka application, “Kongo”, a logistics/IoT delivery simulation using Kafka 1.0.

Kafka 2.0 came along shortly afterwards in mid-2018, and had significant improvements for Kafka Streams, Kafka® Connect, and more. I developed “Anomalia Machina“, a real-time anomaly detection application. This included several Kafka Streams examples (e.g. for truck overload detection).

Between 2018 and 2021, I created other Kafka 2.0 applications (e.g. real-time data processing pipeline using Kafka Connect), explored MirrorMaker2 (MM2), and conducted experiments with ZooKeeper™ (which at the time was the Kafka meta-data manager) and Debezium (e.g. PostgreSQL source, Cassandra source).

Kafka 2.8.0 (April 2021) introduced early access to replace ZooKeeper with a self-managed quorum, evolving into “KRaft” by 3.2 (May 2022). I investigated KRaft’s impact on performance and scalability, noting significant improvements. KRaft became production-ready in 3.3.1 (October 2022, see updated results and “Making Apache Kafka Even Faster And More Scalable”, Community over Code NA, 2024).

In 2023, Karapace, an open-source Kafka schema registry, was introduced, reflecting Kafka’s maturity and enterprise use (Karapace blog series).

Kafka 3.6.0 (2023) brought the major shift of Kafka Tiered Storage, enabling independent compute and storage elasticity. See my blog series on Tiered Storage and a series (and calculator) on sizing Kafka clusters for Tiered Storage.

Throughout these 7 years of changes, I leveraged new features without worrying about upgrading or migrating existing programs, thanks to the NetApp Instaclustr Managed Kafka service handling the complexities.

Kafka 4.0 changes

So, what does this mean for Kafka application developers moving from 3.X to 4.0?

Apache Kafka 4.0 introduces several significant changes and new features that application developers should be aware of. There are a lot of new KIPs included in 4.0, which add to the functionality, including for the Brokers, Controller, Producer, Consumer, Admin Client, Tools, Streams, and Connect APIs (see here). Many of these KIPs are designed to improve the reliability of the underlying system and may not impact Kafka clients directly. There are also lots of bug fixes.

But it’s worth reading through the relevant KIPs for your application to see if there are any potential changes you can benefit from to improve the functionality or performance of your application in 4.0. For example, here’s one I found that may be useful if you are planning on using Kafka Tiered Storage to keep lots of historical records:

KIP-1106: Add duration based offset reset option for consumer clients

Kafka consumers support the auto.offset.reset config option, which is used when there is no initial offset in Kafka, or if the current offset does not exist anymore on the server. This config previously supported earliest/latest/none options. Consumer resets might force applications to reprocess large amounts of data from earlier offsets. With infinite storage, it’s beneficial to have a duration based offset reset strategy. This KIP allows applications to consume/initialize from a fixed duration when there is no initial offset in Kafka.

However, for the rest of this blog, I will mainly focus on major changes and things to watch out for.

Removal of ZooKeeper

It feels like Kafka has been “abandoning” ZooKeeper for ages now (e.g. 2022 blog “Apache Kafka KRaft Abandons the ZooKeeper”), but finally, KRaft (Kafka Raft, KIP-500) is now the default Kafka metadata management system, completely replacing ZooKeeper. This change simplifies cluster management and enhances scalability. If you are using the NetApp Instaclustr managed Kafka service, this migration will be done transparently for you (automatically and at no additional costs). This is a good thing as even though it’s possible to migrate clusters manually, it’s non-trivial (there’s a whole KIP-866 devoted to ZooKeeper to KRaft migration).

New Consumer Group Protocol (KIP-848)

The next generation consumer rebalance protocol improves stability and scalability by eliminating global synchronization barriers during consumer group rebalances. You will need to update your client-side code to use Kafka 4.0. There are some configuration changes as well, see my latest blog for more details.

Queues for Kafka (shared group)

This introduces the concept of shared groups, allowing multiple consumers per partition and enabling queue-based messaging patterns. Watch out for a future blog exploring this topic. It’s only early access in 4.0 so you will need to test this out in a non-production environment to explore potential benefits for your use cases.

Deprecated APIs

In Kafka 4.0 some APIs have been deprecated or removed, see the full list here (there are many deprecated fields, methods and constructors that I haven’t listed here).

Some of the changes are highlighted here.

Streams API changes

  • org.apache.kafka.streams.kstream.Transformer
    • Deprecated in favour of api.Processor
  • org.apache.kafka.streams.kstream.TransformerSupplier
    • Use api.ProcessorSupplier instead
  • org.apache.kafka.streams.kstream.ValueTransformer
    • Replaced by FixedKeyProcessor
  • org.apache.kafka.streams.kstream.ValueTransformerSupplier
    • Use FixedKeyProcessorSupplier

Deprecated classes and Enum classes

  • org.apache.kafka.streams.kstream.ForeachProcessor
    • Should no longer be used
  • org.apache.kafka.streams.processor.MockProcessorContext
    • Use MockProcessorContext instead
  • org.apache.kafka.clients.consumer.OffsetResetStrategy
    • Replaced by AutoOffsetResetStrategy
  • org.apache.kafka.common.ConsumerGroupState
    • Use GroupState instead
  • org.apache.kafka.streams.Topology.AutoOffsetRest
    • Use org.apache.kafka.streams.AutoOffsetReset instead

As part of KIP-896, several old client protocol API versions have been removed to reduce complexity and improve maintainability. In 4.0 support has been removed for multiple older versions of many APIs. I suggest you identify and update any usage of deprecated APIs in your codebase to their recommended replacements, and test compatibility with the new API versions before migrating to production.

KIP-1030: Change constraints and default values for various configurations

KIP-1030 could be useful to check as it changes a bunch of configuration defaults and constraints and provides a helpful “motivation for the change” explanation for each.

KIP-896: Remove old client protocol API versions in Kafka 4.0

This KIP removes support for old protocol API versions – the baseline protocol API version is now Kafka 2.1. This will only impact clients using very old code as Kafka 2.1 was released when I started using Kafka, way back in 2018—it’s definitely time to upgrade if you haven’t already done so.

KIP-653: Upgrade log4j to log4j2

The title of this KIP says it all really—time to upgrade to Apache log4j2!

Minimum Java versions

As per the normal Kafka release process, support for Java versions is limited to ensure the efficiency of testing and removal of older Java versions. For Apache Kafka 4.0, the minimum required Java versions are:

  • Java 11 for Kafka Clients and Kafka Streams
    • Java 11, 17 and 23 are supported
  • Java 17 for Kafka Brokers, Connect, and Tools
    • Java 17 and 23 are supported.

Reasons for this update in Java versions include security, performance, and maintainability. This compatibility matrix is useful for understanding version/component compatibility. You need to ensure your development and production environments are running at least Java 11 for clients and Java 17 for brokers and other components (although for NetApp Instaclustr managed Kafka we will naturally handle the broker side upgrades for you). It’s also worth thoroughly testing your applications with the new Java versions to identify any compatibility issues or dependency issues (i.e. libraries used in your application may also require newer Java versions).

Finally, this is only a summary of changes. There’s a complete list of changes and how to upgrade to Kafka 4.0 here.

Update!

My recent blog on the Next Generation of the Consumer Rebalance Protocol is here.

I discovered (more or less by accident, as my performance experiment started with the classic protocol and then moved on to the new consumer protocol, but with the same consumer group ID), that KIP-848 and Kafka 4.0 support automatic migration of consumer groups from the classic to the consumer protocol. The new broker configuration property, group.consumer.migration.policy, is set to “bidirectional” by default which enables converting between classic and consumer groups automatically —in both directions—i.e. upgrading from class to consumer or downgrading from consumer to classic groups. Other options are upgrade, downgrade or disabled.

Conclusion

Apache Kafka 4.0 marks a revolutionary step forward for streaming technology. From replacing ZooKeeper with KRaft (making future Kafka cluster operation simpler), to introducing cutting-edge features like duration-based offset resets, developers can expect streamlined operations and improved scalability. Another highlight is the next-generation consumer group protocol, designed to eliminate global synchronization barriers during rebalances, providing enhanced stability and performance for client applications. By following the migration tips outlined here, you’ll ensure a smooth transition while taking full advantage of Apache Kafka 4.0’s features.

That’s all for now—have fun at the Kafka “4.0” birthday party!

Try 4.0 out here with our free trial on the Instaclustr Managed Platform.