What’s new in Kafka® 4.1.0? Introducing the new Streams Rebalance Protocol

It’s been a while since I’ve looked at new Apache Kafka versions and features (Kafka 4.0.0 back in July/August 2025, just after we released 4.0 as GA on our managed platform). There were lots of new additions in 4.0 to talk about, for example, the next generation consumer rebalance protocol, share groups (queues for Kafka), and more:

1. What’s New In Apache Kafka 4.0?
2. Rebalance your Apache Kafka® partitions with the next generation Consumer Rebalance Protocol—up to 20x faster!
3. Apache Kafka “Turbo” Rebalancing with the Next Generation Consumer Rebalance Protocol
4. Kafka® 4.0 unveiled: Key changes and how they impact developers
5. Apache Kafka 4.0 share groups: What you need to know about queues for Kafka
6. Top use cases for queues for Kafka: Unlocking the potential of Apache Kafka® 4.0 share groups
7. Increased lingering in Apache Kafka 4.0 can increase latency
8. Join the Queue for Apache Kafka® Share Groups

Since then, Kafka 4.1.0 was released (September 2025, see detailed release notes), with around 472 Kafka Improvement (KIPs), including new features, improvements, bug fixes, tests, and more—well done to the Apache Kafka open source community! Kafka 4.1.1 (a bugfix release) was made available on the NetApp Instaclustr Managed Platform in December 2025.

So, what’s changed from 4.0 to 4.1.0? What are the most interesting improvements (for me at least)? In this blog, we focus on a new improvement, the Streams Rebalance Protocol.

Streams Rebalance Protocol (KIP-1071) in early access

The Rakaia River (NZ) wetland rewilding returned the river’s eastern branch (right) to its original course (Source: Wikipedia)

Did you know that “rebalancing” streams is beneficial for rewilding river systems? Enabling natural processes for single channel rivers, allows them to have more space and dynamically wander and widen into multiple “braided” streams, encouraging more natural flows, supporting higher biodiversity, and becoming more self-sustaining ecosystems. There are lots of braided rivers in NZ, including the Rakaia River in the South Island (see image above). This sounds like a good idea for Kafka Streams!

The 4.1.0 release also introduces (in early access) a new Streams Rebalance Protocol (KIP-1071) based on the new consumer group protocol (KIP-848). Now Kafka Streams can “join the groups”, with a new streams group (as well as the existing share groups or consumer groups).

KIP‑1071 introduces a new, broker‑driven rebalance protocol specifically for Kafka Streams applications.

A Kafka Streams application participates in the new Streams group. A rebalance occurs whenever the group’s membership or workload assignment must change. The triggers match the underlying consumer‑group mechanics, plus Streams‑specific reasons tied to task assignment and topology. So, Streams rebalances are potentially more frequent and complex! Causes of a rebalance in Kafka Streams include:

• A Streams application instance joins the group (scale-out)
• A Streams application instance leaves the group (scale-in or failure)
• Heartbeat/session timeout issues
• Slow processing causing max.poll.interval.ms to be exceeded, causing the consumer to be removed from the group.
• Topic or partition changes
• Topology changes in Kafka Streams—e.g. adding/removing sub-topologies or source topics—can trigger a rebalance.
• And migration between classic consumer groups and Streams groups and group coordinator changes will trigger a rebalance.

Why you need this Apache Kafka 4.1.X feature

KIP-1071 becomes necessary in scenarios where Kafka Streams workloads outgrow the limitations of the classic consumer‑group–based rebalance process. You are likely to need KIP-1071 when:

1. You need stable, faster rebalances

KIP‑1071 eliminates client‑side coordination and moves assignment decisions to the broker, reducing the “stop‑the‑world” effect seen with classic rebalances. Use it if your streams application suffers from frequent consumer churn, long rebalance pauses or lack of transparency about why rebalances happen.

2. You want to scale horizontally

If you are scaling a Streams application across many instances, KIP-1071 provides smoother task assignment, reduced thrashing of tasks, better broker-driven coordination for large groups. This helps large, distributed apps avoid unpredictable task movement and rebalance storms.

3. You require separate streams groups

Under KIP-1071, a Streams application registers as a streams group, not a consumer group. This is needed when your topology is complex, you require Streams-specific metadata tracking, or you want the broker to maintain task and topology metadata. This separation gives Streams applications their own lifecycle and reduces conflicts with consumer group semantics.

4. You want the broker to maintain topology metadata for Streams

KIP-1071 stores topology metadata on the broker, allowing continuous assignment computation, centralized consistency and improved failure detection. Use KIP-1071 whenever topology coordination needs to be more authoritative and less dependent on clients to coordinate among themselves.

5. You are upgrading to Kafka 4.x

Kafka 4.1 and 4.2 increasingly promote KIP-1071 as the future of Streams coordination.
You need it when adopting new features requiring the modern protocol and preparing for the deprecation of older rebalancing mechanisms.

How to enable KIP-1071 in Kafka 4.1.1

Ready to test this out? You can enable the early access KIP-1071 feature with a few simple steps:

1. Shut down all instances of your Kafka Streams application.
2. Wait for the session.timeout.ms to expire so the consumer group becomes empty. Alternatively, you can force an explicit group leave using the kafka-streams-groups.sh tool.
3. Update your application’s configuration by adding the following property:
   
   Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "your-application-id"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "IP:9092"); // Enable the Streams Rebalance Protocol props.put("group.protocol", "streams"); KafkaStreams streams = new KafkaStreams(builder.build(), props);

   1 2 3 4 5 6 7 8

   Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "your-application-id"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "IP:9092");   // Enable the Streams Rebalance Protocol props.put("group.protocol", "streams");   KafkaStreams streams = new KafkaStreams(builder.build(), props);

4. Restart the application instances.

More details on upgrading consumer groups can be found here.

If you are interested to find out more about Kafka Streams applications, I’ve written a few blogs in the past, including:

• Introduction to Kafka Streams  
• Apache Kafka Streams truck overload example 
• Dr Black has been murdered in the Billiard Room with a Candlestick! Whodunnit?! 
• A Kafka Streams Toy and Box Packing Solution  
• Tracing Kafka Streams with OpenTelemetry   

Next steps

To get the most out of these Apache Kafka 4.1.X features, we recommend testing the Streams Rebalance Protocol in a controlled environment. You can try out this early access feature today by starting a 30-day free trial of our Managed Apache Kafka platform. Dive in, experiment with the new streams group, and watch your application scale with enhanced stability.