Using Apache Kafka Share Groups / Queues
Overview
Kafka Share Groups (also known as Queues for Kafka) is a feature introduced in Apache Kafka 4.0 that provides queue-like semantics for Kafka consumers. This feature allows multiple consumers to share the load of processing messages from a topic, similar to traditional message queue systems.
Share Groups enable consumers to work together to process records from partitions as fast as possible. Unlike traditional consumer groups, share groups allow the number of consumers to exceed the number of partitions, enabling finer-grained sharing of records with less strict ordering requirements.
Cluster Requirements
You must have at least Apache Kafka 4.1 or later to use the Share Groups (Queues) feature. Note that in Kafka 4.1, this feature is not supported for production purposes, and is also not advised for production use on our managed clusters, i.e. it is not covered under production tier SLAs.
How Share Groups Work
Comparison with Traditional Consumer Groups
Traditional Kafka consumer groups (classic and consumer protocols) provide a balance of ordering and scalability, but limit the number of consumers in a group to strictly less than or equal to the number of partitions. Exactly one consumer is assigned to each topic partition, ensuring ordering per partition.
Share Groups work differently:
- Multiple consumers per partition: Consumers in a share group can consume records from topic-partitions that may be assigned to multiple consumers
- No partition-consumer limit: The number of consumers can exceed the number of partitions
- Cooperative processing: Records are processed “cooperatively” based on record availability without the constraint of partition-consumer mappings
- Individual record processing: Records are conceptually processed (and acknowledged) individually but can be batched for efficiency
Ordering Guarantees
Important: Share Groups provide less ordering compared to classic/consumer groups:
- Partial ordering only: Records returned in a batch for a share-partition are guaranteed to be in order
- No guarantees between batches: There are no guarantees about the ordering between different batches
- Redelivery: When there are timeouts/errors and redelivery occurs, ordering is not guaranteed
This reduced ordering is by design, as share groups are optimized for use cases where:
- Ordering doesn’t matter (as much)
- Events can be processed independently and concurrently by a pool of consumers
- Processing is based on demand/availability rather than strict 1-1 mapping between consumers and partitions
Delivery Guarantees
Share Groups provide “at least once” delivery guarantees. Records may be redelivered in case of timeouts or errors.
Configuration Limits
Share Groups have the following configuration limits and defaults:
- Maximum consumers: Default is 200, with a maximum of 1,000
- Delivery count limit: 5 (maximum number of times a record can be delivered before being skipped)
- Record lock duration: 30 seconds (how long a record is locked to a consumer)
- In-flight records per share partition: 200 (maximum number of records that can be in-flight per partition; 2,000 in Kafka 4.1)
Enabling Share Groups / Queues
For Kafka 4.1.x
For Kafka 4.1.x clusters, customers can enable the Share Groups feature themselves using the kafka-features.sh CLI command script.
Enable the Feature
Run the following command on one of your Kafka nodes:
|
1 |
./kafka-features.sh --command-config kafka.properties --bootstrap-server <bootstrap_server_ip>:9092 upgrade --feature share.version=1 |
Disable the Feature
To disable the Share Groups feature, you can downgrade the feature version back to 0:
|
1 |
./kafka-features.sh --command-config kafka.properties --bootstrap-server <bootstrap_server_ip>:9092 downgrade --feature share.version=0 |
Using Share Groups / Queues
Client Configuration
To use Share Groups on the client side, consumers need to configure a group.id in your consumer properties. Unlike traditional consumer groups, you use a new consumer class: KafkaShareConsumer.class or the kafka-console-share-consumer.sh CLI tool.
Java Consumer Example
Here’s an example of using Share Groups with the Java client:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
import java.time.Duration; import java.util.List; import java.util.Properties; import org.apache.kafka.clients.consumer.KafkaShareConsumer; public class ShareGroupExample { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "<node_ip>:9092"); props.put("group.id", "my-share-group"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); // Create the Kafka share consumer KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props); // Subscribe to topics consumer.subscribe(List.of("test")); // Poll for records while (true) { var records = consumer.poll(Duration.ofMillis(100)); for (var record : records) { System.out.printf("Consumed record with key %s and value %s and partition %s%n", record.key(), record.value(), record.partition()); // Process the record } } } } |
Maven Dependency:
|
1 2 3 4 5 |
<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>4.1.1</version> </dependency> |
Acknowledgment Modes
Share Groups support two acknowledgment modes:
- Implicit Acks (Default): Automatic acknowledgment on the next poll() call
- Explicit Acks: Fine-grained control over which records are considered processed vs. failed
Explicit acknowledgment enables more control but requires additional code to manage acknowledgments.
CLI Consumer Example
You can also test Share Groups using the Kafka CLI:
|
1 |
./kafka-console-share-consumer.sh --consumer-config consumer.properties --bootstrap-server <bootstrap_server_ip>:9092 --topic test |
Multiple instances of this command will form a share group and distribute records among them, even if there are more consumers than partitions.
Example Consumer Configuration:
|
1 2 3 4 5 6 7 8 9 |
bootstrap.servers=<node_1_ip>:9092,<node_2_ip>:9092,<node_3_ip>:9092 key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer group.id=my-share-group security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="ickafka" \ password="<ickafka_password>"; |
Use Cases
Share Groups are designed for specific use cases where traditional consumer groups have limitations:
1. Scaling Consumers for Load Peaks
Problem: When producer rate increases (potentially temporarily), you may already have the maximum permitted consumers (equal to the number of partitions). With traditional consumer groups, you cannot add more consumers to handle the increased load.
Solution: Share Groups allow you to scale consumers beyond the number of partitions. You can dynamically increase the number of consumers during peak load and reduce them when load returns to normal.
Example: If you have 100 partitions and normally process 10,000 messages/second, but production rate spikes to 20,000 messages/second, you can add more consumers (e.g., 200 consumers) to handle the increased load without repartitioning.
2. MaintainingThroughput and Reduce Latency for Slow Consumers
Problem: When record processing time is high (due to complex processing or slow third-party systems), this slows down consumers, increases latency and lag, and reduces throughput.
Solution: Share Groups enable a larger pool of consumers (greater than the number of partitions) to handle separate messages, achieving target throughput and reducing end-to-end latency. Additionally, by using smaller batch sizes (even batch size = 1) and more consumers, you can minimize end-to-end processing latency
When to Use Share Groups vs. Consumer Groups
| Criteria | Share Groups | Traditional Consumer Groups |
| Order is critical |
✗ |
✓ |
| Scale consumers beyond number of partitions |
✓ |
✗ |
| Variable processing times / need to handle load spikes |
✓ |
✗ |
| Event-style application when events can be processed independently |
✓ |
✗ |
| Strict no-duplicates / no-drops required (exactly-once semantics) |
✗ |
✓ |
