Instaclustr is pleased to announce the public preview release of Apache Kafka version 3.0.0 on the Instaclustr Managed Platform. The new version is available paired with version 3.0.0 of Kafka Connect. Please note that our preview release, though functionally complete, is governed by different SLA terms in contrast to our GA (Generally Available) release which is covered by our default SLA. For specifics on the applicable SLA, please refer to our webpage here.
Apache Kafka is the leading distributed event streaming platform and is open source under the Apache 2.0 License. Instaclustr provides our customers managed and supported open source software allowing them to avoid the risks of both vendor lock-in and unsupported software with open source that’s really open source.
Compared to the previous release, version 3.0.0 of Apache Kafka includes some new features, deprecations and removal of previously deprecated items, improvements to KRaft (Apache Kafka’s inbuilt component meant to obviate the need for Apache ZooKeeper in the future). It’s also important to be aware that our preview of Kafka 3.0.0 is still leveraging Apache ZooKeeper, in line with the project’s recommendations, as KRaft is not yet suitable for production workloads. However, we will be making available a preview of ZooKeeper-less Kafka in early 2022.
Following are some highlights from the changes contained in Kafka 3.0.0:
Kafka
- Apache ZooKeeper upgraded to version 3.6.3:
This version upgrade follows on from ZooKeeper version 3.5.9, which was included with Apache Kafka 2.7.1 and later. This release contains many bug fixes (including CVE fixes) and some new features. For more details, please refer to the Apache ZooKeeper 3.6.3 Release Notes.
- Producer will enable strongest delivery guarantee by default (KIP-679):
This change modifies certain defaults (enable.idempotence
is set to true; acks
is set to all
) in the Producer config, along with some other modifications, to make record delivery guarantees stronger by default. As modification to the default for enable.idempotence
could cause compatibility issues for some users, it is advised you go through the migration plan outlined in KIP-679.
- Increase default consumer session timeout (KIP-735):
As part of this change, the default consumer session timeout has been increased from 10s to 45s to reduce the risk of spurious failures causing unnecessary rebalances to be triggered. Spurious failures can occur, for example, due to transient network/load failures. Such situations can lead to two rebalances being triggered: once when a group member failure is detected and later when that member rejoins the group. With the new session timeout, the member should have sufficient time to reconnect with the coordinator.
- Change Kafka streams default replication factor config (KIP-733):
The default replication.factor=1
puts the onus on users to change the config when application moves to production (as best practice recommends RF=3). For Kafka versions higher than 2.4.0, users can look to change config to -1 ( to pick the broker default for replication factor), however starting streams 3.0.0 this is no longer needed as the streams replication factor config will now be -1 by default.
- Some performance related improvements:
- Update
FindCoordinator
to resolve multiple Coordinators at a time (KIP-699):
Currently, to look up the coordinator for a resource, theFindCoordinator
API can be used. To do this when multiple consumer groups are involved, theFindCoordinato
r would need to be called once for each consumer group, which can negatively affect performance. With KIP-699,FindCoordinator
has been renamed toFindCoordinators
, and now allows multiple such requests to be clubbed into one call, with the result containing a corresponding number of coordinators.
- Extend
OffsetFetch
requests to accept multiple consumer group IDs (KIP-709):
This change extends the functionality of theOffsetFetch
API to be able to fetch offsets for multiple consumer groups. This was previously not the case asOffsetFetch
could only support fetching the offset for one consumer group at a time.
- Update
- Significant deprecations:
- API deprecations that were removed:
- Remove deprecated
--zookeeper
in topicCommands (KAFKA-12596) - Remove deprecated
--zookeeper
in ReassignPartitionsCommands (KAFKA-12597) - Remove deprecated
--zookeeper
in ConfigCommand (KAFKA-12598) - Remove deprecated
kafka-preferred-replica-election
command (KAFKA-8405) - Remove deprecated
SimpleAclAuthorizer
(KAFKA-12590)
- Remove deprecated
Kafka Connect
- Connector API to restart connector and tasks (KIP-745):
Currently, if a Connector or Task instance started by Kafka Connect experiences an error, it is marked as failed and exposed as such through the Connect REST API. To restart them, another REST API must be called once for each such instance. This change enables a single call to the Connect REST API to restart all failed instances.
- Removal of Connect’s internal converter properties (KIP-738):
The internal.key.converter
and internal.value.converter
Connect worker properties, and other properties prefixed with those names were deprecated with Kafka 2.0.0. Now, with Kafka 3.0.0, they’ve been removed and instead JsonConverter will be used with both, key and value converters. Please follow the migration plan provided on the KIP-738 page, if you’re impacted by this change.
If you would like to try our Kafka or Kafka Connect service, log in to our Management Console, create a test Kafka Connect cluster, and associate it with your Instaclustr managed Kafka cluster. If you haven’t created a Kafka cluster on our platform, create a test Kafka cluster and use the 30-day free trial. If you’d also like a free trial of a Kafka Connect cluster, please contact our Sales team.