In-place Resizing for Kafka
Nodes in a Kafka cluster can be resized in-place via the Console or by issuing a request to our Provisioning API. Moving to a new node size lets you scale the CPU core count, memory quotas as well as increase disk capacity to meet changing demands. You may resize a subset of nodes that match a purpose e.g. resize only the broker nodes in your cluster.
Instaclustr recommends that you first test your applications against in place resizing in a non-production setting to verify that the resize process doesn’t interrupt your applications.
During a cluster resize operation one or more nodes will temporarily become unavailable as they are resized (for resizing involving more than a disk resize). Kafka, as a whole, will handle these transient outages, but client applications may need to be programmed with certain considerations in mind.
Instaclustr will only concurrently resize nodes within the same rack, to ensure that replicas stored in other racks remain available during the resize.
The cluster should have topics with a replication factor greater than one, and this replication factor should not equal to the min.insync.replicas config of that topic.
- Current in place resizing is only supported on AWS Nodes which are EBS backed.
- Disks can be increased but not decreased
- Resizing where the disk is modified can only be done once every 6 hours due to EBS limitations
- Resizing only works with single disk node sizes.
- The nodes selected for resize must all be the same size.
It is recommended to produce with the “acknowledgement” setting to “all” to ensure no data would be lost.
We suggest using all brokers that are provided for connection configuration.
To in-place resize a data centre click the Resize… button on the Cluster Details page for the cluster you wish to resize nodes in.
You will be able to select which nodes you want to resize by selecting the node purpose from those available to your cluster. You can resize Broker or Dedicated ZooKeeper nodes. Resizing both Broker and Dedicated ZooKeeper nodes needs to be conducted in 2 separate operations.
The Resize Data Centre page lets you select the desired target node size from within the size class and specify the resize concurrency factor and notification options. The difference in CPU core count, memory and disk quota and total monthly price are listed on the page.
The Concurrency Factor is a number, from 1 to the count of nodes in the largest rack, that specifies how many nodes may be resized at the same time. To ensure the data’s availability within the cluster, Kafka cluster data centres are only allowed to be resized one node at a time. Any resize requests with a concurrency factor greater than one will not be processed.
By default Instaclustr will send your accounts’ designated support contacts an email once the resize is complete. You may opt-out of this notification by deselecting the Notify this accounts’ designated support contacts on resize completion checkbox.
If the selected data centre isn’t resizable then an explanation will be displayed on the Resize Data Centre page.
While a resize is in progress, the Cluster Details page will display the progress of the resize. You will see individual nodes switch from Running to Pending → Provisioning/Provisioned and then back to Running once their size has been changed.
In some rare cases, it may be desirable to cancel a resize operation while it is in progress. This can be achieved via the API. This may leave the cluster in a non-uniform state with some nodes already resized. Please contact our support team to assist in this case, to ensure your cluster is correctly sized.