In-place Resizing for Apache Kafka

Nodes in a Kafka cluster can be resized in-place via the Console or by issuing a request to our Provisioning API. Moving to a new node size lets you scale the CPU core count, memory quotas as well as increase disk capacity to meet changing demands. You may resize a subset of nodes that match a purpose e.g. resize only the broker nodes in your cluster.

Preparation

Instaclustr recommends that you first test your applications against in place resizing in a non-production setting to verify that the resize process doesn’t interrupt your applications.

During a cluster resize operation one or more nodes will temporarily become unavailable as they are resized (for resizing involving more than a disk resize). Kafka, as a whole, will handle these transient outages, but client applications may need to be programmed with certain considerations in mind.

Instaclustr will only concurrently resize nodes within the same rack, to ensure that replicas stored in other racks remain available during the resize.

Topic Requirements

The cluster should have topics with a replication factor greater than one, and this replication factor should not equal to the min.insync.replicas config of that topic. 

Other Constraints

• Disks can be increased but not decrease.
• Resizing only works with single disk node sizes.
• The nodes selected for resize must all be the same size.
• AWS:
  • Current in place resizing is only supported on AWS Nodes which are EBS backed.
  • Resizing where the disk is modified can only be done once every 6 hours due to EBS limitations.
• GCP:
  • Current in place resizing is only supported on a GCP Node which has a Persistent disk. (Persistent disks are referred to as ‘Zonal PD SSD’ on our console, an abbreviation for ‘Zonal Persistent Disk SSD’.)
• AZURE_AZ:
  • Current in place resizing is only supported on a AZURE_AZ Node which has a Premium SSD disk.
• AZURE:
  • Not Supported.

Producer Requirements

It is recommended to produce with the “acknowledgement” setting to “all” to ensure no data would be lost.

Clients Settings

We suggest using all brokers that are provided for connection configuration. 

Resizing

To resize data centre nodes, navigate to the Resize page for your Kafka cluster.

You will be able to select which nodes you want to resize by selecting the Node Purpose from those available to your cluster. You can resize Broker, Dedicated ZooKeeper, or Dedicated KRaft Controller nodes. You may only resize one node purpose at a time and will need to perform multiple resize operations to resize multiple node purposes.

The Concurrency Factor is a number, from 1 to the count of nodes in the largest rack, that specifies how many nodes may be resized at the same time. To ensure the data’s availability within the cluster, Kafka cluster data centres are only allowed to be resized one node at a time. Any resize requests with a concurrency factor greater than one will not be processed.

Finally, select a desired target node size that you wish to resize to.

Once you have selected a new size, you will be shown a summary of the resize, which will include the new price for managing your cluster. Click on Resize Data Centre to begin the resize process.

You can also select to notify your accounts’ designated support contacts via email once the resize is complete. You may opt-out of this notification by deselecting the Notify this account’s designated support contacts on resize completion checkbox.

If the selected data centre isn’t resizable then an explanation will be displayed on the Resize Data Centre page.

While a resize is in progress, the Cluster Details page will display the progress of the resize. You will see individual nodes switch from Running to Pending → Provisioning/Provisioned and then back to Running once their size has been changed.

In some rare cases, it may be desirable to cancel a resize operation while it is in progress. This can be achieved via the API. This may leave the cluster in a non-uniform state with some nodes already resized. Please contact our support team to assist in this case, to ensure your cluster is correctly sized.