NetApp announces that Instaclustr for Apache Kafka now offers Tiered Storage for new clusters created on Google Cloud Platform (GCP). This feature gives customers a powerful way to optimize storage costs and improve scalability for their Kafka workloads. 

What is Tiered Storage in Kafka and why it matters 

Tiered Storage allows Kafka to offload older log segments from local disk to cost-effective cloud object storage, such as Google Cloud Storage (GCS), without sacrificing the ability to query and utilize this data in real-time streaming applications. And all of this can be achieved without any changes to your existing Kafka clients. 

With Tiered Storage, you benefit from: 

  • Reduce infrastructure costs. Move infrequently accessed data to more cost-effective storage. Additionally, data that would have previously been replicated across multiple brokers for fault tolerance is instead moved to remote storage. This leads to a reduction in storage and network (during cluster topology changes) costs compared to storing data on local storage, allowing you to choose whether to reduce your total cost of ownership, or increase retention periods, or a bit of both. 
  • Simplify cluster scaling.  By separating the storage layer from the Kafka brokers, Tiered Storage allows for independent scaling of storage and compute resources. For example, as data volumes grow, the storage can be expanded more economically by adding more capacity in the tiered storage layer rather than scaling up the broker storage or adding more brokers to the cluster.   
  • Extend retention periods for compliance or analytics. You can now keep data indefinitely without worrying about the physical limitations of the local storage attached to your Kafka cluster by having data moved to limitless remote storage.   

For a deeper dive into the concept, check out our Kafka Tiered Storage overview. 

How it works on Instaclustr for Apache Kafka 

When Tiered Storage is enabled on your Instaclustr managed Kafka cluster: 

  • Active segments remain on local disk for fast access 
  • Older segments are automatically offloaded to Google Cloud Storage (GCS) 
  • Kafka clients continue to consume data seamlessly, regardless of where it resides 

For detailed steps, on setting up GCS as the remote storage and how to enable and use Tiered Storage with an Instaclustr for Apache Kafka cluster, visit our Using Kafka Tiered Storage guide. 

Get started 

Tiered Storage for Apache Kafka on GCP is available in public preview for NetApp Instaclustr customers. It is offered as an Enterprise Add-On but is included in Instaclustr Enterprise Pricing. Once Enterprise Pricing is applied to your cluster, all enterprise features are available to you at no extra cost. Please contact your Customer Success Representative for more details around pricing applicable to you. 

For our Managed Platform customers creating new clusters with Kafka 3.9.1 and greater, they will have the option to enable Tiered Storage during the setup process. For customers who already have existing Kafka clusters running version 3.9.1 or greater, please open a ticket with our Support team providing details of the cluster you would like this feature enabled on. We will have this available in the coming weeks. 

Please note that during this preview stage, SLAs are not applicable and there may be substantial changes prior to GA release. Please contact the Instaclustr Sales team if you would like to explore using the preview release for production usage. We’re continuing work to make sure we can make this feature GA over the next couple of months. 

Stay tuned for more updates as we continue to enhance Kafka on Instaclustr with features that improve performance, scalability, and cost efficiency.