Talk to an Expert Try for free
Free trial

Search

Kafka® Diskless:
Redefining Data Streaming Architecture

Kafka® Diskless:
Redefining Data Streaming Architecture

Kafka Diskless (KIP-1150) introduces a new architecture for Apache Kafka® and at NetApp, we’re actively working to bring this capability to our managed platform.

By separating storage from compute, diskless Kafka has the potential to significantly reduce infrastructure costs and simplify operations for selected workloads.

Kafka product hero image texture

How Kafka Diskless Works: A Deep Dive into KIP 1150

In the ever-evolving world of data streaming, Kafka Diskless is an emerging paradigm shift. By decoupling storage from compute, this innovation addresses scalability, cost-efficiency, and operational simplicity.

Kafka without Diskless

Kafka without Diskless diagram

Kafka Diskless

Kafka Diskless diagram

Kafka Diskless, as proposed in KIP 1150, reimagines the traditional Kafka architecture by eliminating the dependency on local disk storage, making brokers effectively stateless. This approach leverages external object storage systems, enabling a clean separation of compute and storage layers while simplifying scaling and failure recovery.Note though that Kafka Diskless is still evolving. Many operational and performance characteristics are still being discussed in the Apache Kafka community.

Track updates for KIP 1150­­

Core Mechanics of Kafka Diskless

icon scale
Separation of Storage and Compute
Kafka brokers no longer store data locally. Instead, they act as intermediaries, fetching and processing data from external storage systems like S3 or GCS.
icon reduced complexity
Object Storage Integration
Data is written directly to object storage, ensuring durability and scalability without the need for local disk management.
application icon
Optimized Data Flow
By offloading storage responsibilities, Kafka brokers focus solely on processing, improving scalability and elasticity.

Why Kafka Diskless Matters:
Key Benefits for Engineers and Architects

Kafka Diskless offers a host of advantages that address the challenges of traditional Kafka deployments. Here’s why platform engineers and software architects should take note:

Technical Benefits

  • Scalability: Seamlessly scale storage independently of compute, accommodating growing data volumes without over-provisioning.
  • Performance Optimization: Optimised architecture can improve scalability and elasticity.
  • Simplified Operations: Eliminate the need for disk management, reducing operational complexity and failure points.

Operational Advantages

  • Cost Efficiency: Pay only for the storage and compute you use, optimizing resource allocation.
  • Cloud-Native Compatibility: Leverage cloud object storage for enhanced durability and availability.
  • Future-Proof Architecture: Align with modern trends in distributed systems design, ensuring long-term viability.

The Evolution of Streaming

At NetApp Instaclustr, we believe Kafka Diskless represents a pivotal moment in the evolution of data streaming. We are actively evaluating Kafka Diskless through internal previews, benchmarking, and collaboration with the Kafka community. Our goal is to make this capability available on the Instaclustr platform as soon as it meets the reliability and performance expectations of production workloads.

Track updates for KIP 1150

Why We Support Kafka Diskless

icon falling prices
Significant Customer Cost-Savings
Our clients benefit from reduced costs and improved system performance.
icon magnifying glass
Alignment with Industry Trends
The shift towards decoupled architectures mirrors the broader movement in cloud-native technologies.
icon system star
Commitment to Innovations
As leaders in managed Kafka services, we are committed to supporting cutting-edge advancements like KIP 1150.

Dive Deeper into Kafka Diskless

Blog
Visual Guide to Apache Kafka Diskless Topics & Cloud Costs
Explore the new Apache Kafka Diskless Topics proposal. Learn about the architecture and how it reduces cloud costs by eliminating...
Blog
When (and when not) to use Apache Kafka® Diskless Topics
Discover the best use cases for Kafka Diskless Topics. Learn how this diskless architecture impacts latency, slow producers, and...
Documentation
Kafka “Diskless” – Proposals, Status & Insights

FAQs

  • What does "Kafka Diskless" mean? +

    Kafka Diskless refers to a new architecture where Kafka brokers no longer store data on local disks. Instead, they act as intermediaries, processing data that is stored in external object storage. This approach simplifies operations, improves scalability, and reduces costs.

  • What is KIP 1150? +

    KIP-1150 is the Apache Kafka Improvement Proposal for “Diskless Topics.” The community accepted it in March 2026 as the agreed direction for a new topic type that stores durable data in object storage instead of broker-local disks. KIP-1150 defines the vision and requirements; implementation is being specified in follow-on KIPs, including KIP-1163 (produce/consume paths) and KIP-1164 (diskless coordinator), which remain under discussion.

  • What workloads are best suited for Kafka Diskless? +

    Diskless Kafka is best suited for workloads prioritizing cost efficiency and scalability over ultra-low latency, such as analytics pipelines, event ingestion, and batch-oriented consumers.

  • What are the key components of Kafka Diskless? +

    The key components include:

    • Kafka Brokers: Handle data processing and act as intermediaries.
    • Object Storage Systems: Store data externally, ensuring durability and scalability.
    • Producers and Consumers: Operate as usual, with no changes required to their configurations
  • What object storage systems are supported by Kafka Diskless? +

    Kafka Diskless is compatible with popular object storage systems such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. The specific implementation may vary depending on the configuration.

  • Does Kafka Diskless require changes to existing Kafka clients? +

    No, Kafka Diskless is designed to be backward-compatible. Existing Kafka clients (producers and consumers) can operate without modification, as the changes are implemented at the broker and storage levels.

  • How does Kafka diskless reduce costs? +

    While Diskless Kafka reduces costs by eliminating the need for replication at the Kafka layer and leveraging object storage durability the biggest cost-savings come from its design which avoids inter–availability zone traffic, which for most Kafka workloads can be about 70%-80% of the total infrastructure costs.

  • Are there performance trade-offs with Kafka Diskless? +

    Yes. Diskless Kafka may introduce higher latency compared to traditional Kafka deployments, as data must be written to remote storage before acknowledgement. Once Kafka’s diskless implementation is available, we should have more clarity on the precise performance impact. But, it is expected that diskless will be best suited for workloads that can tolerate higher latency.

  • Is Kafka Diskless available today on Instaclustr? +

    Kafka Diskless is currently being evaluated and tested internally. It will be made available on the Instaclustr platform after upstream Kafka support matures and meets production requirements.

  • Will I need to migrate existing clusters? +

    Migration approaches will depend on the final upstream implementation. We expect options for standing up new diskless clusters and migrating workloads, rather than in-place upgrades.

Prepare for
Kafka Diskless
Today

Kafka Diskless is more than just an architectural shift—
it’s a game-changer for data streaming. Whether you’re looking to optimize your current Kafka deployment or explore the possibilities of a diskless future, we’re here to help.

Book a consultation with our Kafka experts
Contact us