Kafka Diskless refers to a new architecture where Kafka brokers no longer store data on local disks. Instead, they act as intermediaries, processing data that is stored in external object storage. This approach simplifies operations, improves scalability, and reduces costs.

KIP 1150 is a Kafka Improvement Proposal, one of three competing ones, that talked about “Kafka Diskless.” KIP-1150 has been accepted by the Kafka community as the design choice for Kafka’s future implementation for diskless. It redefines Kafka’s architecture by decoupling storage from compute, enabling Kafka brokers to operate without relying on local disk storage. Instead, data is stored in external object storage systems like Amazon S3 or Google Cloud Storage.

Diskless Kafka is best suited for workloads prioritizing cost efficiency and scalability over ultra-low latency, such as analytics pipelines, event ingestion, and batch-oriented consumers.

The key components include:

• Kafka Brokers: Handle data processing and act as intermediaries.
• Object Storage Systems: Store data externally, ensuring durability and scalability.
• Producers and Consumers: Operate as usual, with no changes required to their configurations

Kafka Diskless is compatible with popular object storage systems such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. The specific implementation may vary depending on the configuration.

No, Kafka Diskless is designed to be backward-compatible. Existing Kafka clients (producers and consumers) can operate without modification, as the changes are implemented at the broker and storage levels.

While Diskless Kafka reduces costs by eliminating the need for replication at the Kafka layer and leveraging object storage durability the biggest cost-savings come come from it’s design which avoids inter–availability zone traffic, which for most Kafka workloads can be about 70%-80% of the total infrastructure costs.

Yes. Diskless Kafka may introduce higher latency compared to traditional Kafka deployments, as data must be written to remote storage before acknowledgement. Once Kafka’s diskless implementation is available, we should have more clarity on the precise performance impact. But, it is expected that diskless will be best suited for workloads that can tolerate higher latency.

Kafka Diskless is currently being evaluated and tested internally. It will be made available on the Instaclustr platform after upstream Kafka support matures and meets production requirements.

Migration approaches will depend on the final upstream implementation. We expect options for standing up new diskless clusters and migrating workloads, rather than in-place upgrades.