Cluster Config Backup

Kafka is a highly distributed system and can be configured to provide a high level of resilience on its own. Kafka’s replication factor can help survive the loss of several brokers and when it is stretched across multiple datacenters, we can survive the loss of data centers as well.

In addition, we backup important Kafka configurations so that it can be restored to a new Kafka cluster in the case of total Zookeeper failure. The backup will be a tar file of the Zookeeper snapshot and transaction directories which is all that is required to restore a Zookeeper ensemble. We backup the entire Zookeeper data directory which contains topics, their configuration, users, acls and passwords. The backup runs every 24 hours and stores it on Amazon S3 storage.

For a Kafka cluster with KRaft, we backup the entire KRaft cluster metadata directory which contains topics, their configuration, users, acls and passwords. The backup occurs every 24 hours and stores the data in cloud storage, depending on the cluster’s provider: Amazon S3 for AWS, Azure Blob Storage for Azure, and Google Cloud Storage for GCP.

If and when you want to restore the configuration to a new Kafka cluster, please send an email to our support team.