Cluster Config Backup

Kafka is a highly distributed system and can be configured to provide a high level of resilience on its own. Kafka’s replication factor can help survive the loss of several brokers and when it is stretched across multiple datacenters, we can survive the loss of data centers as well.

In addition, we backup important Kafka configurations so that it can be restored to a new Kafka cluster in the case of total Zookeeper failure. The backup will be a tar file of the Zookeeper snapshot and transaction directories which is all that is required to restore a Zookeeper ensemble. We backup the entire Zookeeper data directory which contains topics, their configuration, users, acls and passwords. The backup runs every 24 hours and stores it on Amazon S3 storage.

For a Kafka cluster with KRaft mode the configuration is stored as a Kafka topic and replicated across the KRaft controller nodes in the cluster, ensuring high availability and tolerance to node failure while providing the same level of resilience as that of the Kafka cluster. Given this, we do not back up KRaft configuration in a separate directory.

If and when you want to restore the configuration to a new Kafka cluster, please send an email to our support team.