Setting up Mirror Maker

Menu

Mirror Maker is a feature included in Kafka which allows for maintaining a replica of a Kafka cluster in a separate data centre. It uses the existing consumer and producer APIs to achieve this.

Prerequisites

The Apache Kafka package installation comes bundled with a number of helpful command line tools to communicate with Kafka in various ways. To get these tools you will need to download and install a Kafka release from here.

Using client ⇆ broker encryption (SSL)

If you have chosen to enable client ⇆ broker encryption, see here for information on the certificates required to establish an SSL connection to your cluster.

Setting up Mirror Maker

This example demonstrates how to set up a standalone Mirror Maker instance. To create a distributed Mirror Maker cluster, simply repeat the following steps on any number of machines. As long as the config files are the same, the Kafka clusters will ensure the load is balanced and fault tolerant across the Mirror Maker instances.

Consumer Configuration

The consumer is the part of Mirror Maker which reads data from the source cluster it is intending to replicate.

Create a file named consumer.properties with the following content:

If your cluster does not have client ⇆ broker encryption enabled, your file should instead look like this:

Make sure the bootstrap.server IPs, truststore location if using SSL, and password are correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Producer Configuration

The producer is the part of Mirror Maker that uses the data read by the and replicates it to the destination cluster.

Create a file named producer.properties with the following content:

If your cluster does not have client ⇆ broker encryption enabled, your file should instead look like this:

Make sure the bootstrap.server IPs, truststore location if using SSL, and password are correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Running Mirror Maker

Now that we have configured Mirror Maker’s producers and consumers, we are almost ready to run Mirror Maker.

Tweak Kafka memory options with the following command as the defaults are not enough to accomodate Mirror Maker:

Run mirror maker using the script found in the bin directory of your Kafka installation:

The “–whitelist” parameter configures which topics are mirrored. It can be a comma separated list of topic names or a Java regular expression. The value “.*”  copies all topics on the source cluster to the destination cluster. The “exclude.internal.topics=true” in consumer.properties ensures Mirror Maker does not copy any of Kafka’s internal topics. A blacklist option is also available.

The “–num.streams” option determines the number of consumers Mirror Maker should use. The more consumers you use the less time it takes for data to be replicated from the source cluster to the destination cluster.

Once Mirror Maker is running it will not output anything. The best way to test is to consume from the destination cluster to see if messages are being replicated.

Best Practices

It is a good idea to create any topics that are being mirrored on the destination cluster before starting Mirror Maker. Mirror Maker can create the topics automatically but they may not retain the exact same configuration as the originals.

If you are running multiple Mirror Maker instances, it is recommended that all instances share the same group-id in the consumer properties.

Site by Swell Design Group