Kafka Combined with Instaclustr Products

Menu

This article contains two examples of integrating Instaclustr Kafka with Instaclustr Cassandra:

  • Kafka Connect with a Cassandra sink
  • Kafka Connect with a Cassandra Source and a file sink

Using client ⇆ broker encryption (SSL)

If you have chosen to enable client ⇆ broker encryption on your Kafka cluster, see here for information on the certificates required to establish an SSL connection to your Kafka cluster.

Kafka Connect with a Cassandra Sink

In this example we will be using Kafka Connect to store messages from an Instaclustr Kafka topic into an Instaclustr Cassandra table.

Prerequisites

The Apache Kafka package installation comes bundled with a number of Kafka tools. For this example we are going to use the connect-standalone.sh tool. To get this tool you will need to download and install a Kafka release from here. We will also be using a third party plugin to allow us to connect to Cassandra that can be found here. Once downloaded, open the archive and copy kafka-connect-cassandra-1.1.0-1.1.0-all.jar to a new directory named plugins somewhere on your computer. This example has been tested with Kafka 1.1.0.

Kafka Connect Configuration

Before we can use Kafka Connect we need to configure a number of things. Basic configuration requires the following configuration options. Make a file connect.properties with the following content:

Make sure to replace the bootstrap.servers with the IP address of at least one node in your cluster.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

In order to use Kafka Connect with Instaclustr Kafka we also need to provide authentication credentials. If your cluster has client ⇆ broker encryption enabled you will also need to provide encryption information. For more information on using certificates with Kafka and where to find them see here. Add the following to your connect.properties file, ensuring the password is correct:

If your cluster does not have client ⇆ broker encryption enabled, add the following to your connect.properties file:

If your cluster does have client ⇆ broker encryption enabled, add the following to your connect.properties file, ensuring the truststore location is correct:

Create Kafka Topic

For Kafka Connect to work, sources and sinks must refer to specific Kafka topics. Before we can run Kafka Connect we need to create a topic to be used for storing the messages produced by Kafka Connect. Use the guide here to create a new topic called orders-topic with a replication factor of 3.

Create Cassandra Table

For the Cassandra Sink plugin to work, a Cassandra table must be specified as the desired location to store output to. First, create a new key space for the table:

Next, create a new table in your Cassandra cluster called order_events with the following schema:

Cassandra Sink Configuration

Now that Kafka Connect is configured, we need to configure the sink for our data. In this case we will be using the Cassandra Sink from the stream reactor project. The Cassandra Sink will insert all records from a Kafka topic into a specified Cassandra table. Make a file cassandra-sink.properties with the following content:

Make sure to replace the contact points, username, and password if necessary. If your Cassandra cluster requires encryption, add the following to your cassandra-sink.properties, ensuring the path to the trust store is correct:

Note: you can download a zip containing the truststore.jks file from the Connection Info page of your Cassandra cluster.

Start Kafka Connect

Now that all the Kafka Connect components in this example are configured, we can start Kafka Connect from the command line like so:

Test Kafka Connect

Once Kafka Connect has started, it’s time to test the configuration.

First, follow the guide here to setup a Kafka console producer, changing the topic name to orders-topic.

Once you’ve setup a console producer, send the message following message to Kafka:

After a short delay you should see the message you sent to Kafka appear as a record in the Cassandra table:

Kafka Connect with a Cassandra Source and a File Sink

In this example we will be using Kafka Connect to read new records from an Instaclustr Cassandra table and output them to a text file.

Prerequisites

The Apache Kafka package installation comes bundled with a number of Kafka tools. For this example we are going to use the connect-standalone.sh tool. To get this tool you will need to download and install a Kafka release from here. We will also be using a third party plugin to allow us to connect to Cassandra that can be found here. Once downloaded, open the archive and copy kafka-connect-cassandra-1.0.0-1.0.0-all.jar to a new directory named plugins somewhere on your computer.

Kafka Connect Configuration

Before we can use Kafka Connect we need to configure a number of things. Basic configuration requires the following configuration options. Make a file connect.properties with the following content:

Make sure to replace the bootstrap.servers with the IP address of at least one node in your cluster.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

In order to use Kafka Connect with Instaclustr Kafka we also need to provide authentication credentials. If your cluster has client ⇆ broker encryption enabled you will also need to provide encryption information. For more information on using certificates with Kafka and where to find them see here. Add the following to your connect.properties file, ensuring the password is correct:

If your cluster does not have client ⇆ broker encryption enabled, add the following to your connect.properties file:

If your cluster does have client ⇆ broker encryption enabled, add the following to your connect.properties file, ensuring the truststore location is correct:

Create Kafka Topic

For Kafka Connect to work, sources and sinks must refer to specific Kafka topics. Before we can run Kafka Connect we need to create a topic to be used for storing the messages produced by Kafka Connect. Use the guide here to create a new topic called orders-topic with a replication factor of 3.

Create Cassandra Table

For the Cassandra Sink plugin to work, a Cassandra table must be specified as the desired location to store output to. If necessary, create a new key space for the table:

If necessary create a new table in your Cassandra cluster called order_events with the following schema:

Cassandra Source Configuration

Now that Kafka Connect is configured, we need to configure the sink for our data. In this case we will be using the Cassandra Source from the stream reactor project. The Cassandra Source will insert all records from a Kafka topic into a specified Cassandra table. Make a file cassandra-source.properties with the following content:

Make sure to replace the contact points, username, and password if necessary. If your Cassandra cluster requires encryption, add the following to your cassandra-source.properties, ensuring the path to the trust store is correct:

Note: you can download a zip containing the truststore.jks file from the Connection Info page of your Cassandra cluster.

File Sink Configuration

Now that our data source is configured, we need to configure the sink for our data. In this case we will be using the Connect File Sink that is provided with Apache Kafka. The Connect File Sink will pipe all changes in a Kafka topic to a file. Make a file connect-file-sink.properties with the following content:

Make sure to replace the file option with the path to the desired input file.

Start Kafka Connect

Now that all the Kafka Connect components in this example are configured, we can start Kafka Connect from the command line like so:

Test Kafka Connect

Once Kafka Connect has started, it’s time to test the configuration.

Insert some new rows into your Cassandra order_events table:

After some delay you should see the rows you inserted into your Cassandra table appear in your output text file:

FREE TRIAL

Spin up a cluster in less
than 5 minutes.
(No credit card required)

Sign Up Now
Close

Site by Swell Design Group