Use Kafka with Java

In this example we will be using the official Java client maintained by the Apache Kafka team. A list of alternative Java clients can be found here.

Dependencies

Add the kafka_clients package to your application. This package is available in maven. Pass the version of Kafka running on your cluster as the version of the package. e.g. for a Kafka 2.8.1 cluster, add the dependency:

Using client ⇆ broker encryption (SSL)

If you have chosen to enable client ⇆ broker encryption, see here for information on the certificates required to establish an SSL connection to your cluster.

Producing Messages

Client Configuration

Before creating a Kafka producer client, you first need to define the configuration properties for the producer client to use. In this example we provide only the required properties for the producer client. See here for the full list of configuration options.

Create a new file named producer.properties, ensuring the password, truststore location, and bootstrap servers list are filled out and correct:

Note: Users can manually specify ssl.protocol and ssl.enabled.protocols to override the default values. See the official Kafka documentation for your version for more information on these parameters.

If your cluster does not have client ⇆ broker encryption enabled your file should instead look like this:

Make sure the password and bootstrap servers list are correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Java Code

Now that we have a properties file we can create a Kafka producer.

First, load the properties:

Before we can send a message we first need to create a ProducerRecord object:

Now we can create the producer itself, and send it to the cluster:

Note: We use the producer’s flush method here to ensure the message gets sent before the program exits. In normal operation the producer will send messages in batches when it has either accumulated a certain number of messages, or has waited a certain amount of time.

Consuming Messages

Client Configuration

As in the producer example, before creating a Kafka consumer client, you first need to define the configuration properties for the consumer client to use. In this example we provide only the required properties for the consumer client. See here for the full list of configuration options.

Create a new file named consumer.properties, ensuring the password, truststore location, and bootstrap servers list are filled out and correct:

Note: Users can manually specify ssl.protocol and ssl.enabled.protocols to override the default values. See the official Kafka documentation for your version for more information on these parameters.

If your cluster does not have client ⇆ broker encryption enabled your file should instead look like this:

Make sure the password and bootstrap servers list are correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Java Code

Now that we have a properties file we can create a Kafka producer.

First, load the properties:

Once we’ve setup the consumer properties, we can create the consumer itself. We subscribe the consumer to the topics we wish to receive messages from, and then use the consumer’s poll method to consume a single batch of messages. Combined with a loop, we can continually consume messages from Kafka as they are produced:

Putting Them Together

Now that we have a consumer and producer set up, it’s time to combine them.

Start the consumer

Start the consumer before starting the producer because by default consumers only consume messages that were produced after the consumer started.

Start the producer

Now that the consumer is setup and ready to consume messages, you can now start your producer.

If the consumer and producer are setup correctly the consumer should output the message sent by the producer shortly after it was produced:

Need Support
Learn More

Already have an account?
Login to the Console

Experiencing difficulties on the website or console?
Status page for known incidents


Don’t have an account yet?
Sign up for a free trial

Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console.