Configuring the Keystore to Use Client ⇄ Broker Encryption & Mutual Authentication (mTLS) For Apache Kafka Client

Overview 

This article describes how to configure a local keystore to enable Apache Kafka clients to connect to the cluster with Client ⇄ Broker Encryption & Mutual Authentication (mTLS) and a connection example. 

Prerequisites 

• You must have a Kafka cluster with mTLS enabled – see documentation for more help with this. 
• You must have the signed user certificates available – see documentation for more help with this. 
• You must allow the client IP to connect to the mTLS ports through your cluster firewall – see documentation for more help with this. 
• You have keytool, a key and certificate management utility which is bundled with the Java Development Kit (JDK). For more information refer to the keytool documentation. Another tool or service could be used instead as many options are available. 

Configuring the Keystore to Use mTLS Authentication with Apache Kafka Clients 

1. Add the cluster CA X.509 certificate to the keystore. Ensure it is the same the keystore used while creating the certificate signing request. An example code snippet for this using keytool in a terminal is as follows
   
   Add Instaclustr RootCA to the keystore

   Shell

   keytool -keystore <path to keystore, e.g. keystore.jks> -alias <alias e.g CAroot> -import -noprompt -file \ <e.g. cluster-ca-certificate.pem> -storepass <store password used at setup> -keypass <key password used at setup>

   1 2	keytool -keystore <path to keystore, e.g. keystore.jks> -alias <alias e.g CAroot> -import -noprompt -file \ <e.g. cluster-ca-certificate.pem> -storepass <store password used at setup> -keypass <key password used at setup>

   
   For example:
   
   
   Shell

   keytool -keystore keystore.jks -alias CAroot \ -import -noprompt -file cluster-ca-certificate.pem \ -storepass yourkeystorepassword -keypass yourkeystorepassword

   1 2 3

   keytool -keystore keystore.jks -alias CAroot \ -import -noprompt -file cluster-ca-certificate.pem \ -storepass yourkeystorepassword -keypass yourkeystorepassword

   
   For more information on where to find the CA X509 certificate, refer to the instructions here. 
2. For instructions on how to generate the signed certificate, please refer to the instructions here. Add the signed client certificate to the keystore. An example code snippet for this using keytool in a terminal is as follows
   
   Shell

   keytool -keystore <path to keystore, e.g. keystore.jks> -alias <alias, e.g. mtls> -import -noprompt -file \ <path to signed certificate, e.g. mtls-ca-signed.crt> -storepass <store password used at setup> -keypass <key password used at setup>

   1 2	keytool -keystore <path to keystore, e.g. keystore.jks> -alias <alias, e.g. mtls> -import -noprompt -file \ <path to signed certificate, e.g. mtls-ca-signed.crt> -storepass <store password used at setup> -keypass <key password used at setup>

   
   For example:
   
   
   Shell

   keytool -keystore keystore.jks -alias ickafka \ -import -noprompt -file ickafka-ca-signed.crt \ -storepass yourkeystorepassword -keypass yourkeystorepassword

   1 2 3

   keytool -keystore keystore.jks -alias ickafka \ -import -noprompt -file ickafka-ca-signed.crt \ -storepass yourkeystorepassword -keypass yourkeystorepassword

   
   For more information on where to find the CA X509 certificate, refer to the instructions here. 

Example Connection 

Now that you have the keystore set up to connect to your Kafka cluster, a range of connection examples are available on the Connection Information page using different connection methods. As an example, the following steps show how to connect a consumer and producer using the Kafka CLI.

1. Download and unzip a copy of Kafka. Kafka can be downloaded from https://kafka.apache.org/downloads and choose appropriate version.  
2. Create a configuration file (e.g. kafka.properties) containing the following properties:
   
   Example Config properties

   security.protocol=SSL ssl.truststore.location=<path to trustore, e.g. truststore.jks> ssl.truststore.password=<trustrore password> ssl.keystore.location=<path to keystore, e.g. keystore.jks> ssl.keystore.password=<keystore password> ssl.key.password=<key password used at setup>

   1 2 3 4 5 6

   security.protocol=SSL ssl.truststore.location=<path to trustore, e.g. truststore.jks> ssl.truststore.password=<trustrore password> ssl.keystore.location=<path to keystore, e.g. keystore.jks> ssl.keystore.password=<keystore password> ssl.key.password=<key password used at setup>

3. Start a console producer using the following command 
   
   Start a kafka producer

   Shell

   kafka-console-producer.sh --broker-list <ip, e.g.3.220.172.4>:<port, e.g., 9082> \ --topic test-topic --producer.config \ <path to producer config, e.g. kafka.properties>

   1 2 3

   kafka-console-producer.sh --broker-list <ip, e.g.3.220.172.4>:<port, e.g., 9082> \ --topic test-topic --producer.config \ <path to producer config, e.g. kafka.properties>

4. In a separate terminal, start a consumer with the following command
   
   Run a kafka consumer

   Shell

   kafka-console-consumer.sh --bootstrap-server <ip, e.g.3.220.172.4>:<port, e.g. 9082> --topic <topic>\ --consumer.config <path to consumer config, e.g. kafka.properties>

   1 2	kafka-console-consumer.sh --bootstrap-server <ip, e.g.3.220.172.4>:<port, e.g. 9082> --topic <topic>\ --consumer.config <path to consumer config, e.g. kafka.properties>

5. Type “instaclustr” in the producer terminal and press Enter. Confirm that “instaclustr” is received in the consumer terminal. 

Additional Resources 

Refer to the following resources for further information on mTLS with Apache Kafka: 

• Useful information on MTLS with Apache Kafka 
• Creating an Apache Kafka cluster with mTLS authentication 
• Managing certificates on mTLS enabled Apache Kafka clusters 
• Converting Keystore (JKS) to PEM file