Managing Certificates on a Client ⇄ Broker Encryption & Mutual Authentication (mTLS) Enabled Apache Kafka® Cluster

Overview

This article describes how to manage certificates for a Apache Kafka Cluster with Client ⇄ Broker Encryption & Mutual Authentication (mTLS).

Prerequisites

You need to have a running Instaclustr Apache Kafka cluster with Client ⇄ Broker Encryption & Mutual Authentication (mTLS) enabled. Please refer to creating an Apache Kafka cluster with mTLS authentication for guidance if you require a new Kafka cluster.
This guide uses keytool – a key and certificate management utility which is bundled with the Java Development Kit (JDK). More information can be found on the keytool documentation. Another tool or service can be used instead, and many options are available.

Managing Client Certificates Using Instaclustr Console

mTLS Kafka clients require a Kafka user to connect to a Kafka cluster as well as a certificate to authenticate the connection.

You need to generate a Certificate Signing Request (CSR) for the user and upload it to console for the server. The server will generate a signed user certificate using the Instaclustr Certificate Authority (CA).

This signing process allows the server to recognise the Kafka client’s certificate during the authenticating handshake for the user and is fundamental to the mutual aspect of mTLS.

Add a User for mTLS

Click on the created cluster and enter the Users page.
Click the Add New User button to add a new user.
Complete the form fields with the desired username, password, and permissions. For example, this guide shows how to create a user with the username “mtls” and standard permissions. You need to choose a Kafka User Authentication Mechanism but that will not be used for an mTLS connection. Instead, the signed user certificate with TLS will be used.
Click Add User.
Refer to Apache Kafka User Management for additional information on creating users and permission options. The username should match the Common Name (CN) provided while generating the CSR in the following section.

Generate a Certificate Signing Request

Generate a keystore for the client and then generate a CSR. We will use keytool for this guide. However, another tool or service could be used for keystore management.

Example command to generate a keystore with keytool in a terminal

keytool –genkey –noprompt –alias <alias e.g., mtls> –keyalg RSA –dname\
 "CN=<Kafka username>, OU=<organization unit>, O=<organization>, C=<country code, e.g., AU>"\
 -keystore <path to keystore, e.g., keystore.jks> -storepass <password for keystore> –keypass <password for keys>

1

2

3

keytool –genkey –noprompt –alias <alias e.g., mtls> –keyalg RSA –dname\

"CN=<Kafka username>, OU=<organization unit>, O=<organization>, C=<country code, e.g., AU>"\

-keystore <path to keystore, e.g., keystore.jks> -storepass <password for keystore> –keypass <password for keys>

Example command to generate a CSR with keytool in a terminal

keytool –keystore <path to keystore, e.g., keystore.jks> -alias <alias e.g., mtls> \ –certreq –file <filename for CSR, e.g., mtls.csr> -storepass <password for keystore>
1
2
keytool –keystore <path to keystore, e.g., keystore.jks> -alias <alias e.g., mtls> \
–certreq –file <filename for CSR, e.g., mtls.csr> -storepass <password for keystore>

Generate a Signed User Certificate

Lastly, you will need to provide Instaclustr CA with the CSR so that it can generate a signed user certificate which can be downloaded for use with your clients. Ensure that the Common Name (CN) matches the username when you add the signed user certificate to your client keystore.

Click Manage Certificates on the row of the newly created user.
Click Add New Certificate to create a user certificate by uploading a CSR.
Click SELECT A CLIENT CERTIFICATE FILE, navigate to the newly created CSR file, select it, and then click Upload. Console will validate the request, and it will appear in the CERTIFICATE SIGNING REQUEST box. Alternatively, you can paste the request straight into this box instead of uploading the file.
Enter the desired validity period of the signed user certificate in months and then click Create.
The newly created Signed User Certificate will now be listed on the Kafka User Certificate Management page.
Download the certificate by clicking Download.

Congratulations, you have now configured the certificates for mTLS authentication and are one step closer to a more secure infrastructure. See how to configure your local client keystore with an example on how to connect your client with mTLS authentication to complete securing your infrastructure.

Managing Client Certificates Using Instaclustr APIv2

Instaclustr APIv2 provides capabilities to sign your certificates via REST endpoints. See Generate a Certificate Singing Request above for more information on CSR generation.

Refer to this documentation for more information and options to POST the CSR.

Note: Each line in the CSR must be separated with a \r\n when passed in a JSON request body.

The API can also be used to list the certificate IDs for a user and download a Kafka certificate using the certificate ID. See the linked documentation for more information on how to list the certificate ID and download the signed certificates

Managing Client Certificates Using Instaclustr Terraform Provider v2

Instaclustr provides Terraform capabilities with Instaclustr Terraform Provider v2, see Using the Instaclustr Terraform Provider for more information.

For more information on the capabilities to sign and import the signed certificate as a resource, refer to this Instaclustr Terraform Documentation.

Limitations

There is no way to revoke client certificates, but the equivalent goal (for example, preventing access to the topics on your cluster for a particular user) can be achieved by changing the associated users Kafka ACLs.
Client certificates need to be signed using the Instaclustr Certificate Authority (CA) for your cluster. Importing and using your own CA for certificate signing is not supported.