Creating an Apache Kafka® Cluster With Client ⇄ Broker Encryption & Mutual Authentication (mTLS) Enabled

Overview:

In this guide, we will walk you through how to create an Apache Kafka cluster with Client ⇄ Broker Encryption & Mutual Authentication (mTLS) enabled using Instaclustr Console.

We will provide step-by-step instructions and additional resources on mTLS with Apache Kafka. For general guidance on creating an Apache Kafka Cluster, please refer to Creating an Apache Kafka Cluster.

Limitations

There is no way to revoke client certificates, but an equivalent objective (for example, preventing access to the topics on your cluster) can be achieved by modifying the associated Kafka ACLs.
Client certificates need to be signed using the Instaclustr Certificate Authority (CA) for your cluster. You cannot use another CA for certificate signing.
MTLS authentication on Instaclustr for Apache Kafka is only supported for Kafka versions 2.8.2 or later

Provisioning Using the Instaclustr Console

Log in to Instaclustr Console
Log in to your Instaclustr account and access the Console. If you don’t have an account, sign up for a free trial on the Instaclustr website.
Create a New Apache Kafka Cluster
Click the “Create Cluster” button and select “Apache Kafka” from the list of available services.
Configure the Cluster
Fill in the required fields for the new cluster, such as the cluster name, provider, data center, and instance type. Then, select the version of Apache Kafka you want to use and choose the number of nodes for your cluster.
Enable mTLS
Under the Security tab, select Client ⇄ Broker Encryption & Mutual Authentication (mTLS) to enable mTLS. This adds Public and Private Kafka listeners to the cluster on ports 9082 & 9083 respectively for use of mTLS authentication. In addition, the default listeners are also present on the standard ports (9092 and 9093)
Complete the Cluster Creation
Once you have configured your cluster, review your settings and click Create Cluster to complete the creation process. Instaclustr will provision your cluster and provide you with the necessary connection details.
Check Client ⇄ Broker Encryption & Mutual Authentication
Once your cluster is created, navigate to the Details page to check if Client ⇄ Broker Encryption & Mutual Authentication has been enabled

Congratulations! You have successfully created an Apache Kafka cluster with mTLS enabled using Instaclustr Console.

Note: It is useful to understand how enabling Client ⇄ Broker Encryption & Mutual Authentication differs from enabling Client ⇄ Broker Encryption. The former adds a mTLS authenticated Kafka listener to ports 9082 & 9083 whereas the latter configures the security protocol on the default Kafka ports 9092 & 9093. For this reason, the two options are independent and enabling or disabling one has no impact on the other.

Name	Client ⇄ Broker Encryption	Client ⇄ Broker Encryption & Mutual Authentication
Authentication	SASL	mTLS
Ports	Public: 9092 Private: 9093	Public: 9082 Private: 9083
Security Protocol	sasl_ssl	ssl

Using clients to connect with an mTLS enabled clusters requires further steps, including some on the client side. Please continue reading through the pages provided in the Additional Resources section below.

Provisioning Using Instaclustr API

Instaclustr APIv2 can be used with your provisioning keys to create a Kafka cluster via REST endpoints.

An mTLS enabled Kafka cluster can be created via a POST request to the following endpoint:

https://api.instaclustr.com/cluster-management/v2/resources/applications/kafka/clusters/v2

1	https://api.instaclustr.com/cluster-management/v2/resources/applications/kafka/clusters/v2

Set clientBrokerAuthWithMtls to true in the request body. Please refer to the Provisioning API documentation for more information and options.

Provisioning Using Instaclustr Terraform Provider v2

Instaclustr provides Terraform capabilities with Instaclustr Terraform Provider v2. See Using the Instaclustr Terraform Provider for more information on how to set up the Terraform Provider in your environment.

Once you have set up the Terraform Provider, refer to the documentation to create a Kafka cluster with mTLS enabled. Ensure that you have the clientBrokerAuthWithMtls option set to true.