This page describes the steps that need to be taken to connect clients in one VPC to a PrivateLink enabled Kafka cluster in another VPC. In short, customers need to create a security group, an endpoint with the endpoint service name of the created Kafka PrivateLink cluster, and a Route 53 record to the endpoint created. Kafka Add-Ons require additional Route53 records.
If the PrivateLink Kafka cluster and client are in different AWS accounts, please ensure the AWS Principal of the client AWS account is added to allow cross-account access to the endpoint. For more info, please read Managing Principal ARNs of a PrivateLink Kafka Cluster.
Create Endpoint Security Group
A security group is required to allow traffic from your VPC to the PrivateLink endpoint. Your Instaclustr provided Kafka cluster already has a security group applied to ensure the security of your cluster, but an Endpoint cannot be created without having an associated security group. Therefore the security group being created here can be as permissive or as strict as you require. These steps will outline the recommended security group configuration.
In the AWS Console, select the appropriate region for your VPC and navigate to the Security Groups page.
Click “Create security group”.
Under the heading “Security group name”, enter a name for the security group. For example, “companyName-kafka-privatelink-group”. The name cannot be edited after creation, thus it is recommended to use a naming convention which allows easy identification of the PrivateLink endpoint and destination cluster it will be associated with.
Under the heading “Description”, enter a short description of the security group. For example, “Allows access to Kafka cluster via PrivateLink”.
Under the heading “VPC”, search for and select the VPC which will connect to the Kafka cluster via PrivateLink.
Under the Inbound rules heading, click “Add rule”. The rules created here permit traffic outbound from your VPC, inbound to the Kafka cluster over PrivateLink.
Select Custom TCP in the “Type” field. Enter 9091 in the “Port range” field (n.b. Port 9091 is the Kafka Broker listener port. When connecting via PrivateLink, this port is load balanced between all broker nodes in your cluster. This port will not change, and can be verified on your Kafka cluster’s Connection Info page.) Select Custom in the “Source” field, then enter the CIDR of the subnet you will be connecting to the PrivateLink endpoint from (ie. the CIDR within which your EC2 Instance resides). Optionally, add a description in the Description field.
Repeat step 7, using ports 6001-6000+n in the Port range field, where n is the number of nodes in your Kafka cluster. For example, if you are using a 5 node Kafka cluster, the port range should be 6001-6005. n.b. If the number of nodes in the Kafka cluster is increased in the future, this port range will need to be extended to match the new number of nodes in the cluster. The purpose of the ports in this range is to provide a direct connection to a specific broker within the Kafka cluster, where the port to access a broker is 6000 + the broker’s ID, ie. broker with ID of 6 is accessible via port 6006.
Click “Create security group”, the security group should be created successfully. Take note of the security group name, as it will be required when creating the Endpoint in the following steps.
If you have any Kafka add-ons enabled in the PrivateLink Kafka cluster, please add the corresponding ports as well. For Kafka Schema Registry/Karapace Schema Registry, the port is 8085 and the Custom CIDR is the same as the one in step 7. While for Karapace Rest Proxy, there are two rules to add. The first rule has 8083ports and the other is 8091 – 8090+p, where p is the number of Karapace Rest Proxy nodes in PrivateLink Kafka Cluster. The Custom CIDR for both rules are the same as the one in step 7. The Karapace Rest Proxy Nodes are tagged “Karapace REST Proxy” in the Details page in Instaclustr console.
If the PrivateLink Kafka cluster has both add-ons, with 3 Karapace Rest Proxy nodes, expect the following inbound rules.
Create Endpoint and Route 53 Record
On the AWS VPC Endpoint Console page, click “Create endpoint”.
On the creation page under “Service category”, select “Other endpoint services”
In “Service Settings”, paste the endpoint service name from step 2 to the “Service name” field. Click “Verify service”and a green box will appear.
For “VPC”, select the client’s VPC that will connect to the PrivateLink Kafka cluster. Note, the VPC of the client and PrivateLink Kafka cluster should be in the same region as we do not support inter-region access via PrivateLink.
The configuration would then be as follows.
For “Subnets”, please select one “Subnet ID” for each “Availability Zone”. Please note that we recommend that your clients (producers and consumers) be located in separate availability zones to support high availability should any one availability zone be unreachable.
Select the security group that you created earlier. You will get the following configurations. Once done, click “Create endpoint”.
You will be redirected to the main VPC endpoint page which will specifically show the created endpoint. Once the Status of the endpoint reaches Available, copy the first DNS under the “DNS names”.
Find the “URL” in Retrieve Required Cluster Connection Information step 2. Under “Hosted zone configuration”, enter the “URL”without the port and first part of the hostname into “Domain name”. For example for kafka.test.com:9091, enter test.com.
Select “Private hosted zone” as “Type”.
Select the corresponding region and VPC of the client used in step 4.
Ensure you have the following configurations, click “Create hosted zone”.
You will be redirected to the created private hosted zone.
Select “Create record”.
Under “Record name”, place the rest of the “URL”from step 2 without 9091. For instance, the “URL” is kafka.test.com:9091 and we entered test.com in step 14. Thus the remaining URL to be entered is kafka.
Click the toggle “Alias” and select “Alias to VPC endpoint” from the dropdown box.
Select the same region as the one in step 3 in the next dropdown box.
In the last field, paste the endpoint DNS name from step 8.
The resulting configuration is as follows.
The newly created record will be in the hosted zone. You can now connect to the cluster with kafka.test.com:9091.
If you have any Kafka add-ons enabled in the PrivateLink cluster, find the instaclustr domain in the Connection Info page of the add-on. In the following Karapace Rest Proxy example, the Instaclustr domain is 7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com.
Create a new hosted zone with the Instaclustr domain in step 22 by repeating step 9 to 14.