NetApp Closes Acquisition of Instaclustr Read the announcement
Connecting to a PrivateLink Kafka Cluster

Overview

This page describes the steps that need to be taken to connect clients in one VPC to a PrivateLink enabled Kafka cluster in another VPC. In short, clients need to create an endpoint with the endpoint service name of the created PrivateLink Kafka cluster, and a Route 53 record to the endpoint created.

Retrieve Required Cluster Connection Information

  1. Log into the console, click the created cluster and enter the Connection Info page.
  2. The Connection Info page contains the specific connection details for PrivateLink. The endpoint created in the next section uses the endpoint service name shown below.
  3. In the Connection Info page, identify the region of the created Kafka PrivateLink cluster. For instance, the page shows the region is US_EAST_1, hence access to the AWS VPC Endpoint console in the US_EAST_1 region: https://us-east-1.console.aws.amazon.com/vpc/home?region=us-east-1#Endpoints:

    Create Endpoint and Route 53 Record

  4. On the AWS VPC Endpoint Console page, click Create endpoint.
  5. On the creation page under Service category, select Other endpoint services.
  6. In Service Settings, paste the endpoint service name from step 2 to the Service name field. Click Verify service and a green box will appear.
  7. For VPC, select the client’s VPC that will connect to the Kafka PrivateLink cluster. Note, the VPC of the client and Kafka PrivateLink cluster should be in the same region as we do not support inter-region access via PrivateLink.
  8. The configuration would then be as follows.
  9. For Subnets, please select one Subnet ID for each Availability Zone. Please note that we recommend that your clients (producers and consumers) be located in separate availability zones to support high availability should any one availability zone be unreachable.
  10. Only select the Security Group with ic in the Group name. This is the security group managed by Instaclustr. You will get the following configurations. Once done, click Create endpoint.
  11. You will be redirected to the main VPC endpoint page which will specifically show the created endpoint. Once the Status of the endpoint reaches Available, copy the first DNS under the DNS names.
  12. Head to the Route 53 record home page, enter Hosted Zone and select Create hosted zones. https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones?region=us-east-1#
  13. Under Hosted zone configuration, enter the URL without the port and first part of the hostname into Domain name. For example for kafka.test.com:9091, enter test.com.
  14. Select Private hosted zone as Type.
  15. Select the corresponding region and VPC of the client used in step 7. 
  16. Ensure you have the following configurations, click Create hosted zone.
  17. You will be redirected to the created private hosted zone.
  18. Select Create record.
  19. Under Record name, place the rest of the URL from step 2 without 9091. For instance, the URL is kafka.test.com:9091 and we entered test.com in  step 14. Thus the remaining URL to be entered is  kafka.
  20. Click the toggle Alias and select Alias to VPC endpoint from the dropdown box.
  21. Select the same region as the one in step 3 in the next dropdown box.
  22. In the last field, paste the endpoint DNS name from step 10.
  23. The resulting configuration is as follows.
  24. The newly created record will be in the hosted zone. You can now connect to the cluster with kafka.test.com:9091.

Questions

Please contact [email protected] for any further inquiries.

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes