Connect to Kafka and Kafka Connect Using VPC Peering (AWS)Menu
Table of Contents
Setting up VPC Peering
For an overview on VPC Peering see the AWS VPC Peering Guide. Instaclustr supports VPC peering as a mechanism for connecting directly to your Instaclustr managed cluster. VPC Peering allows you to access your cluster via private IP and makes for a much more secure network setup.
Note: If you only intend to connect to your cluster from a peered VPC, then choose Use private IP addresses for node discovery under Kafka Options when you create a new cluster. If you’re peering to an existing cluster, contact support to change the nodes’ broadcast address.
- Once your cluster has been provisioned you can create a VPC Peering request through the Instaclustr dashboard. Click Cluster Settings from the Manage Cluster menu.
- Once you are on the Cluster Settings page, click the VPC Peering Settings button.
- Fill in the required information on the VPC Peering Connections and click Submit VPC Peering Request button.
- If the request is successfully submitted, the request will now appear as “Pending Acceptance” in the bottom table.
- Once you have submitted the peering request, you will need to login to your AWS console to accept the request and add a route in your VPC to our VPC.Note: You will need to explicitly assign the route you created to the subnet of the instances that need to access the peering connection. You can do this by:
- Checking the instance details to find its subnet
- Copying the subnet ID
- Navigating to the VPC section and filtering the Subnets for the copied ID
- Click on the “Route Table” tab and change the assigned route table to the new route table
The destination should be the CIDR range of the private IPs in the VPC you’re routing to
The target needs to be the VPC peer IDOnce the new route table has been assigned to your subnet, the “Main” column will change to “Yes”. We automatically generate the routes within our VPC to ensure traffic is routed correctly to your VPC.
- Once you have accepted the peering request, the VPC peering connection will show up as active in your Instaclustr dashboard.
nc -z <node_private_address> 9042; echo $?
A result of 0 indicates success.
telnet <node_private_address> 9042
A telnet prompt indicates success, enter quit to close the connection. The same test can be run using port 7077 to test Spark connectivity.
A duplicate request for this VPC Peering Connection already exists.
This indicates that an existing peering request for this Account, VPC and network combination already exists. Check the Peering Connection table at the bottom of the page to verify.
Peering Request status is “Failed”
The most common causes of a failed peering request are:
- The VPC ID or the account ID of the peering VPC are incorrect
- The CIDR ranges of the two VPCs overlap
For example, your cluster network is 10.0.0.0/16 and you are trying to peer it with a VPC in the range 10.0.0.0/18. Because AWS will need to route traffic for 10.0.0.0/18 to the peered VPC, the overlap will conflict with addresses in the cluster network and is therefore rejected.
- The cluster VPC and the client VPC are in different regions
For example, your cluster is in US-EAST-1 and you are attempting to peer it with US-WEST-2. AWS does not currently support cross-region VPC Peering connections.
Further details are available on the AWS site.
Using Inter-region VPC-peering, Direct Connect and AWS Transit Gateways
For customers running in their own AWS account, the following additional networking scenarios are supported through Instaclustr’s Custom VPC feature:
These use cases require manual configuration of the VPC connectivity and the design and maintenance of this configuration is a customer responsibility unless otherwise agreed.