Connect to Apache Kafka and Kafka Connect Using VPC Peering (AWS)

Table of Contents

Setting up VPC Peering

For an overview on VPC Peering see the AWS VPC Peering Guide. Instaclustr supports VPC peering as a mechanism for connecting directly to your Instaclustr managed cluster. VPC Peering allows you to access your cluster via private IP and makes for a much more secure network setup. When you create the peering request, you can specify the peer VPC networks as either CIDR blocks or customer-managed AWS prefix list IDs (pl-…). 

Prefix lists are supported only on AWS Bring Your Own Cloud (BYOC) cluster data centres. Create the prefix list in the same AWS region as the cluster data centre. Keep CIDRs inside the list from overlapping the cluster network. You are responsible for the contents of the list. Instaclustr does not read or validate the CIDRs inside the prefix list.

  1. Once your cluster has been provisioned, you can create a VPC Peering request through the Instaclustr console. Navigate to the VPC Peering tab of your cluster and then click on Add New VPC Connection.supporting screenshot
  2. Fill in the required information on the VPC Peering Connections (peer VPC ID, AWS account ID, region, and either CIDR peer networks or prefix list IDs). Click the Submit VPC Peering Request button. If Add VPC peer networks to cluster firewall is selected, Instaclustr adds those CIDRs or prefix list IDs to the cluster firewall. Do not add an overly permissive CIDR, such as 0.0.0.0/0, and do not include one in a prefix list.supporting screenshot
  3. If the request is successfully submitted, the Peering Connection will appear as a new entry in a table with a status of Pending Request.supporting screenshot
  4. In order to accept the request, you will need to login to your AWS console to accept the peering request and add a route in your VPC to our VPC.
    In order to accept the peering request, login to the AWS VPC console and navigate to Peering Connections. Select the pending VPC peering connection and choose Actions, Accept Request.supporting screenshot

    In order to add a route in your VPC to your cluster’s VPC, navigate to the     main route table of your VPC and add a route to your cluster’s network address block. Under Target, select the Peering Connection that you have just created.Note: If you have EC2 instances running in your VPC that need access to the peering connection, make sure that you create an explicit subnet association in your VPC’s main route table to the subnet of that instance.supporting screenshot
  5. We automatically generate the routes within our VPC to ensure traffic is routed correctly to your VPC. Once you have accepted the peering request, the VPC peering connection will show up as active in your Instaclustr dashboard.supporting screenshotYou now can connect to Kakfa brokers on port 9093.

Troubleshooting

A duplicate request for this VPC Peering Connection already exists.

This indicates that an existing peering request for this Account, VPC and network combination already exists.  Check the Peering Connection table at the bottom of the page to verify.

If you still cannot connect to the cluster via your Peered VPC connection, confirm that you have accepted the peering request, through the AWS Console.

Peering Request status is “Failed”

The most common causes of a failed peering request are:

  • The VPC ID or the account ID of the peering VPC are incorrect
  • The CIDR ranges of the two VPCs overlap
    For example, your cluster network is 10.0.0.0/16 and you are trying to peer it with a VPC in the range 10.0.0.0/18.  Because AWS will need to route traffic for 10.0.0.0/18 to the peered VPC, the overlap will conflict with addresses in the cluster network and is therefore rejected.
  • The cluster VPC and the client VPC are in different regions
    For example, your cluster is in US-EAST-1 and you are attempting to peer it with US-WEST-2.  AWS does not currently support cross-region VPC Peering connections.

Further details are available on the AWS site.

Using Inter-region VPC-peering, Direct Connect and AWS Transit Gateways

For customers running in their own AWS account, the following additional networking scenarios are supported through Instaclustr’s Custom VPC feature:

Using AWS Transit Gateways or AWS Direct Connect to connect from clients to AWS hosted Kafka clusters.

These use cases require manual configuration of the VPC connectivity and the design and maintenance of this configuration is a customer responsibility unless otherwise agreed.