Connect to Apache Kafka and Kafka Connect Using VPC Peering (AWS)

Table of Contents

Setting up VPC Peering

For an overview on VPC Peering see the AWS VPC Peering Guide. Instaclustr supports VPC peering as a mechanism for connecting directly to your Instaclustr managed cluster. VPC Peering allows you to access your cluster via private IP and makes for a much more secure network setup.

  1. Once your cluster has been provisioned, you can create a VPC Peering request through the Instaclustr console. Navigate to the VPC Peering tab of your cluster and then click on Add New VPC Connection.
  2. Fill in the required information on the VPC Peering Connections and click the Submit VPC Peering Request button.
  3. If the request is successfully submitted, the Peering Connection will appear as a new entry in a table with a status of Pending Request.
  4. In order to accept the request, you will need to login to your AWS console to accept the peering request and add a route in your VPC to our VPC.
    In order to accept the peering request, login to the AWS VPC console and navigate to Peering Connections. Select the pending VPC peering connection and choose Actions, Accept Request.

    In order to add a route in your VPC to your cluster’s VPC, navigate to the
    main route table of your VPC and add a route to your cluster’s network address block. Under Target, select the Peering Connection that you have just created.Note: If you have EC2 instances running in your VPC that need access to the peering connection, make sure that you create an explicit subnet association in your VPC’s main route table to the subnet of that instance.
  5. We automatically generate the routes within our VPC to ensure traffic is routed correctly to your VPC. Once you have accepted the peering request, the VPC peering connection will show up as active in your Instaclustr dashboard.You now can connect to Kakfa brokers on port 9093.

Troubleshooting

A duplicate request for this VPC Peering Connection already exists.

This indicates that an existing peering request for this Account, VPC and network combination already exists.  Check the Peering Connection table at the bottom of the page to verify.

If you still cannot connect to the cluster via your Peered VPC connection, confirm that you have accepted the peering request, through the AWS Console.

Peering Request status is “Failed”

The most common causes of a failed peering request are:

  • The VPC ID or the account ID of the peering VPC are incorrect
  • The CIDR ranges of the two VPCs overlap
    For example, your cluster network is 10.0.0.0/16 and you are trying to peer it with a VPC in the range 10.0.0.0/18.  Because AWS will need to route traffic for 10.0.0.0/18 to the peered VPC, the overlap will conflict with addresses in the cluster network and is therefore rejected.
  • The cluster VPC and the client VPC are in different regions
    For example, your cluster is in US-EAST-1 and you are attempting to peer it with US-WEST-2.  AWS does not currently support cross-region VPC Peering connections.

Further details are available on the AWS site.

Using Inter-region VPC-peering, Direct Connect and AWS Transit Gateways

For customers running in their own AWS account, the following additional networking scenarios are supported through Instaclustr’s Custom VPC feature:

Using AWS Transit Gateways or AWS Direct Connect to connect from clients to AWS hosted Kafka clusters.

These use cases require manual configuration of the VPC connectivity and the design and maintenance of this configuration is a customer responsibility unless otherwise agreed.

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes