Use VPC Peering (AWS) to Connect to OpenSearch

Setting up VPC Peering

For an overview on VPC Peering see the AWS VPC Peering Guide. Instaclustr supports VPC peering as a mechanism for connecting directly to your Instaclustr managed cluster. VPC Peering allows you to access your cluster via private IP and makes for a much more secure network setup.

Note: If you only intend to connect to your cluster from a peered VPC, then choose Use private IP addresses for node discovery under OpenSearch Options when you create a new cluster. If you’re peering to an existing cluster, contact support to change the nodes’ broadcast address.

  1. Once your cluster has been provisioned, you can create a VPC Peering request through the Instaclustr console. Navigate to the VPC Peering tab of your cluster and then click on Add a New VPC Connection.
  2. Fill in the required information on the VPC Peering Connections and click the Submit VPC Peering Request button.
  3. If the request is successfully submitted, the Peering Connection will appear as a new entry in a table with a status of Pending Request.

  4. In order to accept the request, you will need to login to your AWS console to accept the peering request and add a route in your VPC to our VPC.
    To accept the peering request in the AWS console, navigate to Peering Connections, select the pending VPC peering connection and choose ActionsAccept Request.To add a route in your VPC to your cluster’s VPC, navigate to the main route table of your VPC and add a route to your cluster’s network address block. Under Target, select the Peering Connection that you have just created.Note: If you have EC2 instances running in your VPC that need access to the peering connection, make sure that you create an explicit subnet association in your VPC’s main route table to the subnet of that instance.
  5. We automatically generate the routes within our VPC to ensure traffic is routed correctly to your VPC. Once you have accepted the peering request, the VPC peering connection will show up as active in your Instaclustr dashboard.

Note: To test the peering, you may try netcat or telnet. Port 9042 is the exposed port for CQL:

A result of 0 indicates success.

A telnet prompt indicates success, enter quit to close the connection. The same test can be run using port 7077 to test Spark connectivity.

Troubleshooting

A duplicate request for this VPC Peering Connection already exists.

This indicates that an existing peering request for this Account, VPC and network combination already exists.  Check the Peering Connection table at the bottom of the page to verify.

If you still cannot connect to the cluster via your Peered VPC connection, confirm that you have accepted the peering request, through the AWS Console.

Peering Request status is “Failed”

The most common causes of a failed peering request are:

  • The VPC ID or the account ID of the peering VPC are incorrect
  • The CIDR ranges of the two VPCs overlap
    For example, your cluster network is 10.0.0.0/16 and you are trying to peer it with a VPC in the range 10.0.0.0/18.  Because AWS will need to route traffic for 10.0.0.0/18 to the peered VPC, the overlap will conflict with addresses in the cluster network and is therefore rejected.
  • The cluster VPC and the client VPC are in different regions
    For example, your cluster is in US-EAST-1 and you are attempting to peer it with US-WEST-2.  AWS does not currently support cross-region VPC Peering connections.

Further details are available on the AWS site.

Using Inter-region VPC-peering, Direct Connect and AWS Transit Gateways

For customers running in their own AWS account, the following additional networking scenarios are supported through Instaclustr’s Custom VPC feature:

Inter-region VPC Peering for cross-datacenter communication in multi datacenter OpenSearch clusters. (Note: Inter-region VPC Peering for client to cluster communication is supported using the mechanisms above.)

AWS Transit Gateways to either connect from clients to clusters or between data centers in a multi datacenter OpenSearch cluster.

AWS Direct Connect for either on-prem client to AWS hosted cluster connections or connections between on-prem data centers and AWS data centers in multi datacenter, hybrid-cloud OpenSearch clusters, or to use a private connection for traffic between two AWS data centers.

These use cases require manual configuration of the VPC connectivity and the design and maintenance of this configuration is a customer responsibility unless otherwise agreed.

Table of Contents

Setting up VPC Peering

For an overview on VPC Peering see the AWS VPC Peering Guide. Instaclustr supports VPC peering as a mechanism for connecting directly to your Instaclustr managed cluster. VPC Peering allows you to access your cluster via private IP and makes for a much more secure network setup.

Note: If you only intend to connect to your cluster from a peered VPC, then choose Use private IP addresses for node discovery under Elasticsearch Options when you create a new cluster. If you’re peering to an existing cluster, contact support to change the nodes’ broadcast address.

  1. Once your cluster has been provisioned, you can create a VPC Peering request through the Instaclustr console. Navigate to the VPC Peering tab of your cluster and then click on Add a New VPC Connection.
  2. Fill in the required information on the VPC Peering Connections and click the Submit VPC Peering Request button. 

  3. If the request is successfully submitted, the Peering Connection will appear as a new entry in a table with a status of Pending Request.

  4. In order to accept the request, you will need to login to your AWS console to accept the peering request and add a route in your VPC to our VPC.
    To accept the peering request in the AWS console, navigate to Peering Connections, select the pending VPC peering connection and choose Actions, Accept Request.

    To add a route in your VPC to your cluster’s VPC, navigate to the main route table of your VPC and add a route to your cluster’s network address block. Under Target, select the Peering Connection that you have just created.

    Note: If you have EC2 instances running in your VPC that need access to the peering connection, make sure that you create an explicit subnet association in your VPC’s main route table to the subnet of that instance.

  5. We automatically generate the routes within our VPC to ensure traffic is routed correctly to your VPC. Once you have accepted the peering request, the VPC peering connection will show up as active in your Instaclustr dashboard.

Note: To test the peering, you may try netcat or telnet. Port 9042 is the exposed port for CQL:

nc -z <node_private_address> 9042; echo $?

A result of 0 indicates success.

telnet <node_private_address> 9042

A telnet prompt indicates success, enter quit to close the connection. The same test can be run using port 7077 to test Spark connectivity.

Troubleshooting

A duplicate request for this VPC Peering Connection already exists.

This indicates that an existing peering request for this Account, VPC and network combination already exists.  Check the Peering Connection table at the bottom of the page to verify.

If you still cannot connect to the cluster via your Peered VPC connection, confirm that you have accepted the peering request, through the AWS Console.

Peering Request status is “Failed”

The most common causes of a failed peering request are:

  • The VPC ID or the account ID of the peering VPC are incorrect
  • The CIDR ranges of the two VPCs overlap
    For example, your cluster network is 10.0.0.0/16 and you are trying to peer it with a VPC in the range 10.0.0.0/18.  Because AWS will need to route traffic for 10.0.0.0/18 to the peered VPC, the overlap will conflict with addresses in the cluster network and is therefore rejected.
  • The cluster VPC and the client VPC are in different regions
    For example, your cluster is in US-EAST-1 and you are attempting to peer it with US-WEST-2.  AWS does not currently support cross-region VPC Peering connections.

Further details are available on the AWS site.

Using Inter-region VPC-peering, Direct Connect and AWS Transit Gateways

For customers running in their own AWS account, the following additional networking scenarios are supported through Instaclustr’s Custom VPC feature:

Inter-region VPC Peering for cross-datacenter communication in multi datacenter Elasticsearch clusters. (Note: Inter-region VPC Peering for client to cluster communication is supported using the mechanisms above.)

AWS Transit Gateways to either connect from clients to clusters or between data centers in a multi datacenter Elasticsearch cluster.

AWS Direct Connect for either on-prem client to AWS hosted cluster connections or connections between on-prem data centers and AWS data centers in multi datacenter, hybrid-cloud Elasticsearch clusters, or to use a private connection for traffic between two AWS data centers.

These use cases require manual configuration of the VPC connectivity and the design and maintenance of this configuration is a customer responsibility unless otherwise agreed.

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes