Creating a Kafka cluster in a Custom VPC
This support documentation will walk you through the process of provisioning an Instaclustr Managed Kafka cluster in an existing VPC that you manage. This is only possible for those customers who are running Instaclustr managed cluster in their own cloud account.
Potential Use Cases
Instaclustr has identified the following use cases for this feature:
- Customers with custom/complex network configuration setup. For example, AWS Direct Connect, Inter-region VPC peering or VPC Transit Gateways networking can all be set up once for a VPC and then that VPC reused for multiple clusters.
- Customers looking to minimise AWS network costs. For some use cases, cross VPC traffic can be a significant portion of overall infrastructure costs.
- Customers with a fully automated CI/CD implementation for their application that would like to provision and manage clusters running on the Instaclustr Managed platform. Setting up the VPC and peering once and then recreating clusters as necessary will make it easier to implement a fully automated CI/CD.
Enable the feature for your account
Before using this feature you must have it enabled on your account.
Contact [email protected] to enable the ability to create a Custom VPC cluster.
Console – Create a new cluster.
Once the feature is enabled, you need to supply the following fields when you create a cluster:
Data Centre options
|Infrastructure Provider||Amazon Web Services|
|Provider Account||Your previously setup custom AWS account.*|
|Existing AWS VPC||The VPC you would like to provision into, it must exist in the selected provider account.*|
|Region||The region the VPC resides in.*|
|Custom Name||The name you wish to give your datacentre.|
|Data Centre Network||A CIDR for the private network Instaclustr will use to setup the cluster.**|
* If the provided Account, VPC and Region do not match, provisioning of the cluster or datacentre will fail and you will need to contact [email protected].
** The provided VPC must have this address space available; providing an incorrect CIDR or a Data Centre Network that is already taken will cause provisioning to fail and you will need to contact [email protected].
Provisioning Api – Create a new cluster
For more detailed information about the provisioning api, see our support documentation.
The example request below will create a 3 node Apache Kafka 2.1.1 cluster
"clusterNetwork" : "172.31.96.0/20",
Take care to set the following fields correctly:
- accountName, customVirtualNetworkId & dataCentre
The values for these fields must align with an existing VPC; if not, provisioning of the cluster or datacentre will fail and you will need to contact [email protected].
The desired custom VPC must have this address space available; providing an incorrect CIDR or an address space that is already occupied will cause provisioning to fail and you will need to contact [email protected].
What gets created
This information is used by our provisioning system to set up your cluster datacentre.
The following objects will get created in the Custom VPC:
- An internet gateway (if one isn’t already attached)
- Subnets for the nodes
- Route table to route traffic to these subnets
- Security groups to control access to the nodes
- EC2 Instances – 1 per node
- Public IP address – 1 per node (unless private network cluster)
- EBS disk – as required by node size
Private network clusters have two additional components:
- An EC2 instance acting as an SSH gateway
- A NAT gateway instead of an internet gateway
All of these objects are created by our system and their unique id is stored in our internal database.
Instaclustr tags each object it creates. Custom tags may also be added to your account which will also be applied to all objects created.
In this way, customers can easily identify which objects make up the cluster data centre.
When deprovisioning a cluster, Instaclustr only removes objects that we have created and stored in our database.