Creating a Kafka Connect Cluster
This article describes how to provision a Kafka Connect cluster using the Instaclustr Console. For information on how to provision an Apache Kafka Connect Cluster via the Instaclustr Provisioning API, see this page: Instaclustr Provisioning API
- A Kafka Connect cluster needs a Kafka cluster to connect to. If you haven’t already got one, follow the instructions for creating a Kafka cluster to set one up before continuing with Kafka Connect cluster creation.
- To create a Kafka Connect cluster, navigate to the Clusters Overview page and click the Create Cluster button. This will take you to the Create Cluster page.
- On the Create Cluster page, choose a unique name for your cluster, and select Kafka Connect as the Application. You can also choose to enable Private Network add-on for the Kafka Connect cluster. For more information on Private Network add-on, see Private Network cluster.
- Under Data Centre section, select your Infrastructure Provider, Region, Custom Name, Data Centre Network address block, Node Size, and number of nodes. You can provision in AWS and Azure providers.
- In the Kafka Connect Options section there are several options to configure your Kafka Conect cluster
- Choose if you’d like your current IP to be allowed to access the Kafka Connect REST API. Select this if you will be managing connectors from the same machine that you’re creating the cluster from. Note: Instaclustr detects the IP address of the computer used to access the Dashboard and creates a firewall rule to your computer to connect to the cluster. Certain web proxies may interfere with this mechanism and Instaclustr will see their IP address instead. We suggest you verify the detected address.
- Choose which sort of Kafka Cluster you’d like to target. Choose an Instaclustr managed Kafka Cluster if possible, as this will be easier to configure and come with better integration options. If you need to connect to a non-Instaclustr cluster (e.g. MSK) then select the ‘Other Kafka Cluster’ option. If you do so, read this article on using non-instaclustr Kafka clusters for more information.
- There are three ways of setting up VPC for Kafka Connect cluster. The first option is to provision Kafka Connect cluster in the existing VPC (AWS) / Virtual Network (AZURE) of the target Kafka cluster. This will cause Kafka Connect and Kafka clusters to connect over private IPs. The second option available only for AWS, is to provision Kafka Connect cluster in a new VPC and set up VPC peering relationship between the Kafka connect cluster and the Kafka cluster. This will also cause Kafka Connect to connect over private IPs to the Kafka cluster. However, VPC peering incurs data transfer charges across peering connections. Also, if peering across regions, check the AWS documentation for which regions are supported for cross region peering. The third option is to provision Kafka Connect cluster in a new VPC / Virtual Network without setting up peering. This will cause Kafka Connect and Kafka clusters to connect over public IPs.
- If you plan on using your own custom connectors (bring your own code) on your Kafka Connect cluster, tick the Use Custom Connectors checkbox and fill out the name and access credentials of the cloud storage area where you will be placing your custom connectors. See the article on how to set up a cloud bucket to store custom connectors for more information. If you don’t wish to use custom connectors, simply do not check the box.
- Select SLA Tier you want to tag your cluster with. Non-production clusters may receive lower priority support and reduced SLAs. Production tier is not available when using Developer class nodes.
- The Summary section displays a brief summary of your cluster configurations and pricing details. Click the Terms and Conditions link to open the Instaclustr Terms and Conditions and other policy documents. After going through the document, select the checkbox to accept the Terms and Conditions. Once you are happy with the cluster configuration and have accepted the terms and conditions, click the Create Cluster button to start creating the cluster.
- Provisioning a cluster can take some time depending on the responsiveness of the underlying cloud provider. All status messages will be displayed along with the progress of the cluster creation on the Clusters’ page. Once your cluster reaches RUNNING state (You can check this on the Clusters Overview page), your cluster is ready to go.