Creating a Kafka Connect ClusterMenu
This article describes how to provision a Kafka Connect cluster using the Instaclustr Console. For information on how to provision an Apache Kafka Cluster via the Instaclustr Provisioning API, see this page: Instaclustr Provisioning API
- Kafka Connect needs a Kafka cluster to connect to. If you haven’t already got one, follow the instructions for creating a Kafka cluster to set one up before continuing with Kafka Connect.
- Additionally, it is best to have “auto-create” topics enabled on your Kafka cluster, as this will allow Kafka Connect to seamlessly create the additional topics that it needs to operate. Without this, you will need to set up these topics manually.
- To create a Kafka Connect cluster, navigate to the Clusters Overview page and click the Create Cluster button. This will take you to the Create Cluster page.
- On the Create Cluster page, choose a unique name for your cluster, and select Kafka Connect as the Application
- Under Data Centre section, select your Infrastructure Provider, Region, Custom Name, Data Centre Network address block, Node Size, and number of nodes.
- In the Kafka Connect Options section select which Kafka cluster you would like to target. Additionally you can choose if you’d like your current IP to be allowed to access the Kafka Connect REST API and whether or not to use VPC Peering to connect your Kafka Connect and Kafka networks. If you elect to use VPC peering, Kafka Connect will connect to Kafka on a private IP address. Without VPC peering, it will use the Kafka cluster’s public IP addresses. Finally, if you wish to use your own custom connectors (bring your own code) on your Kafka Connect cluster, you must fill out the name and access credentials of the cloud storage area where you will be placing your custom connectors. See the article on how to set up a cloud bucket to store custom connectors for more information. If you don’t wish to use custom connectors, simply delete all the example text in this box.
Note: Instaclustr detects the IP address of the computer used to access the Dashboard and creates a firewall rule to your computer to connect to the cluster. Certain web proxies may interfere with this mechanism and Instaclustr will see their IP address instead. We suggest you verify the detected address.
- Select SLA Tier you want to tag your cluster with. Non-production clusters may receive lower priority support and reduced SLAs. Production tier is not available when using Developer class nodes.
- The Summary section displays a brief summary of your cluster configurations and pricing details. Click the Terms and Conditions link to open the Instaclustr Terms and Conditions and other policy documents. After going through the document, select the checkbox to accept the Terms and Conditions. Once you are happy with the cluster configuration and have accepted the terms and conditions, click the Create Cluster button to start creating the cluster.
- Provisioning a cluster can take some time depending on the responsiveness of the underlying cloud provider. All status messages will be displayed along with the progress of the cluster creation on the Clusters’ page. Once your cluster reaches RUNNING state (You can check this on the Clusters Overview page), your cluster is ready to go.