This article describes how to provision an Apache Kafka cluster with PrivateLink using the Instaclustr Console. PrivateLink is a networking feature provided by Amazon Web Services (AWS) that provides direct and secure connectivity between AWS VPCs. You can read more about PrivateLink and our support for it here.
The Enterprise Feature called Private Network Cluster is a prerequisite to PrivateLink, and enabling PrivateLink will automatically enable Private Network Cluster.
Kafka REST Proxy, due to a technical limitation, is not supported with PrivateLink. If you have a use case where REST Proxy is required, it is recommended you consider using the newer and up-to-date add-on, Karapace REST Proxy.
A managed Kafka cluster on the Instaclustr platform with PrivateLink enabled is limited to 48 nodes. This limitation comes from the AWS limitation of only allowing 50 listeners per Network Load Balancer.
Creating a Cluster
Log into the console and click the Create Cluster button.
Under the Applications page, select Apache Kafka as the application and AWS as the provider. You should see a checkbox under Enterprise Feature called PrivateLink.
Note: The Instaclustr platform currently only supports AWS PrivateLink.
Click on the PrivateLink feature and a notification box should appear as follows. Click Next.
Under the Kafka Setup page, , first select the security protocol for the cluster using the radio buttons. Then to configure PrivateLink there should be a section called PrivateLink Configuration Settings. In this section, there are 2 fields: Advertised HostName is required to be filled and IAM Principal ARNs is optional
Advertised HostName is required for PrivateLink as it is used by clients to connect to the cluster. For instance, if Advertised HostName is kafka.test.com, when connecting to this cluster, use kafka.test.com:9091. IAM Principal ARNs is optional for PrivateLink and can be adjusted later by following the managing Principal ARNs. IAM Principal ARNs are used to allow connection from your endpoint to the created cluster’s endpoint service. For instance, if arn:aws:iam::123456789012:user/companyNameTest is a real AWS Principal ARN, and is added to this field, you will be able to create an endpoint with the provided endpoint service name. Fill those fields and click Next. Note: Due to the technical limitation of how Kafka REST Proxy operates, its use is not supported with PrivateLink.
Under the Data Centre page, confirm the selected Data Centre is correct. Click Next.
Under the Confirmation page, check that the configurations such as Advertised HostName and IAM Principal ARNs are properly configured, accept Instaclustr terms and conditions, and click Create Cluster.