Creating a PrivateLink Kafka Cluster
This article describes how to provision an Apache Kafka cluster with PrivateLink using the Instaclustr Console. PrivateLink is a networking feature provided by Amazon Web Services (AWS) that provides direct and secure connectivity between AWS VPCs. You can read more about PrivateLink and our support for it here.
In addition to the general limitations referred to in the PrivateLink documentation page, please note:
- The Enterprise Feature called Private Network Cluster is a prerequisite to PrivateLink, and enabling PrivateLink will automatically enable Private Network Cluster.
- Kafka REST Proxy, due to a technical limitation, is not supported with PrivateLink. If you have a use case where REST Proxy is required, it is recommended you consider using the newer and up-to-date add-on, Karapace REST Proxy.
- A managed Kafka cluster on the Instaclustr platform with PrivateLink enabled is limited to 48 nodes. This limitation comes from the AWS limitation of only allowing 50 listeners per Network Load Balancer.
Creating a Cluster
- Log into the console and click the Create Cluster button.
- Under the Applications page, select Apache Kafka as the application and AWS as the provider. You should see a checkbox under Enterprise Feature called PrivateLink.
Note: The Instaclustr platform currently only supports AWS PrivateLink.
- Click on the PrivateLink feature and a notification box should appear as follows. Click Next.
- Under the Kafka Setup page, there should be a section called PrivateLink Configuration Settings. In this section, there are 2 fields: Advertised HostName is required to be filled and IAM Principal ARNs is optional
Advertised HostName is required for PrivateLink as it is used by clients to connect to the cluster. For instance, if Advertised HostName is
kafka.test.com, when connecting to this cluster, use
kafka.test.com:9091. IAM Principal ARNs is optional for PrivateLink and can be adjusted later by following the managing Principal ARNs. IAM Principal ARNs are used to allow connection from your endpoint to the created cluster’s endpoint service. For instance, if
arn:aws:iam::123456789012:user/companyNameTestis a real AWS Principal ARN, and is added to this field, you will be able to create an endpoint with the provided endpoint service name. Fill those fields and click Next.
Note: Due to the technical limitation of how Kafka REST Proxy operates, its use is not supported with PrivateLink.
- Under the Data Centre page, confirm the selected Data Centre is correct. Click Next.
- Under the Confirmation page, check that the configurations such as Advertised HostName and IAM Principal ARNs are properly configured, accept Instaclustr terms and conditions, and click Create Cluster.
Once your cluster is running, further steps will still be required to connect to it.
Please refer to our support page on Connecting to a PrivateLink Kafka Cluster for a guide through this process.
For more information on PrivateLink, please refer to our support documentation page.
Please contact Instaclustr Support for any further inquiries.