Connect to a Private Service Connect Apache Kafka cluster

This guide aims to provide basic instructions on how customers can connect to their Kafka Private Service Connect Cluster from their applications. See Creating a Private Service Connect Apache Kafka Cluster for more information on how Private Service Connect Kafka clusters work and how to create one.

Infrastructure deployment via GCP Console

Retrieve Required Cluster Connection Information

On the console connection information page, retrieve the published Service Attachment names:

These will be the service attachments to connect to in the client VPC.

Create Private Service Connect Endpoints

On the google cloud console, create Private Service Connect endpoints that connect to the above Service Attachments.

  1. On the GCP console, search for Private Service Connect.
  2. Under the Connected Endpoints tab, click on + CONNECT ENDPOINT to create an endpoint:In the Connect Endpoint page, specify the following options:
    • Target: Published Service
    • Target Service: Service attachment path:
      projects/<project name> /regions/<region>/serviceAttachments/<service attachment name above>
    • Endpoint name: choose a name that makes sense in your client VPC.
    • Network: the network where your client applications are. Make sure that this network, or the project this network resides in, is added to the consumer accept list.
    • Subnetwork: the subnetwork where your client applications are.
    • IP address: create a new IP or using a pre-created IP address you like.

    Enable global access: enable it if you wish to allow instances from another VPC to access the endpoint.

  3. Perform the above step for all Service Attachments.

Configure DNS Zone and Entries

Configure DNS entries so your client instances can connect to your cluster over the advertised hostnames you specified:

  1. (Optional) Create an internal Cloud DNS zone that is available in your client VPC.
  2. Create an entry that maps an Advertised Hostname to the IP of a Private Service Connect endpoint. Using the example above:
    kafka-psc-test.instaclustr.com maps to X.X.X.X (the IP of a Private Service Endpoint)
  3. Do the above step for each of the Private Service Connect endpoints.
  4. (Optional) You could also have one DNS name that routes randomly to one of the Private Service Endpoints, this DNS name could be used as bootstrap server for your client applications.

Summary

Above are instructions on how customers might set up their client-side connection to a Private Service Connect cluster. Other infrastructure modifications might be required, such as firewall rules and route table changes. Please contact [email protected] for further questions.

Infrastructure deployment via Terraform

Below are examples for deploying necessary infrastructure to connect to a Private Service Connect Kafka cluster.

Client VPC Network

The following Terraform providers are required:

A typical VPC network would have the following Terraform definitions:

Note that the above examples are simply to provide reference points for the below Terraform resources, but actual configurations will vary based on customers’ needs. For instance, customers might choose to enable auto_create_subnetworks in their VPC.

More configs/subnets can be added, see google_compute_network and google_compute_subnetwork for more information.

IP Addresses

3 internal IP addresses are required because 3 forwarding rules are required to connect to the 3 published Service Attachments.

See google_compute_address for more information.

Forwarding Rules

Forwarding rules are what connect the client VPC to the published services. They serve as the bridge between client applications and Kafka. Three forwarding rules are required to connect to the three published Service Attachments.

Cloud DNS (Optional)

Clients might want to set up a private cloud DNS zone that contains records to resolve the advertised hostnames (provided when provisioning the cluster) to the IPs used by the forwarding rules. This is so that the client applications can connect to Kafka using the advertised hostnames.

Alternatively, clients could insert above DNS entries at the instance/application level, e.g. putting them into /etc/hosts on a client instance.

Summary

Above are simple examples of how customers might set up their client-side connection to a Private Service Connect cluster. Other infrastructure modifications might be required, such as firewall rules changes. Please contact [email protected] for further questions.

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes