GCP Private Service Connect

Private Service Connect (PSC) is a Google Cloud networking feature that allows you to reach a service in another VPC network privately, using an internal IP address in your own VPC network. Traffic stays entirely within the Google Cloud network and never traverses the public internet. You can read more in the Google Cloud Private Service Connect documentation. For the equivalent features on other cloud providers, see our documentation on AWS PrivateLink and Azure Private Link.

For step-by-step provisioning instructions for Kafka clusters, including guidance on sizing Shotover Proxy nodes, see Creating a Private Service Connect Apache Kafka Cluster 

Prerequisites 

  • Client applications must run in a Google Cloud VPC network, from which you create Private Service Connect endpoints. 
  • A Shotover Proxy node size must be selected when the cluster is provisioned. 
  • Shotover Proxy nodes are used to expose the cluster through Private Service Connect. 

Limitations 

  • Private Service Connect must be enabled when the cluster is provisioned. It cannot be added to, or removed from, an existing cluster. 
  • A consumer accept list can contain either GCP projects or VPC networks, but not a mixture of both. Updating the list replaces it in full rather than adding or removing individual entries. 
  • Cluster DNS is not supported on clusters using Private Service Connect. 

Background Knowledge 

How Private Service Connect Works 

Private Service Connect has two halves. Producers who host services and offer private connections through a service attachment in a separate VPC network. And consumers access produced services privately from inside their VPC network. 

Instaclustr operates the producer side, publishing the cluster through service attachments. You operate on the consumer side, creating Private Service Connect endpoints and forwarding rules with internal IP addresses in your own VPC network that reference those service attachments. 

When you provision a cluster with Private Service Connect enabled, we create one service attachment per rack, each in its own private subnet inside the cluster network. A three-rack cluster therefore publishes three service attachments, and you create the corresponding endpoints in your VPC network. Because each rack is published separately, your clients can address individual racks, which is what allows Apache Kafka clients to reach a specific broker. Traffic reaches the nodes through a Shotover Proxy instance. 

 

Private Service Connect diagram

Consumer Accept List 

Service attachments published by Instaclustr use manual connection acceptance, and access is controlled by a consumer accept list. A connection request from a GCP project or VPC network on the list is approved automatically, while a request from anywhere else stays in a pending state until it is accepted manually.  

A list can be scoped either by project or by VPC network, from the GCP Private Service Connect page in the Instaclustr Console, or through the Instaclustr Provisioning API and Terraform provider. 

Note: connection reconciliation is enabled by default, so changes to the accept list affect existing connections. If you remove a project from the list, all connections from that project transition to a pending state. 

Questions 

Please contact [email protected] for any further inquiries.