AWS Private S3 Bucket Integration

ClickHouse clusters on our platform by default have security restrictions preventing egress traffic. But they can be integrated with private AWS S3 buckets to enable access.  

Once a private S3 bucket is integrated, the IAM policy of the cluster role will have access to the specified S3 bucket with either read-only or read/write access. KMS Keys are also available for configuration, which will be attached to the same IAM role of the cluster.  

Clusters on the NetApp Instaclustr managed platform are secured through egress firewall rules to protect against data exfiltration. Integrating with AWS private S3 bucket adds a whitelist rule to the firewall enabling access. Consider the security risk before enabling an AWS Private S3 Bucket integration. 

How To Enable

You can set up ClickHouse Private S3 Bucket Integration using the Instaclustr Console, API, or Terraform provider. Steps for using each method is provided below.

Using the Console

The following steps explain how to integrate a ClickHouse cluster with a private AWS S3 bucket.  

1. First expand the ClickHouse cluster options on the sidebar and select the “Integrations” option in Console. The page will show available types of integrations to add to the cluster.

   

2. Click “Configure” for the row with type “AWS S3 Private” 
   
   
3. For each bucket you wish to add, type it in the “Bucket” text input field and click “Add”. 
   
4. Configure the bucket to have read-only access by ticking the “read-only” checkbox on the table. An unticked checkbox will configure the cluster to have read–write access to the bucket.
   
   
5. Follow the same procedure to add more buckets as you need. You can also remove buckets you no longer need from the list. 
6. Following the same steps, prepare the list of KMS Keys that you wish the cluster to use. You will need this if encryption is enabled on any of your buckets.
7. Click Apply” to submit the request when you are happy with the configuration. It may take a couple of minutes for your changes to be applied.
   
   
8. You will be able to use the buckets once the integration is in status “APPLIED”, you can view the integrated S3 buckets and KMS Keys by clicking “Configure” again.
   
   

To provide more context, below is the IAM Policy that will be attached to the cluster IAM user based on the example configuration above. 

Using the API

You can create, view, and remove your AWS Private S3 Bucket integrations using the Instaclustr API. An example for the POST endpoint is provided below.  

• Endpoint: https://api.instaclustr.com/cluster-management/v2/resources/applications/clickhouse/integrations/s3-private/v2 
• Payload:
  
  
  { "buckets": [ { "arn": "<s3-arn-1>", "readOnly": <true or false> }, { "arn": "<s3-arn-2>", "readOnly": <true or false> } ], "kmsKeyArns": [“<kms-key-arn-1>", “<kms-key-arn-2>”], "clusterId": "<clickhouse-cluster-id>" }

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  {     "buckets":       [         {             "arn": "<s3-arn-1>",             "readOnly": <true or false>         },         {             "arn": "<s3-arn-2>",             "readOnly": <true or false>         }     ],     "kmsKeyArns": [“<kms-key-arn-1>", “<kms-key-arn-2>”],     "clusterId": "<clickhouse-cluster-id>" }

• Example response:
  
  { "buckets": [ { "arn": "arn:aws:s3:::example-riyoa-bucket", "readOnly": true }, { "arn": "arn:aws:s3:::example-riyoa-bucket2", "readOnly": false } ], "clusterId": "68ae9de1-9acf-4d95-8d90-3f3d1b48a92a", "id": "privateS3-68ae9de1-9acf-4d95-8d90-3f3d1b48a92a", "kmsKeyArns": [ "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" ], "status": "NOT_APPLIED" }

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

  {     "buckets": [         {             "arn": "arn:aws:s3:::example-riyoa-bucket",             "readOnly": true         },         {             "arn": "arn:aws:s3:::example-riyoa-bucket2",             "readOnly": false         }     ],     "clusterId": "68ae9de1-9acf-4d95-8d90-3f3d1b48a92a",     "id": "privateS3-68ae9de1-9acf-4d95-8d90-3f3d1b48a92a",     "kmsKeyArns": [         "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"     ],     "status": "NOT_APPLIED" }

For details on all API endpoints, please refer to the Instaclustr API documentation. 

Using the Terraform Provider

You can manage your AWS Private S3 Bucket integrations using Terraform. The steps are as follows:  

1. Follow this support document to set up your Instaclustr Terraform Provider V2.  
2. Populate the resource template below with information about the integration. 
   
   
   resource "instaclustr_clickhouse_integration_private_s3_v2" "example" { cluster_id = "<clickhouse_cluster_id>" buckets { arn = "<bucket_arn_1>" read_only = <true or false> } buckets { arn = "<bucket_arn_2>" read_only = <true or false> } kms_key_arns = [ "<kms_key_arn_1>", "<kms_key_arn_2>" ] }

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

   resource "instaclustr_clickhouse_integration_private_s3_v2" "example" {     cluster_id = "<clickhouse_cluster_id>"     buckets {                arn = "<bucket_arn_1>"                read_only = <true or false>             }     buckets {                 arn = "<bucket_arn_2>"                 read_only = <true or false>             }     kms_key_arns = [             "<kms_key_arn_1>",             "<kms_key_arn_2>"      ] }

3. Follow the Terraform init, plan, and apply cycle to provision the integration.

For details of the Terraform resources, please refer to the Instaclustr Terraform documentation