AWS FSx for NetApp ONTAP Integration: Self-Managed FSx

Use this guide if you have an existing Amazon FSx for NetApp ONTAP file system created outside the Instaclustr platform and you want to integrate it with a NetApp Instaclustr-managed ClickHouse cluster for S3-compatible access.

NetApp Instaclustr-managed ClickHouse with FSx for ONTAP is a full lifecycle solution eliminating the self-managed burden for you. Refer to the NetApp Instaclustr-Managed FSx for ONTAP guide for instructions on integrating the NetApp Instaclustr fully managed solution.

Overview

By following this guide, you will connect your self‑managed Amazon FSx for NetApp ONTAP file system to a NetApp Instaclustr-managed ClickHouse cluster provisioned in an AWS RIYOA account, whether they reside in the same or different AWS accounts.  This guide provides instructions for: 

  • How to enable FSx for ONTAP integration via NetApp Instaclustr Console, API or Terraform 
  • How to self-manage FSx for ONTAP network configuration 

At the end of this process, your NetApp Instaclustr-managed ClickHouse cluster will have an active FSx for ONTAP integration and a Named Collection that can be used with the ClickHouse S3 table engine and S3 table functions to query data.  

Click here to learn more about AWS FSx for NetApp ONTAP integration, including how to use it for accessing file system data through ClickHouse. 

Before, proceeding, please note the following: 

Limitations 

  • Ensure that each NetApp Instaclustr-managed ClickHouse cluster and FSx for ONTAP uses a unique CIDR range. Overlapping CIDRs will prevent successful integration. 

Prerequisites 

  • The NetApp Instaclustr-managed ClickHouse cluster is provisioned in an AWS RIYOA provider account.   
  • NetApp Instaclustr-managed network setup: Please follow the RIYOA Setup Guide before proceeding. These guides are available in the NetApp Instaclustr Console and can be accessed by clicking Directory in the top-left corner, then expanding RIYOA Setup under Guides in the sidebar. 
  • Self managed network setup: If you prefer to manage the networking yourself, you can skip this step, but please note that you will be responsible for ensuring correct networking setup and security. For this, refer to the Manage FSx for ONTAP networking yourself section of this article below. 

How to Enable FSx for ONTAP Integration 

The following steps explain how to integrate an FSx for ONTAP, once you have provisioned a compatible NetApp Instaclustr-managed ClickHouse cluster. The integration can be enabled on the NetApp Instaclustr Console, API, or Terraform provider. 

Using the NetApp Instaclustr Console to enable FSx for ONTAP integration 

  1. In Console, expand the NetApp Instaclustr-managed ClickHouse cluster options on the sidebar and select the Integrations option.

  2. Click Configure for the row with type AWS S3 FSxN.

  3. On the Configure AWS S3 FSxN Integration page, select Your own file system to integrate your existing FSx for ONTAP file system. Enter the required file system details in the provided fields to connect it to your NetApp Instaclustr-managed ClickHouse cluster.
    • Using HTTPS is strongly recommended for ensuring communication between ClickHouse and FSx for ONTAP is encrypted. However if your S3 object store endpoint does not have HTTPS enabled, you can select HTTP as the connection Protocol.
    • Protocol setting can’t be changed via the Instaclustr Console after integration creation. To switch from HTTP to HTTPS, you will have to either delete and recreate the integration on the Instaclustr or contact NetApp Instaclustr Support for assistance.

      You will also need to configure whether your preference is for NetApp Instaclustr to manage your network configuration or whether you prefer to manage the networking configurations yourself. You can enable this option by selecting the relevant checkbox.

      For detailed guidance on configuring networking in your account after integration, please refer to the Manage FSx for ONTAP networking yourself section of this article below.
  4. Once you have ensured that all details are correct, click Add and Apply to apply the integration.
    Clicking Cancel will return to the Integrations page while discarding the integration.

  5. After returning to the Integrations page, the status for the AWS S3 FSxN integration row will indicate that your integration is being processed, and you will be momentarily blocked from accessing the configuration page.

  6. After processing has finished, you will be allowed to re-enter the configuration page.
  7. If you return to the S3 FSxN Integrations page, you will see all your existing integrations listed in a table, with their statuses indicated by the Status field. Clicking Delete for an integration will cause it to be deleted from the cluster.

    Note: If an integration shows a FAILED status, try deleting it and adding it again, making sure all details are entered correctly.
    If this still does not seem to resolve the issue, contact NetApp Instaclustr Support for assistance.

Once your integration has been applied, you can refer to How to Use ClickHouse S3 Table Engine with FSx for ONTAP for guidance on using your integration.

Using the NetApp Instaclustr API to enable FSx for ONTAP integration

You can create, view and remove your FSx for ONTAP integrations using the NetApp Instaclustr API. An example payload for the POST endpoint is provided below.  

Please refer to the endpoint documentation page for further details on all configurable parameters. 

Endpoint:

Payload:

  • The httpsEnabled field controls the protocol used to communicate with your FSx for ONTAP file system. It defaults to true when not specified. When httpsEnabled is true, the caCertificate field is required and must contain the PEM-encoded root CA certificate for your S3 endpoint hosted on FSx for ONTAP. 
  • Using HTTPS is strongly recommended for ensuring communication between ClickHouse and FSx for ONTAP is encrypted. To use HTTP instead (since your S3 object store endpoint does not have HTTPS enabled), set httpsEnabled to false and omit the caCertificate field. . 
  • The settings for httpsEnabled and caCertificate can’t be changed once the integration has been created. If the integration was initially configured with httpsEnabled set to false and you now wish to enable it, you may either delete and recreate the integration or contact NetApp Instaclustr Support for assistance.
  • (Optional) If you want to manage FSx for ONTAP networking yourself and your FSx for ONTAP file system and NetApp Instaclustr-managed ClickHouse cluster reside in different AWS accounts, include the parameter below:

When selfManagedNetwork is set to true, you must omit the providerAccountName and region fields. 

For further details on NetApp Instaclustr API endpoints, refer to the NetApp Instaclustr API documentation. 

Using the NetApp Instaclustr Terraform Provider

You can manage your FSx for ONTAP integrations using NetApp Instaclustr Terraform. The steps are as follows: 

  1. Follow this support document to set up your NetApp Instaclustr Terraform Provider V2.
  2. Populate the resource template below with the required integration details:

  • Setting https_enabled = true (the default) requires ca_certificate to be provided with the PEM-encoded root CA certificate for your S3 endpoint. 
  • Using https_enabled = true is strongly recommended to ensure communication between ClickHouse and FSx for ONTAP is encrypted. To use HTTP instead (since your S3 object store endpoint does not have HTTPS enabled), set https_enabled = false and omit ca_certificate.. 
  • The settings for https_enabled and ca_certificate can’t be changed once the integration has been created. If the integration was initially configured with https_enabled = false and you now wish to enable it, you may either delete and recreate the integration or contact NetApp Instaclustr Support for assistance. 
  • (Optional) If you want to manage FSx for ONTAP networking yourself and your FSx for ONTAP file system and NetApp Instaclustr-managed ClickHouse cluster reside in different AWS accounts, include self_managed_network = true within the fsxn_filesystem block. When self_managed_network is set to true, you must omit the provider_account_name and region fields.

Refer to the resource documentation page for details on all configurable parameters. 

Once your integration has been applied, you can refer to How to Use ClickHouse S3 Table Engine with FSx for ONTAP for guidance on using your integration. 

How to Self-Manage FSx for ONTAP Network Configuration

Note: This section is relevant only if you:  

  1. have your own FSx for NetApp ONTAP file system in a different AWS Account from your NetApp Instaclustr-managed ClickHouse Cluster; and
  2. want to manage the networking configuration yourself to connect it to a NetApp Instaclustr-managed ClickHouse cluster. 

Proper networking configuration is required for communication between your NetApp Instaclustr-managed ClickHouse cluster and the FSx for ONTAP. To enable connectivity, you’ll need to set up an active VPC peering connection and ensure the correct routing tables are in place between your FSx for ONTAP VPC and the NetApp Instaclustr-managed ClickHouse cluster VPC.  

You can follow this guide for step‑by‑step instructions on how to create, update, and delete VPC peering connections.  

Create and Attach an AWS Security Group

To allow communication between your NetApp Instaclustr-managed ClickHouse cluster and FSx for ONTAP file system, an appropriate security group must be created and attached to your FSx for ONTAP resource. This security group should be configured to allow inbound traffic from the ClickHouse cluster on the required ports.

ClickHouse uses port 80 for HTTP and port 443 for HTTPS connections. As such, ensure port 443 is open to inbound traffic from the ClickHouse cluster CIDR if you are using HTTPS, or port 80 if you are using HTTP instead. 

  1. Log into your AWS Account where your FSx for ONTAP resides. 
  2. Select the Region of your FSx for ONTAP, then search for Security Groups and click the option labelled VPC feature.

  3. Click Create security group.

  4. In the field Security group name, enter a name for the security group.
    Note that this name must be unique for the associated VPC and cannot be modified after creation of the group, so it is recommended to use a naming convention which allows easy identification. 
  5. In the field Description, enter a short description of the security group (for example, “Allows access to FSx for ONTAP from NetApp Instaclustr-managed ClickHouse”). 
  6. In the field VPC, search for and select the VPC of your FSx for ONTAP. 
  7. Under the heading Inbound rules, click Add rule. The rules created here permit inbound traffic to the FSx for ONTAP from the cluster.

    Select Custom TCP in the Type field.  Enter 80 in the Port range field. Select Custom in the Source field, then enter the CIDR of your NetApp Instaclustr-managed ClickHouse cluster.Optionally, add a description in the Description field.  
  1. Repeat Step 6 to create a second rule for port 443.
  2. Confirm your security group configuration matches the sample below. Click Create security group.
  3. Attach the security group to the relevant network interfaces of your FSx for ONTAP file system. After completing these steps, your network setup will be ready.

Questions

Please contact NetApp Instaclustr Support for any further inquiries.