# How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline
[Blog](/blog/)>[Technology](/blog/category/technical/)>How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline
How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline
=======================================================================================================================
April 08, 2026 | By [ Walt Ribeiro](https://www.instaclustr.com/blog/author/waltribeiro/)
[ ](https://x.com/intent/tweet?text=How%20to%20build%20a%20streaming%20analytics%20pipeline%20with%20Terraform%20and%20Instaclustr%E2%80%94Part%202:%20Designing%20the%20complete%20data%20pipeline&url=https://www.instaclustr.com/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-2-designing-the-complete-data-pipeline/) [ ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.instaclustr.com/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-2-designing-the-complete-data-pipeline/&title=&summary=How%20to%20build%20a%20streaming%20analytics%20pipeline%20with%20Terraform%20and%20Instaclustr%E2%80%94Part%202:%20Designing%20the%20complete%20data%20pipeline&source=)
Where we left off
-----------------
In [part 1](/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-1-setting-up-your-first-kafka-cluster/), we deployed a managed Apache Kafka cluster with Terraform. That gave us a reliable message broker, but a broker sitting alone isn’t a pipeline—it’s just infrastructure waiting for a purpose.
Today, we add the pieces that turn Kafka into a real-time analytics system:
- **ClickHouse®:** A columnar database that can query hundreds of millions of rows in milliseconds
- **Kafka® Connect:** A distributed framework that streams data between systems without writing code
By the end, you’ll have three clusters talking to each other, all deployed from a single Terraform configuration.
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
Why this architecture?
----------------------
Before we write code, let’s understand why this specific combination matters.
### The problem
Imagine you’re processing user events—clicks, purchases, and page views. Kafka handles the ingestion beautifully: millions of events per second, no problem. But then someone asks, “How many users from California purchased something in the last hour?”
Kafka excels at moving data, not answering questions about it. That’s because it’s a streaming log optimize for throughput and durability—not ad hoc queries. For example, to ask “what happened last Tuesday?” you would need to land that data in a system built for analysis. That’s where ClickHouse comes in.
### The traditional solution (and why it’s painful)
Most teams solve this with custom code: a consumer application that reads from Kafka, transforms the data, and writes to a database. It works, but now you’re maintaining:
- A consumer application
- Connection pooling
- Retry logic
- Schema evolution handling
- Monitoring and alerting
- Deployment pipelines for all of the above
That’s a lot of code for “move data from A to B.”
### The better way: Kafka Connect
Kafka Connect exists precisely for this use case. Instead of writing custom consumers, you deploy *connectors*—pre-built plugins that handle the heavy lifting. Want data in ClickHouse? Deploy a ClickHouse sink connector. Need data from PostgreSQL? Deploy a PostgreSQL source connector.
No custom code. No deployment pipelines for data movement. Just configuration.
Who uses this pattern?
----------------------
This isn’t experimental. Major companies run this exact architecture at scale:
- Cloudflare processes over [6 million HTTP requests per second](https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/) through their ClickHouse-powered analytics pipeline
- Uber runs over [a trillion Kafka messages daily](https://www.uber.com/blog/athenax/) for ride matching, pricing, and analytics
- Spotify streams [hundreds of billions of events daily](https://engineering.atspotify.com/2017/10/big-data-processing-at-spotify-the-road-to-scio-part-1/) through their data infrastructure to power recommendations
The pattern works because it separates concerns: Kafka handles ingestion, Connect handles movement, ClickHouse handles queries. Each component does one thing well.
The architecture
----------------
Here’s what we’re building:
Three clusters, each in its own VPC:
**Cluster** **Network** **Purpose** Kafka 10.0.0.0/16 Message broker for event ingestion Kafka Connect 10.5.0.0/16 Streams data from Kafka to ClickHouse ClickHouse 10.6.0.0/16 Columnar database for analytics The NetApp Instaclustr does support a number of deployment models, such as co-locating these clusters inside the same VPC—but for simplicity today, let’s put them into separate VPC’s.
The firewall rules create a secure data flow: Kafka Connect can reach both Kafka and ClickHouse, but they can’t reach each other directly. Why does this matter? Each component only has access to what it needs. Kafka does not need to query ClickHouse. ClickHouse doesn’t need to read from Kafka (because that’s Connect’s job). By not opening unnecessary paths, you limit the blast radius if something goes wrong.
For example, if an attacker compromises ClickHouse, they can’t pivot directly to Kafka— they would have to go through Connect first, which is an additional barrier.
The complete Terraform configuration
------------------------------------
This builds on [part 1’s](/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-1-setting-up-your-first-kafka-cluster/) foundation. The code below includes everything from part one, plus new resources noted with `#ClickHouse` and `#Kafka Connect` comments.
Create a new directory with these files:
### main.tf
terraform { required\_providers { instaclustr = { source = "instaclustr/instaclustr" version = "~> 2.0" } } } # ================================================================= # Variables # ================================================================= variable "instaclustr\_terraform\_key" { description = "Instaclustr API key" type = string sensitive = true } variable "my\_ip\_address" { description = "Your IP address for firewall rules (CIDR format)" type = string } # ================================================================= # Provider # ================================================================= provider "instaclustr" { terraform\_key = var.instaclustr\_terraform\_key } # ================================================================= # Kafka Cluster # ================================================================= resource "instaclustr\_kafka\_cluster\_v3" "kafka" { name = "pipeline-kafka" description = "Kafka for data pipeline" kafka\_version = "4.1.1" sla\_tier = "NON\_PRODUCTION" auto\_create\_topics = true allow\_delete\_topics = true client\_to\_cluster\_encryption = false private\_network\_cluster = false pci\_compliance\_mode = false default\_number\_of\_partitions = 3 default\_replication\_factor = 3 data\_centre { cloud\_provider = "AWS\_VPC" name = "AWS\_VPC\_US\_EAST\_1" network = "10.0.0.0/16" region = "US\_EAST\_1" number\_of\_nodes = 3 node\_size = "KFK-DEV-t4g.small-5" } } # ================================================================= # ClickHouse Cluster # ================================================================= resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" { name = "pipeline-clickhouse" description = "ClickHouse for real-time analytics" clickhouse\_version = "25.8.11" sla\_tier = "NON\_PRODUCTION" private\_network\_cluster = false data\_centre { cloud\_provider = "AWS\_VPC" name = "AWS\_VPC\_US\_EAST\_1" network = "10.6.0.0/16" region = "US\_EAST\_1" node\_size = "CLK-DEV-m7i.large-50" shards = 1 replicas = 3 } } # ================================================================= # Kafka Connect Cluster # ================================================================= resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" { name = "pipeline-connect" description = "Kafka Connect for streaming to ClickHouse" kafka\_connect\_version = "4.1.1" sla\_tier = "NON\_PRODUCTION" private\_network\_cluster = false data\_centre { cloud\_provider = "AWS\_VPC" name = "AWS\_VPC\_US\_EAST\_1" network = "10.2.0.0/16" region = "US\_EAST\_1" number\_of\_nodes = 3 node\_size = "KCN-DEV-t4g.medium-30" replication\_factor = 3 } target\_cluster { managed\_cluster { kafka\_connect\_vpc\_type = "KAFKA\_VPC" target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id } } } # ================================================================= # Firewall Rules # ================================================================= # Kafka: Allow Kafka Connect + your IP resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "kafka\_firewall" { cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id firewall\_rule { network = "10.2.0.0/16" # Kafka Connect network type = "KAFKA" } firewall\_rule { network = var.my\_ip\_address type = "KAFKA" } } # ClickHouse: Allow Kafka Connect + your IP resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "clickhouse\_firewall" { cluster\_id = instaclustr\_clickhouse\_cluster\_v2.clickhouse.id firewall\_rule { network = "10.2.0.0/16" type = "CLICKHOUSE" } firewall\_rule { network = "10.2.0.0/16" type = "CLICKHOUSE\_WEB" } firewall\_rule { network = var.my\_ip\_address type = "CLICKHOUSE" } firewall\_rule { network = var.my\_ip\_address type = "CLICKHOUSE\_WEB" } } # Kafka Connect: Allow your IP resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "connect\_firewall" { cluster\_id = instaclustr\_kafka\_connect\_cluster\_v2.connect.id firewall\_rule { network = var.my\_ip\_address type = "KAFKA\_CONNECT" } } # ================================================================= # Outputs # ================================================================= # Kafka output "kafka\_cluster\_id" { value = instaclustr\_kafka\_cluster\_v3.kafka.id } output "kafka\_cluster\_status" { value = instaclustr\_kafka\_cluster\_v3.kafka.status } output "kafka\_bootstrap\_servers" { value = join(",", \[for node in instaclustr\_kafka\_cluster\_v3.kafka.data\_centre\[0\].nodes : "${node.public\_address}:9092"\]) } # ClickHouse output "clickhouse\_cluster\_id" { value = instaclustr\_clickhouse\_cluster\_v2.clickhouse.id } output "clickhouse\_cluster\_status" { value = instaclustr\_clickhouse\_cluster\_v2.clickhouse.status } output "clickhouse\_endpoints" { value = \[for node in instaclustr\_clickhouse\_cluster\_v2.clickhouse.data\_centre\[0\].nodes : node.public\_address\] } output "clickhouse\_web\_ui" { value = \[for node in instaclustr\_clickhouse\_cluster\_v2.clickhouse.data\_centre\[0\].nodes : "https://${node.public\_address}:8443"\] } # Kafka Connect output "kafka\_connect\_cluster\_id" { value = instaclustr\_kafka\_connect\_cluster\_v2.connect.id } output "kafka\_connect\_cluster\_status" { value = instaclustr\_kafka\_connect\_cluster\_v2.connect.status } output "kafka\_connect\_endpoints" { value = \[for node in instaclustr\_kafka\_connect\_cluster\_v2.connect.data\_centre\[0\].nodes : "${node.public\_address}:8083"\] }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
terraform {
required\_providers {
instaclustr = {
source = "instaclustr/instaclustr"
version = "~> 2.0"
}
}
}
\# =================================================================
\# Variables
\# =================================================================
variable "instaclustr\_terraform\_key" {
description = "Instaclustr API key"
type = string
sensitive = true
}
variable "my\_ip\_address" {
description = "Your IP address for firewall rules (CIDR format)"
type = string
}
\# =================================================================
\# Provider
\# =================================================================
provider "instaclustr" {
terraform\_key = var.instaclustr\_terraform\_key
}
\# =================================================================
\# Kafka Cluster
\# =================================================================
resource "instaclustr\_kafka\_cluster\_v3" "kafka" {
name = "pipeline-kafka"
description = "Kafka for data pipeline"
kafka\_version = "4.1.1"
sla\_tier = "NON\_PRODUCTION"
auto\_create\_topics = true
allow\_delete\_topics = true
client\_to\_cluster\_encryption = false
private\_network\_cluster = false
pci\_compliance\_mode = false
default\_number\_of\_partitions = 3
default\_replication\_factor = 3
data\_centre {
cloud\_provider = "AWS\_VPC"
name = "AWS\_VPC\_US\_EAST\_1"
network = "10.0.0.0/16"
region = "US\_EAST\_1"
number\_of\_nodes = 3
node\_size = "KFK-DEV-t4g.small-5"
}
}
\# =================================================================
\# ClickHouse Cluster
\# =================================================================
resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" {
name = "pipeline-clickhouse"
description = "ClickHouse for real-time analytics"
clickhouse\_version = "25.8.11"
sla\_tier = "NON\_PRODUCTION"
private\_network\_cluster = false
data\_centre {
cloud\_provider = "AWS\_VPC"
name = "AWS\_VPC\_US\_EAST\_1"
network = "10.6.0.0/16"
region = "US\_EAST\_1"
node\_size = "CLK-DEV-m7i.large-50"
shards = 1
replicas = 3
}
}
\# =================================================================
\# Kafka Connect Cluster
\# =================================================================
resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" {
name = "pipeline-connect"
description = "Kafka Connect for streaming to ClickHouse"
kafka\_connect\_version = "4.1.1"
sla\_tier = "NON\_PRODUCTION"
private\_network\_cluster = false
data\_centre {
cloud\_provider = "AWS\_VPC"
name = "AWS\_VPC\_US\_EAST\_1"
network = "10.2.0.0/16"
region = "US\_EAST\_1"
number\_of\_nodes = 3
node\_size = "KCN-DEV-t4g.medium-30"
replication\_factor = 3
}
target\_cluster {
managed\_cluster {
kafka\_connect\_vpc\_type = "KAFKA\_VPC"
target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id
}
}
}
\# =================================================================
\# Firewall Rules
\# =================================================================
\# Kafka: Allow Kafka Connect + your IP
resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "kafka\_firewall" {
cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id
firewall\_rule {
network = "10.2.0.0/16" \# Kafka Connect network
type = "KAFKA"
}
firewall\_rule {
network = var.my\_ip\_address
type = "KAFKA"
}
}
\# ClickHouse: Allow Kafka Connect + your IP
resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "clickhouse\_firewall" {
cluster\_id = instaclustr\_clickhouse\_cluster\_v2.clickhouse.id
firewall\_rule {
network = "10.2.0.0/16"
type = "CLICKHOUSE"
}
firewall\_rule {
network = "10.2.0.0/16"
type = "CLICKHOUSE\_WEB"
}
firewall\_rule {
network = var.my\_ip\_address
type = "CLICKHOUSE"
}
firewall\_rule {
network = var.my\_ip\_address
type = "CLICKHOUSE\_WEB"
}
}
\# Kafka Connect: Allow your IP
resource "instaclustr\_cluster\_network\_firewall\_rules\_v2" "connect\_firewall" {
cluster\_id = instaclustr\_kafka\_connect\_cluster\_v2.connect.id
firewall\_rule {
network = var.my\_ip\_address
type = "KAFKA\_CONNECT"
}
}
\# =================================================================
\# Outputs
\# =================================================================
\# Kafka
output "kafka\_cluster\_id" {
value = instaclustr\_kafka\_cluster\_v3.kafka.id
}
output "kafka\_cluster\_status" {
value = instaclustr\_kafka\_cluster\_v3.kafka.status
}
output "kafka\_bootstrap\_servers" {
value = join(",", \[for node in instaclustr\_kafka\_cluster\_v3.kafka.data\_centre\[0\].nodes : "${node.public\_address}:9092"\])
}
\# ClickHouse
output "clickhouse\_cluster\_id" {
value = instaclustr\_clickhouse\_cluster\_v2.clickhouse.id
}
output "clickhouse\_cluster\_status" {
value = instaclustr\_clickhouse\_cluster\_v2.clickhouse.status
}
output "clickhouse\_endpoints" {
value = \[for node in instaclustr\_clickhouse\_cluster\_v2.clickhouse.data\_centre\[0\].nodes : node.public\_address\]
}
output "clickhouse\_web\_ui" {
value = \[for node in instaclustr\_clickhouse\_cluster\_v2.clickhouse.data\_centre\[0\].nodes : "https://${node.public\_address}:8443"\]
}
\# Kafka Connect
output "kafka\_connect\_cluster\_id" {
value = instaclustr\_kafka\_connect\_cluster\_v2.connect.id
}
output "kafka\_connect\_cluster\_status" {
value = instaclustr\_kafka\_connect\_cluster\_v2.connect.status
}
output "kafka\_connect\_endpoints" {
value = \[for node in instaclustr\_kafka\_connect\_cluster\_v2.connect.data\_centre\[0\].nodes : "${node.public\_address}:8083"\]
}
### terraform.tfvars
instaclustr\_terraform\_key = "your-api-key-here" my\_ip\_address = "YOUR.IP.ADDRESS.HERE/32"
1
2
instaclustr\_terraform\_key = "your-api-key-here"
my\_ip\_address = "YOUR.IP.ADDRESS.HERE/32"
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
Understanding the new components
--------------------------------
### ClickHouse: The analytics engine
resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" { name = "pipeline-clickhouse" clickhouse\_version = "25.8.11" ... data\_centre { network = "10.6.0.0/16" shards = 1 replicas = 3 } }
1
2
3
4
5
6
7
8
9
10
resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" {
name = "pipeline-clickhouse"
clickhouse\_version = "25.8.11"
...
data\_centre {
network = "10.6.0.0/16"
shards = 1
replicas = 3
}
}
ClickHouse is a columnar database, which means it stores data by column rather than by row. This makes it extraordinarily fast for analytical queries—the kind that scan millions of rows but only need a few columns.
A few configuration notes:
- **`shards = 1`**: We’re using a single shard for simplicity. Production deployments shard data across multiple nodes for horizontal scaling.
- **`replicas = 3`**: Three copies of the data for fault tolerance. If a node fails, queries keep working.
- **`network = "10.6.0.0/16"`**: A separate CIDR block from Kafka. Each cluster lives in isolation.
resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" { name = "pipeline-clickhouse" clickhouse\_version = "25.8.11" ... data\_centre { network = "10.6.0.0/16" shards = 1 replicas = 3 } }
1
2
3
4
5
6
7
8
9
10
resource "instaclustr\_clickhouse\_cluster\_v2" "clickhouse" {
name = "pipeline-clickhouse"
clickhouse\_version = "25.8.11"
...
data\_centre {
network = "10.6.0.0/16"
shards = 1
replicas = 3
}
}
### Kafka Connect: The data bridge
resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" { name = "pipeline-connect" kafka\_connect\_version = "4.1.1" ... target\_cluster { managed\_cluster { kafka\_connect\_vpc\_type = "KAFKA\_VPC" target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id } } }
1
2
3
4
5
6
7
8
9
10
11
resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" {
name = "pipeline-connect"
kafka\_connect\_version = "4.1.1"
...
target\_cluster {
managed\_cluster {
kafka\_connect\_vpc\_type = "KAFKA\_VPC"
target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id
}
}
}
The critical piece here is the `target_cluster` block. This tells Kafka Connect which Kafka cluster to connect to. Notice the Terraform reference:
target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" { name = "pipeline-connect" kafka\_connect\_version = "4.1.1" ... target\_cluster { managed\_cluster { kafka\_connect\_vpc\_type = "KAFKA\_VPC" target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id } } }
1
2
3
4
5
6
7
8
9
10
11
12
13
target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id
resource "instaclustr\_kafka\_connect\_cluster\_v2" "connect" {
name = "pipeline-connect"
kafka\_connect\_version = "4.1.1"
...
target\_cluster {
managed\_cluster {
kafka\_connect\_vpc\_type = "KAFKA\_VPC"
target\_kafka\_cluster\_id = instaclustr\_kafka\_cluster\_v3.kafka.id
}
}
}
This creates an implicit dependency: Terraform won’t create the Connect cluster until Kafka exists. You don’t need to manage ordering manually—Terraform figures it out from the references.
The `kafka_connect_vpc_type = "KAFKA_VPC"` setting places Kafka Connect in the same VPC as Kafka, enabling private communication without traversing the public internet.
### Firewall rules: Security by design
Look at how the firewall rules create a directed data flow:
\# Kafka allows traffic from Kafka Connect's network firewall\_rule { network = "10.2.0.0/16" # Kafka Connect type = "KAFKA" } # ClickHouse allows traffic from Kafka Connect's network firewall\_rule { network = "10.2.0.0/16" # Kafka Connect type = "CLICKHOUSE" }
1
2
3
4
5
6
7
8
9
10
11
\# Kafka allows traffic from Kafka Connect's network
firewall\_rule {
network = "10.2.0.0/16" \# Kafka Connect
type = "KAFKA"
}
\# ClickHouse allows traffic from Kafka Connect's network
firewall\_rule {
network = "10.2.0.0/16" \# Kafka Connect
type = "CLICKHOUSE"
}
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
*Kafka Connect is the only thing that needs to talk to both systems.*
Deploy the pipeline
-------------------
Running these commands will provision all 3 clusters, your pipeline VPC, subnets, route tables, and an EC2 test instance pre-loaded with the Kafka CLI. Think of the Terraform configuration as a recipe—it describes exactly what you want built, and these commands hand that recipe to AWS and Instaclustr to provision everything.
terraform init terraform plan terraform apply
1
2
3
terraform init
terraform plan
terraform apply
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
This deployment takes 20-30 minutes. Terraform creates the resources in dependency order: Kafka first, then ClickHouse and Kafka Connect in parallel, then all the firewall rules.
When complete, you’ll see outputs for all three clusters:
kafka\_bootstrap\_servers = "100.29.129.26:9092,98.95.193.141:9092,100.51.10.12:9092" clickhouse\_endpoints = \["35.168.237.87", "34.226.159.233", "44.217.57.142"\] kafka\_connect\_endpoints = \["34.237.72.188:8083", "44.205.132.50:8083", "3.225.226.67:8083"\]
1
2
3
kafka\_bootstrap\_servers = "100.29.129.26:9092,98.95.193.141:9092,100.51.10.12:9092"
clickhouse\_endpoints = \["35.168.237.87", "34.226.159.233", "44.217.57.142"\]
kafka\_connect\_endpoints = \["34.237.72.188:8083", "44.205.132.50:8083", "3.225.226.67:8083"\]
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
Verify the deployment
---------------------
### Check Kafka Connect
Kafka Connect exposes a REST API. Verify it’s running:
curl https://<CONNECT\_ENDPOINT>/connectors
1
curl https://<CONNECT\_ENDPOINT>/connectors
You should see an empty array `[]` — no connectors deployed yet, but the cluster is healthy.
### Check ClickHouse
Connect using the ClickHouse client:
clickhouse-client --host <CLICKHOUSE\_ENDPOINT> \\ --user icclickhouse \\ --password <PASSWORD\_FROM\_CONSOLE> \\ --secure
1
2
3
4
clickhouse-client --host <CLICKHOUSE\_ENDPOINT> \\
--user icclickhouse \\
--password <PASSWORD\_FROM\_CONSOLE> \\
--secure
Run a test query:
SELECT version();
1
SELECT version();
You should see the ClickHouse version confirming the connection works.
Handling Terraform state issues
-------------------------------
During deployment, you might encounter a 404 error if a cluster is recreated:
![How to build a streaming analytics pipeline with Terraform and Instaclustr—Part 2: Designing the complete data pipeline screenshot]()
This happens when Terraform’s state file references a resource that no longer exists in Instaclustr. The fix is to remove the orphaned reference from state:
terraform state rm instaclustr\_cluster\_network\_firewall\_rules\_v2.kafka\_firewall
1
terraform state rm instaclustr\_cluster\_network\_firewall\_rules\_v2.kafka\_firewall
Then run `terraform apply` again. Terraform will recreate the firewall rules for the current cluster.
This is a normal part of working with Terraform and external providers. The state file is Terraform’s view of reality—sometimes it needs correction.
What we’ve built
----------------
At this point, you have:
- A 3-broker Kafka cluster for event ingestion
- A 3-replica ClickHouse cluster for analytics
- A 3-worker Kafka Connect cluster ready for connectors
- Firewall rules securing communication between them
All from ~150 lines of Terraform.
The infrastructure is ready. In [part 3](/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-3-integrating-with-aws-vpc/), we’ll add AWS VPC integration and switch Kafka Connect to `VPC_PEERED` mode – giving each cluster its own network so that we can connect your own AWS resources to all 3 over private networking.
Clean up
--------
If you’re done for now, run:
terraform destroy
1
terraform destroy
This removes all three clusters and their firewall rules. When you’re ready to continue with [part 3](/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-3-integrating-with-aws-vpc/), run `terraform apply` again with the new settings we’ll discuss in part 3.
Key takeaways
-------------
1. **Kafka Connect eliminates custom consumer code**—use connectors instead of writing data movement logic
2. **Each cluster gets its own network**—isolation by design, with firewall rules controlling communication
3. **Terraform references create implicit dependencies**—the `target_kafka_cluster_id` reference ensures correct ordering
4. **ClickHouse is purpose-built for analytics**—columnar storage makes it orders of magnitude faster for aggregation queries than row-based databases
Why managed infrastructure? Building a streaming pipeline is one challenge. Keeping it running in production is another. Running Kafka, ClickHouse, and Kafka Connect yourself means your team owns the operations too—version upgrades, security patches, and on-call rotations. Instaclustr handles that so your team can stay focused on what the pipeline does, not on keeping it alive. The Terraform configuration in this series is a good example of that division: you simply describe what you want, and Instaclustr provisions and operates it from there.
See you in [part 3](/blog/how-to-build-a-streaming-analytics-pipeline-with-terraform-and-instaclustr-part-3-integrating-with-aws-vpc/), where we connect all of this to your AWS infrastructure, move the entire pipeline into your own AWS account, and connect everything over private networking.
### About the author
**[Walt Ribeiro](https://www.instaclustr.com/blog/author/waltribeiro/)** | Developer Advocate
I’m a Developer Advocate at Instaclustr focused on open-source data engineering - particularly ClickHouse, Apache Iceberg, and Kafka. I create technical content, video guides, and hands-on tutorials to help developers build modern data pipelines.
![mail icon]()#### Get the latest articles for open sourceIn your inbox
### Related content
[ Zero Downtime Migration to Instaclustr
Yes, we can migrate existing Cassandra clusters to Instaclustr without any downtime. Here's what to expect from the process...
](https://www.instaclustr.com/blog/zero-downtime-migration-to-instaclustr/)
[ Workflow Comparison: Uber Cadence vs Netflix Conductor
When choosing what’s right for your company’s opensource workflow needs it is important to know the difference and similarities ...
](https://www.instaclustr.com/blog/workflow-comparison-uber-cadence-vs-netflix-conductor/)
[ Will Your Cassandra Database Project Succeed?: The New Stack
Open source Apache Cassandra® continues to stand out as an enterprise-proven solution for organizations seeking high availability...
](https://www.instaclustr.com/blog/will-your-cassandra-database-project-succeed-the-new-stack/)
×Sign upto ourNewsletter
-----------------------