How do I stream Kafka events to S3?

In this walkthrough, Kafka events are streamed to S3 by using the Kafka Connect Iceberg sink. Kafka Connect consumes records from the user-events topic and writes them into an Iceberg table stored in S3.

What is Apache Iceberg used for?

Apache Iceberg is used to add a table layer on top of files in S3. It tracks schema, metadata, partitions, and snapshots so query engines can work with the data as a table instead of a loose set of raw files.

How to configure Kafka Connect Iceberg sink?

You configure it by creating an Iceberg table first, building and uploading the connector, setting IAM permissions, creating the control topic, and then posting the connector configuration to the Kafka Connect REST API with the Glue catalog, warehouse path, topic, and security settings.

How do I query S3 data in real time?

In this setup, ClickHouse queries the Iceberg table directly from S3 using the iceberg() function. It reads Iceberg metadata first, then fetches only the files needed for the query.

Why use Kafka Connect instead of a Python consumer?

Kafka Connect handles distributed execution, offset management, worker failover, and operational reliability. A custom script may work for development, but Kafka Connect is designed to run the pipeline in production. .