Why Use the Kafka REST Proxy

Apache Kafka is known best as a powerful, open source message streaming and queueing solution. Used across more than 80% of the Fortune 500, Kafka is ubiquitous in supporting event-driven architectures.

At its most basic level, Kafka’s architecture consists of servers and clients, which communicate using a binary protocol over TCP. (A full protocol handshake is not completed because Kafka is designed to process data in a continuous stream.) 

Read Now: Apache Kafka Architecture – The complete guide.

Kafka can combine multiple servers together in a cluster, and clients allow you to read from and write to that cluster. Producers are clients that write events to the cluster, and Consumers read those events. 

The Java client comes as part of the core Kafka codebase, and there are numerous different Kafka clients available to accommodate additional programming languages, including Python, C/C++, Ruby, and Go. Kafka Streams is another client library that is available (read more about it on our comprehensive blog post on the topic here.)

But what if you want to use a language that there’s not a client for? Or if within your broader application stack, you want to be able to process data in Kafka in more of a request-response format in HTTP rather than as an event stream? That’s where the Kafka REST Proxy comes in. 

In this article, we’ll look at the Kafka REST Proxy and show you how to start using it with the Instaclustr Managed Platform.

What Is the Kafka REST Proxy?

The Kafka REST Proxy is a RESTful web API that allows your application to send and receive messages using HTTP rather than TCP. It can be used to produce data to and consume data from Kafka or for executing queries on cluster configuration. 

Whereas Apache Kafka Connect gives you pre-built connectors that allow you to push and pull data to Kafka without having to write any code, Kafka REST Proxy requires more hands-on configuration, and can be a solution when Kafka Connect and or existing client options don’t provide what you need. 

Setting Up the Kafka REST Proxy With Instaclustr Managed Kafka

Log in to the Instaclustr Console. If you’re not yet a customer, you can spin up a cluster for free in minutes here.

From the Console, click “Create” to start spinning up your new cluster. 

Then, name your cluster, select Apache Kafka as your application, and choose your preferred cloud provider:

On the next page, be sure to select Kafka REST Proxy as one of your enterprise add-ons, and then complete the rest of the steps in the cluster setup process in the Console. 

You can retrieve the information you need to connect your application to your new Kafka cluster via the Kafka REST Proxy by visiting the “Connection Info” page for your cluster within the Instaclustr Console. 

Formatting Your First Kafka REST Proxy Request

Each request to the REST Proxy API needs to go with a content type header that includes the serialization format, the API version, and the embedded format. If there is not a specific API version requirement, the API version can be omitted from the header. JSON is always used as the serialization format, while typical embedded formats include JSON, Avro, and binary. So for example, if you’re serializing the data in JSON, using v2 of the API, and embedding the data in JSON, your content header will look like this:

application/vnd.kafka.json.v2+json

You can push messages to your Kafka cluster with this POST request:

POST /topics/(string:topic_name)

Produce messages to one specific partition of a given topic:

POST /topics/(string:topic_name)/partitions/(int:partition_id)

Create a new consumer instance (the consumer needs to then keep communicating with the same Kafka REST instance that handled the request):

POST /consumers/(string:group_name)

You can also make a wide array of other POST and GET API calls with the Kafka REST Proxy. 

Downsides of the Kafka REST Proxy

A big disadvantage of the Kafka REST Proxy is that you do end up paying a performance penalty due to the extra processing steps involved to transform data between different formats. Published benchmarks suggest that the Kafka REST Proxy performs at 67% of write throughput and 50% of read throughput compared to standard clients. For use cases with mission-critical workloads and demanding performance requirements, it is better to go with a Kafka client if at all possible. 

Consumer state is also maintained with the REST proxy service, so if the node with the proxy service fails then the consumer state is lost and needs to be recreated. 

Conclusion

While it incurs a performance penalty, the Kafka REST Proxy can be appropriate for certain use cases, like accommodating programming languages without a standard Kafka client or processing data in a request-response format. To learn more about the Kafka REST Proxy, visit our documentation or sign up for a free trial of our platform to start exploring it for yourself.