Technical — Kafka Monday 11th October 2021

Apache Kafka vs Microsoft Azure Service Bus

By Ritam Das

In the ever-growing field of big data, Apache Kafka has become a widely adopted distributed streaming platform. From message queuing to event streaming, Apache Kafka delivers on a wide range of use cases. Its high throughput, scalability, and fault tolerance make it the clear-cut choice as an open source event handler. While there can be complexity in adopting and operating Kafka, a company like Instaclustr with our Managed Platform and consulting services can help to make adoption easy and smooth.

With that being said, there are alternative technologies that are sometimes considered when evaluating Kafka. In this blog, we will look specifically at one of those alternatives, Azure Service Bus. 

Are Azure Service Bus and Kafka the Same? 

In short, no, but they do share some similar characteristics. It is important to note that Azure Service Bus is a traditional message broker and tailored to somewhat different use cases when compared to Kafka. Simply transferring between these two technologies is not an easy feat and would require overhauling your entire application. The comparison stops at both technologies being message brokers as under the hood they are fundamentally different. 

At a high level, ASB has high processing overhead per message, stronger guarantees around delivery and processing, and typically a “process once” model. Kafka has low overhead processing per message, fewer guarantees around delivery and processing, and typically a “publish once, process multiple times” model. To provide an explicit comparison, it would be best to understand the intended use case and proceed from there. 

Consider the following recommendations for determining which technology fits your purpose:

When to Consider Kafka:

Note that Kafka is very flexible as it can behave like a queue or message bus, but it is way more than that. It’s also an event-driven streaming platform. As a solutions architect, consider using Kafka when:

  • Your application requires real-time data processing (finance domains, IoT devices, etc.)
  • Your solution requires application activity tracking (Kafka’s original purpose when developed by LinkedIn)
  • Your solution requires a guaranteed first-in-first-out (FIFO) ordered delivery
  • Your solution needs to support automatic duplicate detection
  • Your solution needs to group messages into transactions
  • You have a requirement to provide role-based access control to the topics, and different rights/permissions for producers and consumers 
  • Your application requires logging and/or monitoring
  • Your application processes many messages per day (Kafka is an infinite data stream) or experiences high workload (with message sizes up to 1MB as a best practice) 
  • Your application requires specified data retention periods and not long term storage
  • Your application requires the flexibility and scalability of a multi-subscriber pub/sub system 
  • Your application needs to publish and consume batches of messages

When to Consider Azure Service Bus:

  • Your solution needs to receive messages without having to poll the queue
  • Your solution requires a guaranteed first-in-first-out (FIFO) ordered delivery
  • Your solution needs to support automatic duplicate detection
  • Your solution needs to group messages into transactions
  • Your application handles messages that fall in the size range of 64KB to 256KB
  • You have a requirement to provide role-based access control to the queues, and different rights/permissions for producers and consumers 
  • Your queue size won’t grow larger than 80GB (limitation on messages held in queue)
  • Your messaging solution needs to have strong delivery guarantees without the need to build the additional infrastructure components
  • Your solution needs to publish and consume batches of messages

Apache Kafka vs Azure Service Bus

The following tables detail specific areas of comparison between pure open source Apache Kafka vs Azure Service Bus:

  FeaturesOpen Source Apache KafkaAzure Service Bus
Core DatabaseApache Kafka100% Open SourceProprietary
HostingAnywhereAzure only
Use CasesMessagingActivity TrackingMetrics/MonitoringLog AggregationStream ProcessingEvent SourcingCommit LogMessagingMessage QueuingIoT IngestionMetrics/Monitoring
QueuesYesBasic Tier
Scheduled  MessagesYesBasic Tier
TopicsYesStandard and Premium Tier
TransactionsYesStandard and Premium Tier
Duplicate DetectionAutomaticAutomatic (Standard and Premium Tier Only)
SessionsYesStandard and Premium Tier
Message SizeNo Limit(Preferably 1MB)256 Kb/1 MB (Tier-based)
Resource IsolationYesPremium Tier Only
Purging  Topics/QueuesYesNo
Max Queue Size/Message LimitInfinite Data Stream1 GB to 80 GB
Brokered Connections~10k1000 per Management Unit
Retention PolicyNo Set MaximumUp to 30 Days, Depends on Tier
TenancySingle TenancyMulti-Tenancy
On-Prem SolutionsYesNo
Multi RegionYesYes
Multi CloudYesNo
Log CompactionYesNo
Support*Has OS CommunityNo
Availability SLA*<99.9%

*Instaclustr provides 24x7x365 support with 15 minute response time and 99.99% availability SLAs for our Managed Kafka solution. 

Azure Service Bus tiers add complexity and involve quotas, usage limits, and throughput-based pricing, making scaling difficult and cost inefficient. Locking into a vendor such as Azure can have negative future implications when thinking about scalability and the rapidly shifting environment of cloud computing. Scaling with pure Apache Kafka is easy and 100% open source, allowing your business to adapt to any situation quickly.

Sample Connection Code for Azure Service Bus

Sample code snippet to send message to topic (send_topic.py):

Sample code snippet to send messages from topics to queues (send_queue.py): 

Sample Connection Code for Apache Kafka

Here we create a consumer, producer, and topic. This is a simple working example of sending a message from a producer to a topic and then to a consumer. (Simplistic setup ready to consume and produce messages as well as view messages within topic.):

Conclusion

In this post, we discussed key areas of distinction and comparison between pure open source Apache Kafka and Azure Service Bus as stream processing and message broker platforms. We also discussed key considerations to bear in mind when determining what technology best suits your purpose. 

While Kafka and Azure Service Bus share some similar features, their intended use cases differ. Apache Kafka is highly flexible and can serve as both a message queue or stream processing platform. Azure Service Bus is purely a message broker specifically intended for order processing and financial transactions. However, when dealing with data and event streaming, Azure offers Event Grid and Event Hubs aimed at the data streaming use case. But locking into more proprietary tech can have negative future implications on your business operations. 

Want to explore Apache Kafka for your organization’s use cases? Sign up for a free trial of Instaclustr Managed Kafka to get started with Kafka right away.