Instaclustr is pleased to announce the general availability of instances on AWS running Graviton2 processors combined with GP3 volumes for Instaclustr Managed Apache Kafka. With the introduction of the new nodes, Instaclustr’s AWS customers can confidently deploy Apache Kafka on Graviton2 nodes with GP3 volumes to gain both the performance and financial benefits. For an equivalent node size, customers can expect a reduction in their AWS infrastructure fees and improved performance. Graviton2 infrastructure is approximately 20% cheaper than other equivalent instances using the same number of vCPUs and the same amount of RAM. 

Instaclustr also pairs the Graviton2 infrastructure with the new generation of Elastic Block Storage (EBS) GP3 disks. These GP3 disks are cheaper, and they have higher IO throughput compared to GP2 disks. This higher IO throughput is beneficial especially for Kafka because data is flushed to disk, and flushing data to disk can easily become a bottleneck to performance. Upon experimentation, we found that if we stress the Kafka cluster enough, this disparity in the IO throughput results in GP3 disks having a positive impact on the performance/message throughput.

“The addition of AWS Graviton2 instances combined with GP3 volumes is a significant step forward for the Apache Kafka managed service. Customers can be assured that Instaclustr has rigorously tested our managed Apache Kafka offering on the new ARM-based processors. Our performance benchmarks allow our customers to be confident that the performance gains are reflected for realistic workloads. Our development teams have invested a great deal of effort to determine the optimal configuration of each node type. There is an ever-growing list of configuration options which need to be carefully considered and these outcomes will now benefit all of our AWS managed service customers” said Paul Aubrey, VP of Product Management.

Instaclustr conducted a series of benchmarking experiments for all of our EBS-backed Apache Kafka offering’s node sizes. Here is a sample of the results that we produced, with the following experimental setup:

• r5.2xlarge-4500-gp2 v.s. r6g.2xlarge-4500 with GP3 disk (4500 IOPS and 600 MiB/s) 
• 3-node cluster with a replication factor of 3
• Client-broker encryption enabled 
• Message size of 10 KiB each
• Attempted producer message rate (Message per second) [10,000, 50,000, 100,000, 300,000, 500,000, 700,000]

IOPS

ARM (Graviton2)

x86 and GP2

Disk Throughput

ARM (Graviton2) and GP3

x86 and GP2

Kafka Messages Throughput

ARM (Graviton2) and GP3

x86 and GP2

Producer Throughput

1. Attempted producer message rate (messages per second)
2. Producer message throughput. It’s the average rate (messages per second) of message delivery by the producer to Kafka.
3. 90th percentile for producer message latency (in milliseconds). Producer message latency is the total time taken by the producer to deliver a message to Kafka and receive the acknowledgement.
4. Average CPU usage by Kafka cluster (in %)
5. Average memory available on Kafka cluster (in MB)

Consumer Throughput

1. Attempted producer message rate (messages per second)
2. Consumer message throughput. It’s the average rate (messages per second) of message consumption by the consumer from Kafka.
3. 90th percentile for consumer message latency (in milliseconds). Consumer message latency is the total time taken by a message to be delivered by producer to Kafka and consumed by the consumer.

From this data, we can see that the combination of Graviton2 and GP3 outperforms the x86 and GP2 combination. We can also see that the Graviton2 and GP3 combination can make use of the EBS bandwidth better (see https://aws.amazon.com/ec2/instance-types/). 

The following table provides an example of the savings that could be achieved by shifting to Graviton2/GP3-based instances. These calculations are based on a 3-node cluster running in US-East. This cluster is running in Instaclustr’s account and pricing includes the AWS infrastructure and network costs, along with the Instaclustr management fees.

Key features and benefits of the new AWS Graviton2 instances on Managed Apache Kafka include:

• Full selection of new nodes offering the benefits of both Graviton2-based processors and GP3 volumes are available across all of our managed service offerings by default 
• Customers can expect improved price-performance
• Publicly available benchmarks provide customers confidence of performance in real-world scenarios
• Additionally, Instaclustr has determined the situations where additional IOPS and throughput is beneficial, saving customers the overhead of determining the optimal instance configurations. For nodes with less than 1000 GiB of disk, they now have a default of 3000 IOPS as a result of the shift to GP3 volumes.
• Our testing shows that Graviton2 is a better choice for customers under most use cases, so we have chosen to deprecate some older node sizes. We will continue to fully support these instances while we assist customers transitioning to the new-and-improved instance types.

Here at Instaclustr, we have done the hard work so that you can have the confidence and convenience of taking advantage of the latest AWS Graviton2 and GP3 products. Contact our Support or Customer Success teams if you have any questions. New customers can use a free trial of Instaclustr for Kafka here or contact the Instaclustr Sales team.

 ¹Please be aware that AWS has not yet made these new instances available in all Instaclustr-supported regions. Existing, x86 based offerings will continue to be fully supported in regions where Graviton2 is not available.