-
- Technical
Apache ZooKeeper Meets the Dining Philosophers
A ZooKeeper walks into a pub… (actually an Outback pub) The ZooKeeper notices a very rowdy crowd at a round table who appear to be fighting over forks, and she can’t avoid overhearing this conservation: Karl (Marx): “Ludwig, I am hungry please lend me a fork” Ludwig (Wittgenstein): “Karl, I don’t fully understand what you...
Learn MorePaul BrebnerMay 09, 2021 -
- Technical
Apache Kafka MirrorMaker 2 (MM2) Part 2: Practice
In part 1 of this blog series, we focused on MirrorMaker 2 theory (Kafka replication, architecture, components and terminology) and invent some MirrorMaker 2 rules. In this part, we will be more practical, and try out Instaclustr’s managed MirrorMaker 2 service and test the rules out with some experiments.
Learn MorePaul BrebnerMarch 18, 2021 -
- Technical
Apache Kafka MirrorMaker 2 (MM2) Part 1: Theory
In this new two-part blog series we’ll turn our gaze to the newest version of MirrorMaker 2 (MM2), the Apache Kafka cross-cluster mirroring, or replication, technology. MirrorMaker 2 is built on top of the Kafka Connect framework for increased reliability and scalability and is suitable for more demanding geo-replication use cases including migration, special purpose...
Learn MorePaul BrebnerMarch 16, 2021 -
- Technical
Scaling Apache Kafka® Connect Streaming Data Processing (Pipeline Series Part 5)
In Part 4 of this blog series, we started exploring Kafka Connector task scalability by configuring a new scalable load generator for our real-time streaming data pipeline, discovering relevant metrics, and configuring Prometheus and Grafana monitoring. We are now ready to increase the load and scale the number of Kafka Connector tasks and demonstrate the...
Learn MorePaul BrebnerJanuary 25, 2021 -
- Technical
Monitoring Apache Kafka® Connect Pipeline Metrics with Prometheus (Pipeline Series Part 4)
In Part 3 of this blog series, we looked at Apache Camel Kafka Connector to see if it is more or less robust than the connectors we tried in Part 1 and Part 2. Now we start exploring Kafka Connect task scaling. In this blog we will: Change the data source so we can easily...
Learn MorePaul BrebnerJanuary 21, 2021 -
- Technical
Getting to Know Apache Camel Kafka Connectors (Pipeline Series Part 3)
In Part 1 and Part 2 of this blog series we started a journey building a real-time pipeline to acquire, ingest, graph, and map public tidal data using Apache Kafka, Kafka Connect, Elasticsearch, and Kibana. In this blog, we resume that journey and take an Apache “Camel” (Kafka Connector) through the desert (or the Australian Outback) to see if it is more or less robust than the connectors we previously tried.
Learn MorePaul BrebnerDecember 17, 2020 -
- Technical
Building a Real-Time Tide Data Processing Pipeline: Using Apache Kafka®, Kafka Connect, Elasticsearch™, and Kibana™—Part 2
In Part 1 of this blog, we built a simple real-time data processing pipeline to take streaming tidal data from NOAA stations using Kafka connectors, and graph them in Elasticsearch and Kibana. We also tried viewing the data on a Kibana map but ran into a problem! In Part 2 we add the missing geo_points...
Learn MorePaul BrebnerNovember 11, 2020 -
- Technical
Building a Real-Time Tide Data Processing Pipeline: Using Apache Kafka®, Kafka Connect, Elasticsearch™, and Kibana™—Part 1
ApacheCon@Home is over for 2020 and was a resounding success, with close to 6,000 attendees from every continent. As a Platinum Sponsor, Instaclustr ran an ApacheCon Booth and this blog was originally presented on 30 September 2020 as a booth talk. I was one of the 1% attending from Australasia: This, unfortunately, meant I was...
Learn MorePaul BrebnerNovember 05, 2020 -
- Feature Releases
- Technical
Dedicated ZooKeeper for Apache Kafka®
Instaclustr is pleased to announce the release of dedicated Apache ZooKeeper nodes as an additional optional feature of our Managed Apache Kafka offering. Apache ZooKeeper is used for the management and coordination of nodes in Kafka.
Learn MorePaul BrebnerOctober 12, 2020