-
- Dev Rel
- Technical
Running the Apache Camel™ HTTP Kafka Source Connector on Instaclustr Managed Apache Kafka®
Camels drinking from a leaking pipe (Source: Wikimedia) A few years ago I built a “zero-code” data processing pipeline with Apache Kafka® and related open source technologies like OpenSearch®, PostgreSQL®, and Apache Superset™. Here’s the first blog in that series (there were 10 in all; the links to each can be found at the bottom...
Learn MorePaul BrebnerApril 08, 2024 -
- Dev Rel
Tracing Apache Kafka® With OpenTelemetry – “Leave Lots of Traces”
A radio-controlled saltwater crocodile? No, a croc with telemetry – sensors + radio communication! (Source: Wikimedia) 1. Introduction It’s been 5 years since I looked at observability and distributed tracing for Apache Kafka® with open source OpenTracing and Jaeger in this blog. So, I decided it was worth taking another look at tracing Apache Kafka, particularly...
Learn MorePaul BrebnerMarch 15, 2024 -
- Dev Rel
- Technical
An Apache Kafka® and RisingWave Stream Processing Christmas Special
Something has gone amiss in Santa’s workshop! The “toy” Elves and the “box” Elves have stopped talking to each other and have set up 2 separate conveyor belts. The packing Elves therefore need some help to match toys and boxes. In this Christmas Special blog let's give Santa a hand and build a Kafka Elf Packing Assistance Machine. We'll use Apache Kafka combined with Apache Streams or a new open-source Streams processing technology, RisingWave.
Learn MorePaul BrebnerDecember 14, 2023 -
- Technical
Instaclustr Cluster Performance Insights #1: Cluster Size Distribution and Zipf’s Law: Why Open Source Computer Clusters Are Like Galaxies
Image from NASA’s James Webb Space Telescope showing thousands of galaxy clusters (https://webbtelescope.org/contents/media/images/2022/038/01G7JGTH21B5GN9VCYAHBXKSD1?news=true) public domain Recently I decided to take a closer look at the cloud computing clusters that Instaclustr uses to provide managed open source Big Data services to 100s of customers, to see if I could discover any interesting resource and performance/scalability insights....
Learn MorePaul BrebnerDecember 12, 2023 -
- Dev Rel
Apache Kafka® Anti-Patterns and How To Avoid Them
USS Enterprise NCC-1701 Warp Drive (Source: Bryan Alexander, CC BY 2.0, via Wikimedia Commons) As every “Trekky” (a fan of Star Trek) knows, in the Star Trek universe spaceships can travel at speeds faster than light using warp engines fuelled by antimatter. It turns out that antimatter is real enough in our universe, but very...
Learn MorePaul BrebnerNovember 08, 2023 -
- Dev Rel
- Technical
Machine Learning Over Streaming Kafka Data—Part 6: Incremental TensorFlow Training With Kafka Data and Concept Drift
In the “Machine Learning over Streaming Kafka Data” blog series we’ve been learning all about Kafka Machine Learning—incrementally! In the previous part, we connected TensorFlow to Kafka and explored how incremental learning works in practice with moving data. In this part, we introduce concept drift, try and reduce noise, and remove time! 1. Concept Drift ...
Learn MorePaul BrebnerOctober 17, 2023 -
- Dev Rel
- Technical
Machine Learning Over Streaming Kafka® Data—Part 5: Incremental TensorFlow Training With Kafka Data
In the “Machine Learning over Streaming Kafka Data” blog series we’ve been learning all about Kafka Machine Learning – incrementally! In the previous part, we explored incremental training with TensorFlow, but without the complication of using Kafka. In this part, we now connect TensorFlow to Kafka and explore how incremental learning works in practice with...
Learn MorePaul BrebnerSeptember 28, 2023 -
- Dev Rel
- Technical
Machine Learning Over Streaming Kafka Data—Part 4: Introduction to Incremental Training With TensorFlow
Incremental clicker games can be addictive. The kinds involving performing simple actions repeatedly to gain higher levels. (Source: Shutterstock) One of the goals of incremental learning is to train a model continuously from streaming data. Incremental learning from streaming data means you don’t need all the data in memory at once, and the model is as up-to-date...
Learn MorePaul BrebnerSeptember 12, 2023 -
- Popular
- Technical
Q&A with FerretDB
A Ferret in the Wild (Source: Shutterstock) Recently I managed to track down some of the key people behind FerretDB, and they kindly offered to answer some questions I had about FerretDB, the software and the project. Below is my Q&A I conducted with Alexander Fashakin (Technical Writer, FerretDB), Peter Farkas (CEO, FerretDB) and Marcin Gwóźdź (Director...
Learn MorePaul BrebnerAugust 31, 2023