-
- Technical
“Kongo” Part 3: Apache Kafka®: Kafkafying Kongo—Serialization, One or Many topics, Event Order Matters
Kafkafying: the transformation of a primitive monolithic program into a sophisticated scalable low-latency distributed streaming application (c.f. “An epidemic of a zombifying virus ravaged the country”) Steps for Kafkafying Kongo In the previous blog (“Kongo” Part 2: Exploring Apache Kafka application architecture: Event Types and Loose Coupling) we made a few changes to the original...
Learn MorePaul BrebnerApril 26, 2018 -
- Technical
“Kongo” Part 2: Exploring Apache Kafka application architecture: Event Types and Loose Coupling
This is the second post in our series exploring designing and developing and example IOT application with Apache Kafka to illustrate typical design and implementation considerations and patterns. In the previous blog, we introduced our Instaclustr “Kongo” IoT Logistics Streaming Demo Application. The code for Version 1 of the Kongo application was designed as an initial...
Learn MorePaul BrebnerApril 05, 2018 -
- Technical
“Kongo” Part 1: Apache Kafka®—IoT Logistics Streaming Demo Application
What’s a good name to give a demo IoT streaming application dealing with large scale logistics? How about a river… Maybe The “Amazon” application? That’s sort of taken. The Amazon is the longest river and has the most water flow, but what’s the 2nd ranking river? The Congo! The Congo is the 2nd biggest river...
Learn MorePaul BrebnerMarch 15, 2018 -
- Technical
Exploring the Apache Kafka® “Castle” Part B: Event Reprocessing
In this second part of the Apache Kafka Castle blog we contemplate the being or not being of Kafka Event Reprocessing, and speeding up time! Reprocessing Use Cases Reprocess: /riːˈprəʊsɛs/ verb Process (something, especially spent nuclear fuel) again or differently. Repeat event processing is called reprocessing (or sometimes replaying or rewinding), and some reprocessing use...
Learn MorePaul BrebnerJanuary 18, 2018 -
- Technical
Exploring the Apache Kafka “Castle” Part A: Architecture and Semantics
NEWS FLASH Apache Kafka Coming Soon to Instaclustr’s Service Offering! (Source: Wikipedia) If you haven’t read Kafka’s “The Castle” (I haven’t) a few online observations are sufficient for a concise summary (and will save you the trouble of reading it): Time seems to have stopped in the village The story has no ending (Kafka died...
Learn MorePaul BrebnerJanuary 12, 2018 -
- Technical
Apache Kafka® Christmas Tree Light Simulation – Seasons Greetings From Instaclustr
What would you do if you received a request like this? Looks like poor old Santa confused us with a company that produces Instant Forests. But maybe Santa knew I’d been trying out Decision Tree Machine Learning (including Random Forests) on our performance data (see previous blogs on Machine Learning with Spark)? But then I...
Learn MorePaul BrebnerDecember 19, 2017 -
- Technical
Pick‘n’Mix: Apache Cassandra®, Apache Spark™, Zeppelin, Elassandra, Kibana™, and Apache Kafka®
Kafkaesque: \ käf-kə-ˈesk \ Marked by a senseless, disorienting, menacing, nightmarishly complexity. One morning when I woke from troubled dreams, I decided to blog about something potentially Kafkaesque: Which Instaclustr managed open-source-as-a-service(s) can be used together (current and future)? Which combinations are actually possible? Which ones are realistically sensible? And which are nightmarishly Kafkaesque!? In previous blogs,...
Learn MorePaul BrebnerDecember 05, 2017 -
- Technical
Apache Spark™ Structured Streaming With DataFrames
This blog provides an exploration of Spark Structured Streaming with DataFrames The blog extends the previous Spark MLLib Instametrics data prediction blog example to make predictions from streaming data. We demonstrate a 2-phase approach to debugging, starting with static DataFrames first, and then turning on streaming. Finally, we explain Spark structured streaming in more detail...
Learn MorePaul BrebnerNovember 28, 2017 -
- Technical
A Luxury Voyage of (Data) Exploration by Apache Zeppelin
Data Exploration into the cutting-edge technology of Apache Zeppelin (Source: Shutterstock) The catastrophic crash of the Hindenburg in 1937 ended the era of luxury travel in the colossal fast ships of the air that were pushing the boundaries of air travel technology. Zeppelins had many experimental innovations like an auto-pilot, were made from Duralumin girders...
Learn MorePaul BrebnerNovember 09, 2017