Talent Group Melbourne Level 8 Rialto North Tower, 525 Collins Street, Melbourne, VIC 3000 Australia Thursday 5th March 2020

Melbourne Meetup – Scaling a real-time Anomaly detection application to 19 Billion events a day using Kafka, Cassandra, Kubernetes

Register Now

What we’ll do

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters.

In this presentation, Paul will reveal how he architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events.

Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. Paul will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from his experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

Evening Agenda

o 5:30pm – 6:00pm: Getting to the Venue and Social
o 6:00pm – 6:05pm: Welcome
o 6:05pm – 6:50pm: Paul Breber (Topic of the night)
o 6:50pm – 7:05pm: Closing Content: Q&A
o 7:05pm – 7:30pm: Pizza, drinks and networking.

About Presenter – Paul Breber, Technology Evangelist at Instaclustr. He’s been learning new scalable technologies, solving realistic problems and building applications, and blogging about Apache Cassandra, Spark, Zeppelin, and Kafka. Paul has worked at UNSW, several tech start-ups, CSIRO, UCL (UK), & NICTA. Paul has helped pre-empt and solve significant software architecture and performance problems for clients including Defence and NBN Co.

Since learning to program on a VAX 11/780, Paul has extensive R&D and consulting experience in distributed systems, technology innovation, software architecture and engineering, software performance and scalability, grid and cloud computing, and data analytics and machine learning. Paul has an MSc in Machine Learning and a BSc (Computer Science and Philosophy).