Streaming From Backups – Reducing Cluster Load When Adding Nodes
Speaker: Ben Bromhead, Co-Founder & Chief Technical Officer at Instaclustr
One of the difficulties with scaling your cluster to cope with additional traffic is the very act of adding capacity increases the load on the cluster, often at a time when your cluster is already experiencing additional load. New nodes must stream data from the existing nodes that are already under stress from serving reads and writes, further exacerbating your problems. In this talk we explore adding the ability for new nodes to stream existing data from warm sources such as backed-up snapshots, commit logs, etc. The talk will then explore some benchmarks surrounding this new streaming solution and look at some of the caveats we experienced.