The Ring Rebalance That Crushed Node 07
The Ring Rebalance That Crushed Node 07
Your distributed key-value storage cluster holds 40 million cached sessions across 16 physical nodes arranged on a consistent hashing ring. When peak holiday traffic climbs to 85,000 req/s, node utilization is heavily uneven: node-07 runs at 94% CPU and 88% memory while node-12 idles at 22% CPU. The on-call engineer adds 4 new physical nodes to relieve pressure, expecting keys to shed evenly from all 16 existing machines. Instead, because keys were mapped to bare node points on the 32-bit integer ring without virtual nodes (vnodes), the new nodes only took keys from their immediate clockwise neighbors. Node-07's preceding neighbor on the ring didn't change, so its workload remained at 94% CPU while another neighbor had 60% of its data abruptly evicted, triggering a mass reload storm against the primary database. You are asked to redesign the ring partitioning to ensure uniform key distribution across all physical nodes, describe the vnode replication topology, and explain what happens when a node fails.
The Ring Rebalance That Crushed Node 07
Your distributed key-value storage cluster holds 40 million cached sessions across 16 physical nodes arranged on a consistent hashing ring. When peak holiday traffic climbs to 85,000 req/s, node utilization is heavily uneven: node-07 runs at 94% CPU and 88% memory while node-12 idles at 22% CPU. The on-call engineer adds 4 new physical nodes to relieve pressure, expecting keys to shed evenly from all 16 existing machines. Instead, because keys were mapped to bare node points on the 32-bit integer ring without virtual nodes (vnodes), the new nodes only took keys from their immediate clockwise neighbors. Node-07's preceding neighbor on the ring didn't change, so its workload remained at 94% CPU while another neighbor had 60% of its data abruptly evicted, triggering a mass reload storm against the primary database. You are asked to redesign the ring partitioning to ensure uniform key distribution across all physical nodes, describe the vnode replication topology, and explain what happens when a node fails.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).