The GC Pause That Declared the Cluster Dead
The GC Pause That Declared the Cluster Dead
Your distributed NoSQL database cluster spans 1,200 nodes across three data centers. Cluster membership and node health are tracked via an epidemic gossip protocol (SWIM / Cassandra-style gossip). The failure detector uses a simple hard threshold: every node pings 3 random peers every second; if a target node does not respond within 3 seconds (3 missed heartbeats), it is marked as DEAD, its membership status is gossiped across the cluster, and adjacent replica nodes immediately begin an emergency data rebalancing and partition rebuilding procedure. During peak evening traffic, a brief 3.2-second network switch queue buffer bloat causes 40 healthy nodes in Zone B to delay their gossip ACK packets. The failure detector marks all 40 nodes DEAD. The remaining 1,160 nodes frantically initiate petabyte-scale data re-replication to recover the "lost" partitions, saturating inter-rack bandwidth. The resulting network congestion prevents further gossip messages from landing, causing another 80 healthy nodes to be declared dead in a cascading death spiral. You are tasked with replacing the rigid heartbeat detector with a probabilistic failure model (Phi Accrual) and multi-phase suspicion mechanism.
The GC Pause That Declared the Cluster Dead
Your distributed NoSQL database cluster spans 1,200 nodes across three data centers. Cluster membership and node health are tracked via an epidemic gossip protocol (SWIM / Cassandra-style gossip). The failure detector uses a simple hard threshold: every node pings 3 random peers every second; if a target node does not respond within 3 seconds (3 missed heartbeats), it is marked as DEAD, its membership status is gossiped across the cluster, and adjacent replica nodes immediately begin an emergency data rebalancing and partition rebuilding procedure. During peak evening traffic, a brief 3.2-second network switch queue buffer bloat causes 40 healthy nodes in Zone B to delay their gossip ACK packets. The failure detector marks all 40 nodes DEAD. The remaining 1,160 nodes frantically initiate petabyte-scale data re-replication to recover the "lost" partitions, saturating inter-rack bandwidth. The resulting network congestion prevents further gossip messages from landing, causing another 80 healthy nodes to be declared dead in a cascading death spiral. You are tasked with replacing the rigid heartbeat detector with a probabilistic failure model (Phi Accrual) and multi-phase suspicion mechanism.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).