The Network Partition That Elected Two Leaders
The Network Partition That Elected Two Leaders
Your fintech configuration service runs a 5-node distributed consensus cluster (Raft protocol via etcd/Consul) to store global feature flags, payment routing tables, and cryptographic keys. A core switch in data center rack 2 fails, creating a temporary asymmetric network partition: Node 1 and Node 2 can communicate with each other and with external clients, but cannot reach Node 3, Node 4, or Node 5. Node 1 was the existing leader. Meanwhile, Nodes 3, 4, and 5 miss heartbeats from Node 1, enter candidate state, increment the election term, and elect Node 3 as a new leader. For 40 seconds before discovery, clients in zone A attempt to write new payment gateway routes to Node 1, while clients in zone B write routes to Node 3. An engineer panics, believing the cluster has suffered a permanent split-brain that silently diverged financial configuration. You are asked to explain how Raft mathematically guarantees single-leader linearizability during partitions, what happened to Node 1's attempted writes, and how the cluster re-converges when the network heals.
The Network Partition That Elected Two Leaders
Your fintech configuration service runs a 5-node distributed consensus cluster (Raft protocol via etcd/Consul) to store global feature flags, payment routing tables, and cryptographic keys. A core switch in data center rack 2 fails, creating a temporary asymmetric network partition: Node 1 and Node 2 can communicate with each other and with external clients, but cannot reach Node 3, Node 4, or Node 5. Node 1 was the existing leader. Meanwhile, Nodes 3, 4, and 5 miss heartbeats from Node 1, enter candidate state, increment the election term, and elect Node 3 as a new leader. For 40 seconds before discovery, clients in zone A attempt to write new payment gateway routes to Node 1, while clients in zone B write routes to Node 3. An engineer panics, believing the cluster has suffered a permanent split-brain that silently diverged financial configuration. You are asked to explain how Raft mathematically guarantees single-leader linearizability during partitions, what happened to Node 1's attempted writes, and how the cluster re-converges when the network heals.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).