The failure detector removed a healthy machine

Node A stops receiving heartbeats from B in a hypothetical cluster. A announces B is dead and starts moving its data. C can still reach B. The unnecessary move saturates the network and causes more missed heartbeats.

The detector turned one uncertain observation into a self-reinforcing incident.

Ask another observer

SWIM-style detection first probes a peer directly, then asks a bounded number of peers to probe it indirectly. Suspicion spreads with membership updates. A live node can refute older suspicion with a newer incarnation according to the membership protocol. Failure declaration is still a detector decision under timing assumptions, not mathematical proof of death.

Spread knowledge without a full mesh

If every node contacts every other each period, probes grow as $N(N-1)$. Bounded fanout keeps per-node outgoing probe work approximately constant for fixed configuration. Randomized gossip spreads updates in roughly logarithmic rounds under suitable connectivity and sampling assumptions, but tail convergence and partitions matter.

Exchange compact versioned deltas, bound packet/message size, and use reconciliation for missed updates. Authenticate membership messages; otherwise forged “dead” or “new incarnation” claims become an attack path. Membership tombstones and generation rules must prevent stale messages from resurrecting removed identities. Restart and identity reuse require explicit protocol semantics.

A central registry may be simpler for smaller systems and can itself be replicated. Gossip is not automatically the correct answer merely because a cluster has many machines.

A suspicion score is a model, not a probability of death

A phi-accrual detector estimates how surprising the elapsed heartbeat gap is:

$$\phi=-\log_{10}P(\text{heartbeat interval exceeds elapsed time})$$

The distribution is fitted from observed intervals. A large score indicates an unusual delay under that model; it is not the probability that the host is dead. Runtime pauses, load, and network changes can invalidate the model. Implementations use different distributions and tuning; do not copy a threshold as a universal WAN/LAN setting.

Keep detection separate from consensus, write fencing, and data placement. Gossip can inform routing and suspicion; a quorum protocol establishes authoritative ownership where safety requires it. Delay expensive rebalances until the configured evidence and stability policy justify them.

Observe direct versus indirect probe success, pause time, receive-buffer drops, suspicion churn, membership convergence, and movement load. A detector running on an overloaded node may be diagnosing its own inability to process replies.

Mental model: failure detection produces evidence for decisions. It does not remove uncertainty or replace authority.

Transfer: the partition that heals

---

Source trail: 16-gossip-protocol-and-failure-detection. See the source map for consolidation and reference-retained material.