Traffic failed over before authority did
A hypothetical regional outage triggers DNS failover. Requests reach the standby, which serves yesterday's application binary and current-looking data. The old region is isolated from monitoring but still accepts some customers' writes.
Traffic moved. Authority did not.
Decide whether to move before moving
A failed probe may mean a region is down, or only that one observer cannot reach it. Use independent vantage points and a business metric; distinguish network partition from complete loss. Predefine a decision gate covering data risk, writer fencing, capacity, dependencies, and residency constraints.
The old database cannot be contacted. Does changing DNS prevent its existing connections from writing? Can the new database's fence reject writes accepted by a separate old copy?
A regional home assignment can give each key one writer and an epoch. Every accepted write checks valid authority. A takeover must establish that the old home's permission can no longer be exercised, then install new authority and reconcile routing. If you cannot prove that under the failure, choose unavailability/read-only behavior or explicitly accept conflict resolution. An epoch field alone is not magic across disconnected storage copies.
RPO and RTO describe different debts
RPO is the tolerated loss window; RTO is the tolerated service-restoration time. Asynchronous replication exposes unshipped writes even when both regions normally serve traffic. Synchronous cross-region replication can reduce that loss exposure at the cost of network latency and availability dependencies.
Compute recovery along the actual critical path. Detection and a decision gate may precede promotion; capacity preparation may run concurrently. After promotion come routing propagation, resolver caches, connection lifetimes, and client backoff. Adding every stage blindly overestimates parallel work; ignoring the client tail underestimates user recovery.
| Strategy | What is already present | What recovery still needs |
|---|---|---|
| Backup/restore | Independent recoverable data | Infrastructure, restore, verification, routing |
| Pilot light | Core replicated data/minimal services | Fleet startup and dependency readiness |
| Warm standby | Smaller running stack | Capacity increase and safe writer handover |
| Active serving regions | Live traffic paths | Surviving capacity, per-key authority, conflict/loss policy |
No row supplies a universal RTO. Measure a rehearsal with quotas, secrets, certificates, deploy artifacts, data copies, and control-plane dependencies included.
The standby passes its probe but cannot take the workload
In an active-passive design, the recovery region receives no normal user traffic. A pilot light keeps the recovery data and deployment assets ready but still needs the application tier started or deployed. A warm standby already runs a smaller complete stack; a hot standby keeps full serving capacity ready. Even a hot standby still needs a safe data/authority transition and correct routing.
Suppose this exercise needs 50 application instances, each with four vCPUs, at the target workload. The warm standby runs ten. Its probes establish that the small stack works, not that the remaining forty instances can start.
The recovery region permits 120 vCPUs, with no competing allocation. Is scaling from ten instances to fifty a viable recovery step? Would automatic DNS failover repair the shortfall?
Synthetic traffic can exercise a warm stack's certificates, secrets, and read dependencies. A pilot light needs periodic deployment rehearsals as well. Neither test proves the writer handover unless that transition is exercised. Keep the detailed provider procedures on the source reference shelf and measure the whole recovery path.
Convergence is not a business invariant
Last-writer-wins can choose one of two concurrent values while discarding the other's effect. It does not prevent selling the last item twice. A PN-counter can merge per-region increment/decrement components by componentwise maxima and sum their difference, yielding convergent counts. That does not enforce a global nonnegative inventory constraint without additional rights allocation or coordination.
CRDTs require well-defined merge/delivery assumptions. Hybrid logical clocks help order causal events; they do not determine which concurrent business choice should win. Choose multi-writer semantics only where the data type and product can tolerate them.
Recovery continues after traffic returns
Size survivors for moved peak load, cache coldness, retries, and a further zone failure where required. Cells and shuffle sharding reduce shared blast radius when dependencies are also isolated; a shared global database can defeat the isolation.
On return, the old region rejoins as a replica or explicitly reconciled copy. Drain old queues against the same operation identities. Compare unknown operations by durable IDs, retain conflicting histories for review, and perform failback as a new planned handover with a new valid generation. Never point traffic back merely because health checks turned green.
Practice lost connectivity, ambiguous in-flight writes, target overload, and failback. For an illustrative 99.99% availability objective over 30 days, the full-outage budget is 4.32 minutes; partial outages require an explicitly chosen measurement/weighting policy. A failover plan longer than the budget needs either a changed objective or a different design.
Mental model: regional recovery transfers authority, data responsibility, and workload—not just network traffic.
Transfer: no lost orders
An interviewer asks for zero acknowledged-order loss and writes that remain available in both halves of an arbitrary partition. What should you clarify before naming a cloud service?
Source trail: 24-cloud-disaster-recovery-and-multi-region-active-active, 09-multi-region-failover. See the source map for consolidation and reference-retained material.