Multi-Region Failover

Part 0. Start here

The problem: Frankfurt goes dark at 20:00

Seatly sells event tickets. It is Friday, 20:00, the busiest hour of the week. Frankfurt (eu-central-1) stops answering: the front doors, the app servers, the Aurora writer, everything. Frankfurt serves the EU checkout, which is 40% of Seatly's customers (our example's number). The status page promises "automatic failover".

Here is the setup Seatly had last year, the naive one:

• a Route 53 failover record for api.seatly.eu, with health checks at the defaults: a check every 30 s, and 3 failed checks in a row to mark the endpoint unhealthy;
• a DNS time to live (TTL) of 300 s (our number);
• a warm standby in Ireland (eu-west-1) with 15 instances, which scales out by launching new ones;
• an Aurora Global Database secondary in Ireland that nobody has promoted. It is a read-only copy until someone runs a failover.

Two AWS facts shape what happens next. An Aurora Global Database failover loses what replication hadn't shipped yet, which AWS says is "typically measured in seconds", and it takes minutes. And it starts only when someone runs it: nothing promotes the secondary on its own.

The budget: Seatly promises 99.99% availability. A 30-day month has 30 × 1,440 = 43,200 minutes, and 0.01% of that is 43,200 × 0.0001 = 4.32 minutes of allowed downtime a month.

What the naive setup does that night:

1. At about 20:01:30 (3 checks × 30 s = 90 s, "about" because many checkers report), Route 53 marks Frankfurt unhealthy and starts answering with Ireland's address. Clients follow as their cached answers expire, within the 300 s TTL.
2. By about 20:06:30 (6.5 minutes in), EU customers reach Ireland, where every write fails: the database there is still a read-only secondary.
3. On-call notices at about 20:10 (our number) and runs the failover; Ireland takes writes at about 20:12. Meanwhile 15 instances carry the load of 60, 4× their capacity (60 ÷ 15). Auto Scaling launched new instances at about 20:02; Seatly's own measured launch-to-healthy time is about 23 minutes, so they are healthy at about 20:25.
4. At about 20:14, Frankfurt's power returns for a moment. Clients still holding Frankfurt's address write there, and nothing tells Frankfurt it is no longer the writer: two writers now accept orders. On-call shuts Frankfurt down by 20:20.

EU customers can't reliably buy tickets until the new capacity is healthy at about 20:25: about 25 minutes. Weighted by the share of customers hit, 25 × 0.4 = 10 customer-weighted minutes, which is 10 ÷ 4.32 = 2.3 months of budget in one evening.

One more trap hides in the "fix" someone suggests afterwards: make Ireland's health check test a write, so DNS never sends users to a read-only Region. Now both records are unhealthy until promotion. When every record in the group is unhealthy, Route 53 treats them all as healthy (it fails open), so it answers with the primary record: Frankfurt's dead address.

The big picture

Synthesizing vector architecture diagram...

What to notice: Frankfurt is the home and the only writer. Ireland runs a small fleet and keeps a read-only copy that trails by 0.8 s. The probes that judge Frankfurt run in Ireland and Stockholm, never inside Frankfurt. Route 53's answer is decided by routing controls, not by editing records.

What you'll be able to do after this page

1. Name the five decisions of a Region failover in order, say who owns each, and compute the orders at risk from the commit rate and the lag (Part 1).
2. Decide that a Region is gone from outside it, from two places, with a business metric, and tell a network split from an outage (Part 2).
3. Build the decision gate: what it checks, when a person must approve, and how ARC safety rules enforce it (Part 3).
4. Fence the old writer and promote the new one, and say why Aurora's own fence is not enough (Part 4).
5. Move users with pre-created routing controls, and add up the DNS tail and the recovery time (Part 5).
6. Size the survivor for the moved load, check its quota and residency, and pick a rung on the disaster-recovery ladder on equal terms (Part 6).
7. Count and resume the writes caught in the middle, and answer the double-booking drill (Part 7).
8. Bring the old Region back as a replica, reconcile its tail, and fail back on purpose (Part 8).
9. Keep most failures smaller than a Region with cells and shuffle sharding (Part 9).
10. Rehearse failovers and spend an error budget in minutes (Part 10).
11. Trace one checkout through every layer after the move (Part 11).
12. Map all of it to AWS, and name the look-alikes that are not Region failover (Part 12).

You may have arrived from a step that relies on this: step 3.3 of the Netflix case study (move all traffic out of a Region, fast), step 3.4 of the Uber case study (a whole site goes down mid-trip), step 3.4 of the Google Drive loop (EU data stays in the EU, and a Region can fail), or step 3.5 of the payment loop (data must stay in-country, and a Region can fail). This page is the "why" behind all four, and behind about thirty other loop steps that move a Region.

---

Part 1. Five decisions, one home per key, and two numbers

A Region is gone. What has to happen, and in what order? Teams that skip the order get the naive night: users moved before the database can take writes, and an old writer nobody stopped. This Part names the order, the one design rule that makes a move possible at all, and the two numbers every failover is judged by.

Five decisions, in order

Synthesizing vector architecture diagram...

What to notice: every arrow has an owner and evidence. Only one arrow before Moved needs a person: the approval from Gated to Promoted; the clean-up arrows after it are operators' planned work. "Promoted" includes the lease takeover and the new epoch (Part 4); "Moved" is only the routing switch (Part 5).

One home per key

The design rule that makes a Region move safe is one home per key. Every event Seatly sells has exactly one home Region, and only the home writes that event's seats and orders. US events are homed in us-east-1 and EU events in Frankfurt. A failover moves a home: Frankfurt's events get Ireland as their home, with a new epoch, and nothing else changes owner.

With one home per key, "who may sell seat 14C?" always has one answer. Without it (two Regions writing the same event), every seat is exposed to the window in which the two Regions disagree, all the time, not only during failover (Part 7). The home is recorded in one place a split can't move (Part 4), and a write for an event whose home is elsewhere is refused or forwarded, never quietly accepted. Sharding, Hot Keys & Rebalancing (Part 8 there) applies the same "moved" refusal to a key whose shard has moved; this page applies it to a whole Region.

Writes from far away pay for this: a customer in Madrid buying a ticket for a Frankfurt-homed event writes to Frankfurt, wherever the customer is.

RPO and RTO

Every failover is judged by two numbers:

• RPO (recovery point objective): how much acknowledged data you can lose. It is set by replication: whatever the standby hasn't received when the home dies is lost to the standby.
• RTO (recovery time objective): how long customers can't use the service. It is set by the timing chain of the five decisions (Part 5 adds it up).

Our first formula counts the data at risk:

$$\text{orders at risk} \;=\; \text{commit rate} \;\times\; \text{replication lag}$$

For Seatly at 20:00: 200 orders a second × 0.8 s = 160 acknowledged orders that exist only in Frankfurt. This is the same rule the Replication, Quorums & Read-Your-Writes page derives for one database; this page takes what cross-Region replication promises from there and adds only what a Region move needs.

The lag is not a constant. When the link between the Regions breaks, the lag grows by one second every second, and so does the data at risk (Part 2 shows it). The lease the home holds and the epoch it writes with come from Leases, Fencing Tokens & Distributed Locks (Part 7 there); Part 4 here applies them to a whole Region.

Words we use

The example we follow

A real company moves dozens of services between many Regions; that is too much to watch. So one small story runs through the whole page: Frankfurt goes dark, step by step. Every trace on this page was produced by running a private reference simulator of this setup (Regions, links that can be cut, canaries, a three-replica control table, the lease, a lagging database, routing controls, DNS caches, capacity and quota, clients and queues), not worked out by hand.

The story in eleven beats: a split that isn't an outage (Part 2); the gate (Part 3); the fence and the promotion (Part 4); the flip (Part 5); where the load can go, and the ladder (Part 6); the orders caught in the middle (Part 7); Frankfurt comes back (Part 8); the bad deploy that stayed in one cell (Part 9); the drill and the budget (Part 10); one checkout through every layer (Part 11); and on AWS (Part 12). Each Part shows only its own slice of events; the full table is in Part 13.

Snapshot R1: normal Friday, 18:00

---

Part 2. Is it the Region? Detecting from outside

A Region can't judge itself: when it is gone, its own alarms are gone with it. And a Region that looks dead from one place may be perfectly healthy. The first decision is therefore made by others, from more than one place, and it must tell a broken link from a dead Region. At 18:30, two hours before the real outage, Seatly gets exactly that test.

Trace: the link breaks at 18:30

Snapshot R2: the split

Synthesizing vector architecture diagram...

What to notice: only the Ireland–Frankfurt link is cut. Frankfurt still reaches Stockholm, so its lease stays fresh and its CPS keeps arriving. Ireland still reaches Stockholm, so it reads both. One failed vantage point plus a healthy business metric is a split, not an outage.

Vantage points and a business metric

• Vantage points outside the Region. Canaries run in Ireland and in Stockholm, each attached to a VPC and probing Frankfurt's public checkout path every 10 s. Optionally, a third probe runs outside AWS; it is the only one that doesn't share the AWS backbone with replication.
• A business metric. Each Region writes its checkout starts per second (CPS) to the control table every 10 s. Frankfurt's CPS is "down" if it is below its normal band, or missing, for 60 s. The metric catches what probes miss: a front door that answers 200 while checkouts fail.
• Two independent deciders. The rule runs separately in Ireland and in Stockholm, each reading the control table. Both fire; every action they start is safe to repeat (setting a desired capacity of 60 twice is the same as once).
• Nothing on the EU path reads us-east-1. Not the probes, not the metric, not the rule.

The rule has a second clause for a home that has fenced itself: Frankfurt lost the control table (its renewals fail), so it stopped writing at its lease stop time, yet its front door still answers and the canaries stay green. The self-fenced clause catches that case: Frankfurt's lease has lapsed and its CPS is down. The simulator's run of that case: Frankfurt's control-table replica fails at 19:30:00, its last renewal (sent 19:29:59) makes it stop at 19:30:09, its CPS report goes missing, and the rule fires at 19:31:00 with the canary clause false and the self-fenced clause true.

textCopy
EVACUATION RULE (runs every 10 s, in Ireland and in Stockholm, independently)
  canaries_failed = vantage(Ireland).misses >= 3 AND vantage(Stockholm).misses >= 3
  cps_down        = Frankfurt CPS below band, or no report, for >= 60 s
  self_fenced     = lease.home = Frankfurt AND lease.ack_until < my wall clock
  if (canaries_failed AND cps_down) OR (self_fenced AND cps_down):
      page on-call; start safe-to-undo work (capacity); open the gate
  else if one vantage failed:
      alarm "possible split", never move

What a split does to the data at risk

A split doesn't move anyone, but it quietly removes the RPO promise. Suppose Frankfurt had died at 18:49:59, one second before the link came back. Ireland would have been 0.8 + 1,199 = 1,199.8 s behind: 200 × 1,199.8 = 239,960 acknowledged orders (about 240,000) that exist only in Frankfurt.

Aurora has a knob for this. rds.global_db_rpo (minimum 20 s) makes the primary block commits once every secondary is further behind than that. In the simulator, the lag reaches 20 s at 18:30:19.2, so Frankfurt stops taking orders for the remaining 19.7 minutes of the split: a write outage in a healthy Region, to protect a copy nobody is using. AWS also warns that with only two Regions, the parameter's default should stay in the secondary's parameter group; otherwise, after a failover, the new primary can find itself with no caught-up secondary and block its own commits. Seatly keeps the default and alarms on the lag instead: an RPO/availability trade made on purpose.

Route 53 health checks, precisely

Route 53 health checks still matter: they are how routing controls reach DNS (Part 5). But as a detector they need care.

On the night itself (event 10x), Route 53's check for Frankfurt went unhealthy at about 20:01:30, and its metrics sat in us-east-1. Seatly shows it on a dashboard as advice only: the rule never waits for it and never reads it.

Hosts and Regions. Inside a cluster, failure detectors such as SWIM's "suspect, then let others refute" (which the Uber case study describes) or Cassandra's phi-accrual detector (default threshold 8) apply the same idea to machines: don't convict on one observer. This page applies it one level up, to a whole Region.

Trace: detection at 20:00

---

Part 3. Do we move? The decision gate

Knowing that Frankfurt is gone doesn't mean moving is safe. Moving to a Region without capacity, without quota, or into a country the data may not enter makes things worse, and moving while the old writer may still write corrupts data. The second decision is a gate: automation gathers the evidence, and one approval opens it.

Trace: 20:01, the checklist

At 20:01:00 the rule has fired, and automation computes six checks.

Snapshot R3: the gate at 20:01

Check (d) is also the gate's lower bound: no approval can take effect before ack_until + 5 s, whatever the human or the rules say. Here the bound (20:00:14) is hidden under detection (20:01:00); it matters when detection is fast.

textCopy
GATE (automation; the approval is the only human step)
  on rule fired:
      start capacity (warm pool)                   -- safe to undo, start now
      checks = [region_evidence, target_ready, residency_map,
                lease_expired_plus_margin, one_control_stays_on, data_at_risk]
      if data_at_risk > 1000 orders: also require business owner
  approve when all checks pass AND (on-call approves OR pre-approved rules match)
  never before lease.ack_until + margin (on the taker's clock)
  then: take the lease, promote, bump the epoch (Part 4); then flip (Part 5)

Safety rules in the switch itself

ARC's routing controls come with safety rules that the cluster enforces on every change, so a wrong button press is refused by the switch, not by a runbook:

The two gating controls are Seatly's design choice. Automation turns eu-west-1-writable On right after Ireland's database takes its first write at the new epoch (Part 4). The approver (on-call, or a pre-approved rule written for this case) turns eu-west-1-read-only-approved On only when choosing to move users into an explicitly read-only Ireland (Part 5). Until one of them is On, nobody can send EU users to Ireland, not a script, not a tired engineer. A gating rule blocks every change to its target, so a gating control must stay On until its target has moved, in either direction: eu-central-1-writable stays On until the 20:05 flip has turned eu-central-1-on Off, and only then goes Off.

When a rule blocks a move that must happen (a gating control stuck because its automation is down), ARC lets an update call override named safety rules ("Overriding safety rules to reroute traffic": the update call lists the rules to bypass). That is the break-glass for a stuck gate, used only by a written runbook, because it also bypasses the protection the rule exists for.

Why a move to a read-only Region is gated

The naive night sent users to Ireland 6.5 minutes in, where every write failed. A DNS move is only useful if the target can do what users came for. So there are exactly two legal orders:

1. Promote and fence first, then flip. The main line.
2. Flip into an explicitly approved read-only mode, with writes answering 503 Retry-After, and promote later. Part 5 prices it.

What is never legal is a health check moving users on its own to a Region that still refuses writes. That is why Seatly's DNS answers follow routing controls rather than Frankfurt's own health.

---

Part 4. Fencing the old writer, promoting the new one

"Frankfurt is dark, so it can't write" is an assumption, not a fact. A dark Region is often a Region cut off from everyone else, with some app servers still running inside it. If Ireland starts writing while one of them still writes, two homes sell the same seats. The third decision makes the standby safe: the old home stops on its own, the new home waits until that stop is certain, and a number in the new database refuses anything older.

Recap: page 03's handover, at Region scale

Leases, Fencing Tokens & Distributed Locks (Part 7, "Single writers across a failover") gives the recipe: the writer holds a lease; the old writer stops when it can't renew, by its own monotonic clock; the new writer waits out the lease plus a margin; the new writer bumps the epoch in the store it will write before its first write; consumers compare (epoch, version). Page 03 also stops the holder 2 s before its lease ends (a holder margin). This page uses no holder margin: Frankfurt stops exactly at its send time + 10 s, and all the safety margin sits on the taker's side (5 s), which must be larger than the worst gap between Frankfurt's and Ireland's clocks (page 03, Part 5). What's new at Region scale is where the lease lives, whose clock reads it, and the fact that the old home has its own copy of the database.

A lease a split can't move

The lease item eu-writer = {home, epoch, ack_until, rvn} lives in the MRSC control table, with full replicas in Frankfurt, Ireland and Stockholm. A write to an MRSC table succeeds only while its Region can reach at least one other replica. AWS: an isolated Region "can only service eventually consistent reads". So:

• A split between Frankfurt and Ireland leaves Frankfurt with Stockholm: 2 of 3. Frankfurt keeps renewing, and any takeover from Ireland fails its condition (Part 2, event 3).
• A Frankfurt cut off from both can't renew. Its renewals fail, and it stops at its own stop time.
• Stockholm is there only to be the third copy and the second vantage point. It holds no customer data, which also keeps the residency map simple.

Page 03 (Part 8, "Across Regions") explains why an MREC global table can't hold a lock: its conditions are checked on the local replica, and both Regions can "win". The control table must be MRSC.

Whose clock?

ack_until crosses machines, so it is a wall-clock time, and the reader must allow for the two clocks disagreeing. DynamoDB conditions have no function that reads the current time, so :now in the condition is always the taker's clock.

textCopy
Frankfurt (home), every 2 s:
  UpdateItem eu-writer
    Condition: home = :me AND epoch = :e
    Update:    SET ack_until = :my_wall_now_plus_10s, rvn = :new_random
  on success: stop_at = monotonic send time + 10 s
  before every write: if monotonic now >= stop_at, refuse the write

Ireland (taker), after the gate approves:
  UpdateItem eu-writer
    Condition: epoch = :seen AND ack_until < :my_wall_now_minus_5s
    Update:    SET home = :me, epoch = :seen + 1, rvn = :new_random
  clock-free alternative:
    Condition: rvn = :seen   -- after rvn stayed unchanged for 15 s on my monotonic clock

What the simulator printed for the night: Frankfurt's last successful renewal was sent at 19:59:59.0, so ack_until = 20:00:09.0, and any Frankfurt server still alive stops writing at 20:00:09.0 on its own clock. With the two clocks agreeing, Ireland's condition could first pass at 20:00:09.0 + 5 = 20:00:14.0. Ireland polls the item every second (at x.5 s); it first saw the final rvn at 19:59:59.5, so the clock-free variant could claim from 20:00:14.5. The gate closed much later (20:03:00), so neither bound delayed anything tonight.

The margin is only as good as the clock assumption behind it. ack_until is written on Frankfurt's clock and compared with Ireland's, so what counts is how far Ireland's clock is ahead of Frankfurt's, not either clock's offset from true time. The simulator's replay with Frankfurt's clock exact and Ireland's wall clock ahead of it (only the gap between the two clocks counts: Frankfurt 3 s slow and Ireland 3 s fast behaves like the 6 s row):

Offsets from true time add up. Frankfurt 3 s slow and Ireland 4.9 s fast are each "below the margin", yet their clocks are 7.9 s apart: the simulator shows Ireland's condition passing at 20:00:06.1, 2.9 s before Frankfurt stops. So the margin must exceed the worst gap between the two clocks. If each host may be up to x from true time, two hosts can differ by 2x. Alarm on any host whose offset from true time passes half the margin (2.5 s here), or use the clock-free variant, which depends only on Ireland's clock rate over 15 s.

The epoch lives in the database

The lease decides who should write. The epoch decides what a database accepts. Seatly's Aurora database has one row, writer_epoch, and every write transaction checks it:

sqlCopy
-- Ireland's first transaction after promotion (epoch 8 taken in the control table):
UPDATE writer_epoch SET epoch = 8 WHERE id = 'eu' AND epoch = 7;   -- 1 row, or stop

-- every later write transaction, from any app server:
BEGIN;
SELECT epoch FROM writer_epoch WHERE id = 'eu' FOR SHARE;           -- if it isn't my epoch: ROLLBACK and stop
UPDATE seats SET order_id = :order WHERE seat_id = :seat AND order_id IS NULL;
INSERT INTO orders (order_id, epoch, ...) VALUES (:order, :my_epoch, ...);
COMMIT;                                                              -- FOR SHARE keeps the epoch from changing until here

An app server that still believes it writes for epoch 7 (a Frankfurt server following Aurora's global writer endpoint to Ireland, a delayed retry, a worker that wasn't told) has its transaction refused in Ireland. The (epoch, version) pair also travels on every event the outbox emits, so consumers can park anything from an older epoch (the Change Streams & the Transactional Outbox page: the relay resumes under the new epoch).

Trace: the fence and the promotion

Synthesizing vector architecture diagram...

What to notice: nothing on this diagram needs Frankfurt to answer. Frankfurt stops itself at 20:00:09 on its own clock; Ireland's takeover is a condition on a table Frankfurt can't reach; Aurora's attempt to fence Frankfurt simply times out. The epoch is bumped before any customer write reaches Ireland.

Snapshot R4: after the flip

Synthesizing vector architecture diagram...

What to notice: Ireland holds the lease at epoch 8 and its database says 8. Frankfurt's volume still says 7 and holds the 160 orders Ireland never received; nothing in Frankfurt can write, because every server there stopped at 20:00:09 and will refuse until the lease names Frankfurt again.

Aurora: switchover or failover

Aurora Global Database has two ways to move the writer to another Region, and they differ in exactly one thing: whether the old primary is healthy.

Three details the loops rely on:

• The global writer endpoint. Aurora gives a Global Database one writer endpoint whose DNS name follows the primary. AWS advises setting client DNS caches to a low value "such as 5 seconds", because a long cache keeps clients on the old writer (Part 5, the client-cache table).
• Headless secondaries. A secondary cluster may have no DB instance at all, which is cheap (the pilot-light rung in Part 6). AWS: before a switchover or failover to it, "you must add a DB instance to it". That time belongs in the recovery time.
• The target's control plane. failover-global-cluster is an API call to RDS in Ireland. It can't be a pure data-plane action, so it must be rehearsed (Part 10).

Side row 16x, the switchover at failback: Saturday at 06:00 (Part 8), both Regions healthy, Aurora waits until Frankfurt's secondary has every commit, then moves the writer. Nothing is lost, and nothing needs fencing by timeout.

---

Part 5. Moving the users

Ireland can take writes at 20:05:00.2. EU customers are still knocking on Frankfurt's door. The fourth decision moves them, and it has two hard rules: the move must not need the dead Region, and it must not need a control plane that may be having its own bad night (for Route 53's record edits, that is us-east-1). Then comes the part everyone forgets: DNS answers are cached, so users arrive over minutes, not at the moment of the switch.

Trace: the flip at 20:05

Three ways to move traffic

AWS offers three data-plane ways to send users to another Region. All three avoid control-plane calls during the move; they differ in who decides and what they can refuse.

The weakness of plain Route 53 failover records is not us-east-1 (their health checks and answers are data plane too); it is the missing gate and the fail-open rule. That is why Seatly's records follow routing controls instead of Frankfurt's own health.

The DNS tail and client caches

A DNS change is a promise about future lookups. Every cache between Route 53 and the code that opens a connection holds the old answer for a while:

Logins survive the move: sessions sit in the MREC global table, which replicates within about a second, so only the last second or so of session writes (a login at 19:59:59.5) is missing in Ireland until Frankfurt returns (Part 8).

The timing chain

Each decision takes time. Some steps run side by side (capacity starts at detection, while the gate and the promotion run), and some must follow one another (nobody can flip before the promotion, and nobody arrives before the flip). Side-by-side steps cost their maximum; dependent steps add up. That is our second formula:

$$\text{RTO} \;=\; \text{detect} \;+\; \max(\text{capacity},\ \text{gate} + \text{promotion}) \;+\; \text{flip} \;+\; \text{TTL} \;+\; \text{backoff}$$

The gate can't close before ack_until + margin (Part 3), so on a very fast detection the gate term grows to reach that bound.

• Human gate: 60 + max(60, 120 + 120) + 15 + 60 + 30 = 60 + 240 + 105 = 405 s = 6.75 minutes.
• Pre-approved rules: 60 + max(60, 10 + 120) + 15 + 60 + 30 = 60 + 130 + 105 = 295 s ≈ 4.9 minutes. Capacity (ready at 20:02:00) is still hidden under the gate and the promotion (done at 20:03:10).

Synthesizing vector architecture diagram...

What to notice: capacity runs beside the gate and promotion, so its 60 s cost nothing: the longest path is detect, gate, promotion, flip, TTL and backoff, which ends at 405 s (06:45 on the axis, 20:06:45 on the clock).

The night us-east-1goes dark (side replay U)

Seatly's US tier is backup and restore into us-west-2: the US recovery time is hours, by design (Part 6). The more useful question is what else breaks when us-east-1 is the Region that goes dark, and whether the EU plan notices:

The EU plan never needs us-east-1: its probes run in Ireland and Stockholm, its lease lives in three EU Regions, its routing switch is an ARC data-plane call, and its DNS answers were set up in advance.

Read-only mode (side row 19r)

If Aurora's promotion is slow or risky, the approver can move users into an explicitly read-only Ireland first: turn eu-west-1-read-only-approved On at 20:03:00 and flip then. Ireland serves browsing, event pages and seat maps, and answers every write with 503 Retry-After. Reads come back at 60 + max(60, 120) + 15 + 60 + 30 = 285 s (20:04:45). With pre-approved rules the gate takes 10 s, but capacity still has to be running before users arrive: 60 + max(60, 10) + 105 = 225 s (20:03:45). Writes still wait for the promotion and a second flip.

---

Part 6. Where the load can go: capacity, quota, residency, and the ladder

A move is only as good as the Region it moves to. Ireland must be able to run Frankfurt's peak, plus the surge of retries that follows every outage, inside its own account limits, without moving any EU customer's data out of the EU, and with every key, secret and image it needs already in place. Part 3's check (b) and check (c) are where these are tested; this Part is how they're built.

Trace: 20:01, capacity starts

Capacity options on equal terms

Why 512 looked fine (event K0, last month's game day). Ireland's quota was 512 vCPUs while Frankfurt's was 640. Stopped warm-pool instances don't count toward the running On-Demand quota, so every check that looked at Ireland's usage (120 vCPUs) against 512 passed. The game day started the warm pool: 480 running, 32 left, and the surge's 15 instances couldn't launch. Seatly raised the quota to 640. The rule it learned: quota parity with the home, checked against what will be running at failover. ARC's Region switch has built-in service-quota checks that compare the Regions' quotas every 24 hours and can request increases; Seatly runs the same comparison itself.

Sizing the target (event K1)

The simulator's utilization in Ireland, with load arriving at 20:06:45 and the surge lasting until about 20:11:45:

The 80% target was not met during the surge. The warm pool covered the load itself, but the headroom depended on new launches, and new launches take 23 minutes. For five minutes Ireland ran with no headroom at all: slower responses and some timeouts are likely (the simulator doesn't charge them to the budget). The cheap fix is a warm pool of 60 instead of 45, so that all 75 are stopped-and-ready: stopped instances cost only their volumes. Sizing also works per AZ: Frankfurt runs 20 instances in each of three AZs so that losing one AZ still leaves 40, the peak; Ireland's 75 should be spread the same way.

Residency and prerequisites

The default target map was written by someone thinking only about capacity: "spread to spare capacity". Check (c) of the gate is where it meets the residency rule:

Residency also limits how often you can move. With two EU Regions, the EU zone can fail over once: while Frankfurt is out, Ireland has no legal partner, so a second failure means waiting, not moving. Every copy counts, not only the one users are routed to: standbys, backups, witnesses and keys.

• An MRSC witness holds data (event K3). Had Seatly used an MRSC table for seats with Frankfurt, Ireland and a witness in us-east-1, the witness "contains data written to global table replicas", so EU seat data would sit in the US. All three MRSC Regions must be inside the allowed zone.
• Global Accelerator's failover ignores dials (side row K2). Suppose one accelerator served both EU and US, with endpoint groups Frankfurt (dial 100), Ireland (dial 0, the standby) and us-east-1. If Ireland's health check passes, the accelerator moves EU users to Ireland as soon as it marks Frankfurt unhealthy, before promotion: exactly the ungated move Part 3 forbids. If Ireland's check fails by design until promotion, then from Frankfurt's failure until the promotion, the accelerator's failover ignores the dials, tries the closest endpoint groups, and sends EU users' new connections to us-east-1. The simulator's replay with Frankfurt and Ireland unhealthy routes an EU client to us-east-1. The fix is one accelerator (or one set of DNS records) per residency zone, so the only groups it can fail over to are legal ones.
• Prerequisites (side row K4). Ireland must already have everything its fleet needs to start and decrypt: KMS multi-Region replica keys (related multi-Region keys share key material and key ID, so data encrypted in Frankfurt decrypts in Ireland "without re-encrypting or making a cross-Region call"; a replica works even when the primary is unreachable, and an existing single-Region key can't be converted, so this is decided when the data is first encrypted), secrets, TLS certificates and container images. A key or image that exists only in Frankfurt makes Ireland's servers unable to start or decrypt. Gate check (b) verifies them.

The ladder

"How ready is the standby?" is a ladder of choices, each rung costing more while idle and needing less at failover time. AWS's disaster-recovery whitepaper labels the rungs' recovery times as hours, tens of minutes, minutes and real time; the numbers below are Seatly's, from Formula 2 with the same inputs as the main line.

Synthesizing vector architecture diagram...

What to notice: each rung to the right has less to start at failover time. Every rung except hot standby and active-active still depends on something starting during the disaster; that dependency, not the RPO, is what the extra cost buys away.

Side replay L1: the same EU service on five rungs

The same EU service, the same 200 orders a second, the same 0.8 s lag, the same 2-minute human gate and 120 s promotion. Instances are counted across both EU Regions (Frankfurt's 60 included):

Hot standby's RTO equals warm standby's here, because capacity (60 s) runs beside the gate and promotion (240 s). Both rows arrive with 60 instances and run at 100% through the surge; keeping 75 running (5× the warm row's 15) would hold 80%. What 4× the running cost in Ireland (60 instances instead of 15) buys is no capacity risk and no control-plane call for capacity during the disaster: AWS calls a recovery Region that can take the full load as deployed "statically stable". Active-active's gain is that only half the customers are hit (405 s × 0.2 = 81 s = 1.35 customer-weighted minutes instead of 2.7) and that its capacity is proven every day; its cost is dark capacity in both Regions and a conflict rule for anything homed in both. AWS Elastic Disaster Recovery, which replicates servers into a staging area, is a managed version of the pilot-light rung.

Active-passive vs active-active

Seatly's US tier sits on the bottom rung on purpose (side row L2): hourly backups copied to us-west-2, so its RPO is up to an hour plus the copy time and its RTO is hours. That is a stated business decision about the US catalogue, not an accident. Every workload gets its own rung.

Shared limits

Several resources are shared by everything that happens during a failover, and each must be counted with all its consumers:

---

Part 7. Writes caught in the middle

Some orders were in the air when Frankfurt went dark. Some had been committed and acknowledged but never reached Ireland; some had been sent with no reply at all. A failover doesn't make them disappear: it makes them unknown, and unknown orders become lost money, double charges or double-sold seats unless something resumes them on purpose. This Part counts them and follows one seat through the gap.

Trace: 160 orders and 60 unknowns

The rules behind this (derive every ID from the idempotency key, resume unknowns with the same key, and dedup in an inbox at the home) belong to Idempotency & Effectively-Once Processing; this page only counts what a Region move leaves for them. One detail is failover-specific: Ireland's dedup windows start empty. Anything Ireland keeps to recognize repeats (an inbox, a cache of recent keys) has only what was replicated. A request Frankfurt finished but Ireland never heard of looks brand new, which is why the IDs must be derived, so that a second copy collides with the first by ID in reconciliation.

Seat 14C

Synthesizing vector architecture diagram...

What to notice: nobody did anything wrong at 20:07:10. Ireland checked 14C against the only data it had, and the seat was free there. The double sale was created at 20:00:00, when 0.8 s of Frankfurt's commits became unreachable; reconciliation at 21:10 is the first moment anyone can see it.

Drill 05, both questions

The Booking That Existed in Frankfurt but Not in Virginia runs three Regions active-active on the seats, with asynchronous replication of 200 to 400 ms. Side row 22a replays it on Seatly: without homes, an EU phone routed to Ireland sees 14C free 0.4 s after Frankfurt sold it, and a second buyer can take it there.

• Q1, what went wrong: two Regions could both accept a claim on the same seat, each checking only its own copy, inside the replication window. The fix is exactly Seatly's home per event: a claim is a conditional write ("set 14C sold only if it is free") at the single owner of that event, wherever the request landed. Browsing stays local and may be a few hundred milliseconds stale; claiming does not. With homes, the window reopens only during a failover, which is event 22.
• Q2, why not make every write synchronous to every Region: read the last row of the table below. Every write, including the millions that don't need it, pays a round trip to the slowest Region, and a slow or cut-off Region stalls writes everywhere. Scope synchronous replication to contested writes.

Stores at failover

What each store promises across Regions is covered in Replication, Quorums & Read-Your-Writes. This table adds only what matters on the night:

What a secondary accepts, it must forward

The payment provider (PSP) sends a webhook when a payment settles. Frankfurt's payments settle all evening, and their webhooks now arrive at Ireland.

The general rule, for every secondary endpoint (a standby Region, a backup mail server, a regional webhook receiver): anything a secondary accepts is stored durably and forwarded to the key's current home, or to reconciliation. It never answers success and drops the data. The email loop (step 3.4, the backup MX at priority 20) goes one step further: its standby relays everything it accepts to the home continuously, in normal times too, so the forward path is exercised every day and not discovered broken during an outage.

---

Part 8. Failback: bringing the old Region back

At 20:40 Frankfurt's power returns. It has servers that were writers two hours ago, a database volume with 160 orders Ireland never saw, a queue of messages nobody processed, and caches full of old answers. The fifth decision is the cleanup: Frankfurt must come back as a replica, never as a writer, its leftovers must be sorted by ID, and the home moves back only when Seatly chooses.

Trace: Frankfurt returns at 20:40

A counterfactual the simulator keeps in the full table (24x): Frankfurt's servers trust Aurora's fence and their own configuration instead of the lease. One of them writes to a local queue that no longer drains anywhere, or to the old volume before Aurora detaches it. The write is acknowledged and never reaches Ireland. The refusal in event 24 is the old home's own protection; nothing else covers its orphaned copy.

Snapshot R5: Frankfurt back, refusing

Late arrivals and old queues

The MREC tail arrives late (event 26, 20:40). Carts and sessions written in Frankfurt's last second never reached Ireland. AWS: for an isolated MREC replica, "any data not yet replicated to other Regions will be replicated when the replica becomes healthy". They arrive now, and last writer wins: a cart a customer changed in Ireland since 20:06 keeps the newer version; a cart untouched since 19:59:59.5 gets its lost item back. That is fine for a cart; it would not be fine for a seat, which is why seats aren't in MREC.

The old queue (event 27, 20:50). Frankfurt's SQS queue holds 36 messages. Frankfurt's workers stay stopped: if they started, they would process against Frankfurt's stale database and its stale inbox. Instead, a drain job in Ireland reads the messages and checks each against Ireland's inbox (the home's record of what has been processed). Message IDs are derived from the order key and the event type, so a message about an order its customer re-created in Ireland carries the same ID as Ireland's own message and is recognised:

textCopy
DRAIN THE OLD QUEUE (a job at the home; the old Region's workers never start)
  for each message m in Frankfurt's queue:
    if m.message_id is in Ireland's inbox:   drop it          -- already processed
    elif m.order_id exists in Ireland:       in one transaction: insert m.message_id into the inbox and apply the effect;
                                             then send any email or PSP call with m.message_id as its idempotency key
    else:                                    send it to reconciliation, with the message

The relay itself resumes in Ireland under epoch 8 (Change Streams & the Transactional Outbox); the old queue is drained, never replayed.

Reconciling the tail

Failing back on purpose

The failback's write pause (from the simulator; the switchover time is our number): EU writes stop at 06:00:00 and resume at 06:02:00.2, when Frankfurt holds epoch 9. That is about 2 minutes, the switchover itself. Clients whose DNS still points at Ireland get a redirect to Frankfurt from then on, so the DNS tail costs an extra hop, not more refusals. At 40% of customers, that is 2.0 × 0.4 = 0.8 customer-weighted minutes (Part 10 counts it).

The epoch goes up to 9, never back to 7: every writer, message and cached token from before the failback is then older than the current one, including anything left over from Frankfurt's first tenure.

This failback moves all EU events at once. A gradual return (5% of events a minute, or city by city as the ride-sharing loop does) needs a lease and an epoch per cell or per group of keys, because each group then has its own home during the move. Part 9 says why that is its own design.

Synthesizing vector architecture diagram...

What to notice: Frankfurt writes nothing until the last two boxes, and it gets the home back with a new epoch (9), not its old one. Every box before the switchover can take hours; none of them is urgent once Ireland is serving.

Snapshot R6: failback done, Saturday 06:00

---

Part 9. Shrinking the blast radius: cells and shuffle sharding

Most outages are not a Region going dark. They are a bad deploy, a bad config, or one tenant's traffic that breaks something. If such a failure takes down a whole Region, someone will propose evacuating it, and the evacuation carries the bad change to the next Region. The cheapest failover is the one you never need, because the failure stayed small. At 19:00, an hour before the real outage, Seatly's cells get tested.

Trace: the 19:00 deploy

Synthesizing vector architecture diagram...

What to notice: the router maps each event to exactly one cell, so the bad config hit only the quarter of events in cell 2. The wave stopped at the bake alarm; cells 3 and 4 never got the change.

Cells

A cell is a complete copy of the stack (front door, app servers, workers, its own database partition) that serves a fixed slice of keys. A thin cell router maps each key to its cell. Three rules make cells work:

• Deploy one cell at a time, bake, then continue; never two Regions at once. The cell is the unit you deploy and roll back.
• The router is critical and simple. It holds a mapping and nothing else, so it rarely needs to change.
• A cell is not automatically the unit you fail over. Moving one cell's keys to another Region is a smaller Region failover, and it needs its own lease and epoch per cell (the gradual failback of Part 8 needs the same).

Shuffle sharding

Cells split customers into a few large groups; shuffle sharding gives each tenant its own small, random-looking combination of workers, so that almost no two tenants share the same set. Amazon's Builders' Library explains it with eight workers and each tenant on two of them:

• There are C(8, 2) = 8 × 7 ÷ 2 = 28 possible pairs, so 28 shards.
• A poisoned tenant (one whose requests crash workers) takes down its own two workers. Every other pair is one of three kinds:

With the same eight workers in four fixed pairs, a poisoned tenant takes down a quarter of all tenants; with shuffle sharding, 1/28, seven times fewer. The same idea scales: Route 53 gives each hosted zone 4 of 2,048 virtual name servers, about 730 billion combinations (C(2,048, 4) ≈ 7.3 × 10¹¹), arranged so that no two domains share more than 2. The metrics loop (step 3.1) puts each tenant's queries on 4 of 50 queriers: C(50, 4) = 230,300 shards. Shuffle sharding needs clients that retry on another worker; without retries, sharing one worker still hurts.

---

Part 10. Proving it: game days and the error budget

A failover plan that has never run is a guess. Quotas drift, keys get created in one Region only, runbooks name endpoints that were renamed, and the warm pool that looked full has no capacity behind it. And "four nines" means nothing until it is turned into minutes that one night can spend. This Part does both: it spends the month's budget, then rehearses what could spend it next time.

The budget

• 99.99% of a 30-day month: 30 × 1,440 = 43,200 minutes × 0.0001 = 4.32 minutes.
• 99.99% of an average month (30.44 days): 30.44 × 1,440 = 43,833.6 minutes × 0.0001 ≈ 4.38 minutes. State which month you use; the loops use both.
• Weight each outage by the share it hits. An outage that hits 40% of customers for 10 minutes spends 4 customer-weighted minutes.

This month's spending (event 30)

Synthesizing vector architecture diagram...

What to notice: the flat line is the month's 4.32-minute budget. One Region failover alone spent 2.7 of it; the month's total bar (3.2) leaves 1.12 minutes for everything else. Without cells the deploy would have spent 2.0, and the month would have been over budget (4.7).

One Region failover can spend most of a 99.99% month even when it goes well. The naive night in Part 0 would have spent 10 minutes, 2.3 months' worth.

Rehearsing the split, the move and the return (event G1)

Next month's game day, one rehearsal per risk:

Target-side control planes (Aurora's failover call and the warm-pool start both run in Ireland during the disaster) can't meet the "data planes only" rule. They are exactly the steps that must be rehearsed, because they are the ones that can fail on the day.

Scoring a drill. Score by the business metric, not by "the runbook finished": how long until EU CPS was back in band, and how long each phase took (detection, gate, promotion, flip, tail). The Netflix case study (step 3.5, "Did the evacuation work?") scores its evacuations the same way. ARC's Region switch uses application health alarms to report the actual recovery time of each run.

---

Part 11. End to end through the layers

"Failover" touches every layer between a customer's phone and the disk, and each layer learns about the move in its own way and at its own speed. Following one checkout through all of them explains why the move takes minutes, and where the two protections of Part 4 sit.

One checkout after the flip (event 31, 20:07:30)

Synthesizing vector architecture diagram...

What to notice: the highlighted boxes are the guards: the worker's own lease check (cheap, but only as good as the worker's code); Protection 2, the epoch check inside Ireland's writer, where every write lands; and Protection 1 in Frankfurt's own servers, because nothing in Ireland can reach Frankfurt's old volume.

---

Part 12. On AWS

The five decisions are the same on any cloud. On AWS, each has a service that carries it, a set of documented limits, and a few services that look like failover and aren't. The first rule of the whole page applies here too: a failover plan should use data-plane operations during the disaster, and call control planes only for steps that have been rehearsed.

Managed services that use it

Running it yourself

Look-alikes that are not this mechanism

---

Part 13. What you've learned

Back to Frankfurt at 20:00

The naive night cost about 25 minutes of broken checkout, 2.3 months of budget, and two writers. Seatly's Friday cost 6.75 minutes (4.9 with pre-approved rules). Here is what each piece did:

• Detection from outside (Part 2) ignored the 18:30 split, because only one vantage point failed and Frankfurt's CPS and lease stayed healthy (R2). At 20:00 both vantage points failed and CPS went missing: the rule fired at 20:01, from Ireland and Stockholm, without touching us-east-1.
• The gate (Part 3) refused the us-east-1 share of the default map, confirmed quota and prerequisites, and waited for a person, while the warm pool started anyway (R3).
• The lease and the epoch (Part 4): Frankfurt stopped itself at 20:00:09; Ireland took the lease at epoch 8 only after ack_until + 5 s, promoted, and bumped writer_epoch before any customer write (R4).
• The flip (Part 5) went through the third ARC endpoint and was gated on eu-west-1-writable; the DNS tail brought the bulk of clients over by 20:06:45.
• The orders caught in the middle (Part 7): 120 of the 160 lost orders came back through the app journal with the same IDs; 40 and one double-sold seat went to reconciliation.
• Frankfurt came back refusing (R5), was rebuilt as a replica, had its queue drained against Ireland's inbox, and got its home back by switchover at epoch 9 (R6).

What it costs

1. Data: 160 acknowledged orders missing until reconciliation, one seat sold twice; RPO 0 would cost a cross-Region round trip on every contested write.
2. Time: 6.75 minutes (62.5% of a 99.99% month) with a human gate, or 4.9 (45.5%) with pre-approved rules.
3. Money: a warm pool of 45 stopped instances (volumes only) plus 15 running, where a hot standby would keep all 60 running; ARC at $2.50 × 730 ≈ **$1,825 a month**.
4. Discipline: an epoch on every write, a refusal that never expires, quota parity, and monthly rehearsals.

The whole story, event by event

Side rows and side replays are marked "(side)".

The cheat card

Failure checklist

• Does any detection signal, dashboard or runbook step depend on the failed Region or on us-east-1?
• Does the rule require two vantage points and the business metric, and does a missing report count as down?
• Can DNS, a health check or an accelerator move users on its own to a Region that can't take writes?
• Does the old home stop by its own monotonic clock, and refuse writes on restart unless the lease names it, forever?
• Is the taker's margin larger than the worst gap between the holder's and the taker's clocks, and is each host's offset alarmed below half the margin?
• Is the epoch checked inside the new home's database on every write transaction, and bumped before the first one?
• Is the standby's quota at parity with the home, counted against what will be running after the warm pool and the surge start?
• Is every copy in the target map (standbys, witnesses, backups, keys) inside the allowed zone?
• Are keys, secrets, certificates and images already present in the standby Region?
• Do secondary endpoints store and forward everything they accept?
• After failover, do the old Region's workers stay stopped and its queues drain against the home's inbox?
• Has every step that calls a control plane in the target Region been rehearsed this quarter?

Think-first drills

Interview questions

Where to go next

• Leases, Fencing Tokens & Distributed Locks: the lease and margin arithmetic behind Part 4 (Part 7 there), and why an MREC table can't hold a lock (Part 8 there).
• Idempotency & Effectively-Once Processing: derived IDs, resuming unknowns and inboxes, which Parts 7 and 8 only count.
• Change Streams & the Transactional Outbox: the relay that resumes under the new epoch (Part 8).
• Replication, Quorums & Read-Your-Writes: what MREC, MRSC and Aurora Global Database promise (Parts 1 and 7).
• Event Time, Watermarks & Checkpoints: moving stream jobs with their home (Part 11).
• Write-Ahead Log, fsync & Group Commit: durability of one copy, the layer below all of this.
• Sharding, Hot Keys & Rebalancing: partitioning and the permanent "moved" refusal this page applies to a Region (Parts 1, 4 and 9).
• Drill: The Booking That Existed in Frankfurt but Not in Virginia, both questions answered in Part 7.
• Loops that move a Region: the key-value store (steps 3.1, 3.3, 3.4, 3.5), the message queue (steps 3.1, 3.2, 3.3), chat (step 3.5), payment (step 3.5), the digital wallet (step 3.4), hotel reservations (step 3.6), the transactional outbox (step 3.4), ride-sharing (step 3.6), Google Drive (step 3.4), email (step 3.4), the mobile stock-trading app (step 3.3), the Netflix case study (steps 3.2 to 3.5) and the Uber case study (step 3.4).