The Home Region That Was Slow but Not Down
The Home Region That Was Slow but Not Down
This situation follows the lesson Multi-Region Failover. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Mealo is a food-delivery company in the United States. Customers order from 1,200 restaurants through the app, and couriers are assigned to orders. At lunch, from 12:00 to 13:00, Mealo takes 150 orders a second. Every order is one write transaction: the order row and the courier assignment.
Mealo's setup follows the design of the lesson Multi-Region Failover, with the United States as its only residency zone:
| Setting | This challenge | At real scale |
|---|---|---|
| Regions | Home: Oregon (us-west-2), which writes every order. Warm standby: Ohio (us-east-2). Third copy of the control table and second set of canaries: Canada Central (ca-central-1), with no customer data | A platform with several homes, one per country |
| Orders and assignments | Aurora PostgreSQL Global Database: writer in Oregon, secondary in Ohio, replication lag 0.5 s | Lag "typically measured in seconds" |
| Control table | A DynamoDB global table in multi-Region strong consistency (MRSC) mode with full replicas in Oregon, Ohio and Canada Central. It holds the item us-writer = {home, epoch, ack_until, rvn} and each Region's business metric | MRSC runs in exactly three Regions |
| Writer lease | Oregon's app servers renew every 2 s, at the odd seconds (11:59:59, 12:00:01, 12:00:03, …), writing ack_until = send time + 10 s. A taker's condition is epoch = :seen AND ack_until < :taker_wall_now − 5 s. Oregon holds epoch 12; the clocks agree | Same as in the lesson |
| Epoch | A writer_epoch row in the Aurora database, read in every write transaction; 12 | Same |
| Canaries | In Ohio and Canada Central, a checkout probe every 10 s with a 5 s timeout; a run that times out is a miss, recorded when its timeout ends; a vantage point fails after 3 misses in a row | Same |
| Business metric | Each Region writes its orders per second to the control table every 10 s, covering the previous 10 s. Oregon's normal lunch band is 100 to 200. "Down" means below the band, or missing, for 60 s | Same |
| Evacuation rule | Both vantage points failed and the metric down or missing, or the home has self-fenced and the metric is down; it runs in Ohio and in Canada Central | Same |
| Gate | Six checks as in the lesson's section Snapshot R3: the gate at 20:01; a person approves in 2 min, or pre-approved rules in 10 s when all six checks pass; no approval before ack_until + 5 s on the taker's clock; above 1,000 orders at risk the business owner must sign | Same |
| Routing | Amazon Application Recovery Controller (ARC) routing controls us-west-2-on and us-east-2-on behind the Route 53 failover records for api.mealo.com, with the gating control us-east-2-writable; TTL 60 s; retry backoff capped at 30 s; the flip reaches DNS in 15 s | Same |
| Capacity | Ohio's warm pool is InService 60 s after the rule fires, with quota to spare | Same |
| Promotion | An Aurora failover or an Aurora switchover to Ohio takes 120 s (an assumption, the same number the lesson uses) | Same |
| Redirect mode | A Region's app servers that are not the home answer every write with a redirect to the current home; a redirected request succeeds with one extra hop | Same |
| Budget | 99.99% of a 30-day month = 43,200 × 0.0001 = 4.32 minutes. An outage is weighted by the share of customers affected; while a share of checkouts fails, that share of customers counts as affected (a modelling choice) | Same |
| Client timeout | The app gives a checkout 5 s, then shows an error | Same |
Synthesizing vector architecture diagram...
us-west-2-on is On, so they reach Oregon's app servers. Those servers write to the Aurora writer, whose storage layer is slow from 12:00, and they keep renewing the writer lease every 2 s in the control table, drawn in the panel Canada Central, because DynamoDB in Oregon is not affected. Replication to the Aurora secondary in the panel Ohio, warm standby still runs with a 0.5 s lag, because committed writes replicate as before; only committing is slow. The canaries in Ohio and Canada Central call Oregon's checkout path every 10 s, and their calls time out. So the probes say "dead", the lease says "alive", and both are telling the truth about what they measure.What happens. At 12:00:00 the storage layer of Oregon's Aurora cluster slows down. The cause is in the storage layer that every instance of the cluster shares, so promoting a reader instance in another Availability Zone would not help; Oregon's RDS events say so from 12:01. Checkout latency at the 99th percentile goes from 300 ms to 9 s, and 60% of checkouts time out at the app's 5 s limit. Oregon's app servers, its DynamoDB replica and the network are healthy. The canary run that started at 12:00:00 finished just before the slowdown and succeeded; every later run times out. Oregon's report at 12:00:10, covering 12:00:00 to 12:00:10, says 60 orders a second. AWS repairs the storage layer at 12:40, but nobody knows that at 12:00.
Your task
Say what Mealo's automation does from 12:00:00, what stops it, and what you do instead. A complete answer gives: when the rule fires and when each of the six gate checks passes or fails, with the arithmetic; what a forced failover would lose and what it would risk; the move you choose, step by step, with its recovery time and the customer-weighted minutes it spends against the 4.32-minute budget; and the one risk that move still carries.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Home Region That Was Slow but Not Down
This situation follows the lesson Multi-Region Failover. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Mealo is a food-delivery company in the United States. Customers order from 1,200 restaurants through the app, and couriers are assigned to orders. At lunch, from 12:00 to 13:00, Mealo takes 150 orders a second. Every order is one write transaction: the order row and the courier assignment.
Mealo's setup follows the design of the lesson Multi-Region Failover, with the United States as its only residency zone:
| Setting | This challenge | At real scale |
|---|---|---|
| Regions | Home: Oregon (us-west-2), which writes every order. Warm standby: Ohio (us-east-2). Third copy of the control table and second set of canaries: Canada Central (ca-central-1), with no customer data | A platform with several homes, one per country |
| Orders and assignments | Aurora PostgreSQL Global Database: writer in Oregon, secondary in Ohio, replication lag 0.5 s | Lag "typically measured in seconds" |
| Control table | A DynamoDB global table in multi-Region strong consistency (MRSC) mode with full replicas in Oregon, Ohio and Canada Central. It holds the item us-writer = {home, epoch, ack_until, rvn} and each Region's business metric | MRSC runs in exactly three Regions |
| Writer lease | Oregon's app servers renew every 2 s, at the odd seconds (11:59:59, 12:00:01, 12:00:03, …), writing ack_until = send time + 10 s. A taker's condition is epoch = :seen AND ack_until < :taker_wall_now − 5 s. Oregon holds epoch 12; the clocks agree | Same as in the lesson |
| Epoch | A writer_epoch row in the Aurora database, read in every write transaction; 12 | Same |
| Canaries | In Ohio and Canada Central, a checkout probe every 10 s with a 5 s timeout; a run that times out is a miss, recorded when its timeout ends; a vantage point fails after 3 misses in a row | Same |
| Business metric | Each Region writes its orders per second to the control table every 10 s, covering the previous 10 s. Oregon's normal lunch band is 100 to 200. "Down" means below the band, or missing, for 60 s | Same |
| Evacuation rule | Both vantage points failed and the metric down or missing, or the home has self-fenced and the metric is down; it runs in Ohio and in Canada Central | Same |
| Gate | Six checks as in the lesson's section Snapshot R3: the gate at 20:01; a person approves in 2 min, or pre-approved rules in 10 s when all six checks pass; no approval before ack_until + 5 s on the taker's clock; above 1,000 orders at risk the business owner must sign | Same |
| Routing | Amazon Application Recovery Controller (ARC) routing controls us-west-2-on and us-east-2-on behind the Route 53 failover records for api.mealo.com, with the gating control us-east-2-writable; TTL 60 s; retry backoff capped at 30 s; the flip reaches DNS in 15 s | Same |
| Capacity | Ohio's warm pool is InService 60 s after the rule fires, with quota to spare | Same |
| Promotion | An Aurora failover or an Aurora switchover to Ohio takes 120 s (an assumption, the same number the lesson uses) | Same |
| Redirect mode | A Region's app servers that are not the home answer every write with a redirect to the current home; a redirected request succeeds with one extra hop | Same |
| Budget | 99.99% of a 30-day month = 43,200 × 0.0001 = 4.32 minutes. An outage is weighted by the share of customers affected; while a share of checkouts fails, that share of customers counts as affected (a modelling choice) | Same |
| Client timeout | The app gives a checkout 5 s, then shows an error | Same |
Synthesizing vector architecture diagram...
us-west-2-on is On, so they reach Oregon's app servers. Those servers write to the Aurora writer, whose storage layer is slow from 12:00, and they keep renewing the writer lease every 2 s in the control table, drawn in the panel Canada Central, because DynamoDB in Oregon is not affected. Replication to the Aurora secondary in the panel Ohio, warm standby still runs with a 0.5 s lag, because committed writes replicate as before; only committing is slow. The canaries in Ohio and Canada Central call Oregon's checkout path every 10 s, and their calls time out. So the probes say "dead", the lease says "alive", and both are telling the truth about what they measure.What happens. At 12:00:00 the storage layer of Oregon's Aurora cluster slows down. The cause is in the storage layer that every instance of the cluster shares, so promoting a reader instance in another Availability Zone would not help; Oregon's RDS events say so from 12:01. Checkout latency at the 99th percentile goes from 300 ms to 9 s, and 60% of checkouts time out at the app's 5 s limit. Oregon's app servers, its DynamoDB replica and the network are healthy. The canary run that started at 12:00:00 finished just before the slowdown and succeeded; every later run times out. Oregon's report at 12:00:10, covering 12:00:00 to 12:00:10, says 60 orders a second. AWS repairs the storage layer at 12:40, but nobody knows that at 12:00.
Your task
Say what Mealo's automation does from 12:00:00, what stops it, and what you do instead. A complete answer gives: when the rule fires and when each of the six gate checks passes or fails, with the arithmetic; what a forced failover would lose and what it would risk; the move you choose, step by step, with its recovery time and the customer-weighted minutes it spends against the 4.32-minute budget; and the one risk that move still carries.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).