Retries, Timeouts, Backpressure & Load Shedding

Part 0. Start here

The problem: an optional widget took the home page down

A streaming service's home page shows rows of titles, and each title shows its star rating. The ratings come from a separate ratings service, and they are optional: the page works fine without stars. One evening ratings slows from 30 ms to 5 seconds per call, and the home page stops loading for everyone. The numbers below come from step 2.3 of the Netflix case study: one API instance serves 3,333 home-page requests a second, each request calls ratings once, and all requests share one pool of 1,000 threads.

Once the pool is full, every new request waits for a thread, including the ones that never call ratings. An optional star rating has taken the whole page down.

Two more facts make it worse:

• Retries multiply. If five layers each make 3 attempts, the bottom layer sees 3 × 3 × 3 × 3 × 3 = 243 calls for one user action (AWS Builders' Library).
• The outage can outlive its cause. In an example from Bronson and colleagues' paper on metastable failures (HotOS 2021), a database is fast only below 300 queries a second, and it normally receives 280. A 10-second network outage at the switch makes clients time out and retry. After the network is back, the retries hold the load at 560 a second, and goodput stays at zero until the load drops under 150 a second or the retries drop under 20 a second.

The big picture

Synthesizing vector architecture diagram...

What to notice: three layers each make 3 attempts. Product pages never call stock, but they run on the same 400 threads as the cart views and checkouts that do. The red database is the only thing that gets slow.

What you'll be able to do after this page

1. Explain with Little's law why a slow dependency takes down callers that don't use it, and tell goodput from throughput (Part 1).
2. Set a timeout from measured latency, and send each attempt's own deadline down the call chain (Part 2).
3. Decide which errors to retry, where, and how many times, and count the retries you didn't configure (Part 3).
4. Spread retries and periodic work with jitter, and say what jitter does not fix (Part 4).
5. Cap retries with a budget, and hedge a slow replica without adding load in an overload (Part 5).
6. Let a dependency refuse work early and cheaply, find its real queue, and put priority and an age limit there (Part 6).
7. Configure a circuit breaker per operation, and predict how it behaves on a dependency that is slow rather than dead (Part 7).
8. Size a bulkhead with Little's law, and know when it saves you and when it is only a backstop (Part 8).
9. Degrade the optional part of a page instead of failing the whole page (Part 9).
10. Diagnose a failure that outlives its trigger, and get out of it (Part 10).
11. Trace one checkout through every layer, with the deadline and the guard at each hop (Part 11).
12. Map every guard to AWS, count the retries AWS makes for you, and name the look-alikes (Part 12).

You may have arrived from a step that relies on this: step 2.3 of the Netflix case study (one slow service took down the home page), step 3.1 of the Shopify case study (a slow payment provider ties up all the workers), step 3.4 of the rate limiter loop (the backend is slow, but nobody is over their limit), step 2.6 of the job scheduler loop (a target is slow and we're making it worse) or step 2.5 of the chat loop (6.7 million phones reconnect at once). This page is the "why" behind all five, and behind the retry, timeout and shedding lines in 36 of the 37 interview loops.

---

Part 1. Why slow is worse than down

A dependency that is down fails fast: the caller gets an error in a millisecond and moves on. A dependency that is slow holds everything that waits for it. This Part names the one law behind that, the one measurement that tells a healthy system from a busy one, and the example the rest of the page follows.

Little's law

$$\text{requests in flight} \;=\; \text{arrival rate} \;\times\; \text{time each one spends inside}$$

It holds for any stable system: a thread pool, a connection pool, a queue, a whole service. The hook used it twice: 3,333 a second × 0.03 s ≈ 100 threads in use, and 3,333 × 5 s = 16,665 threads needed. A call that gets 100 times slower needs 100 times the threads. When the pool runs out, every request waits for a thread, whatever it was going to do.

Two consequences run through the whole page:

• A thread (or a connection, or a buffer) is held for the whole wait, retries and backoff included.
• A longer timeout makes this worse, not better. It raises the time each request spends inside, so the same arrival rate needs more threads.

Goodput is not throughput

Throughput counts everything a system finishes. Goodput counts only the requests answered successfully within the caller's deadline. The difference is waste: a query that finishes after the user has closed the page, or a reply to an attempt the caller has already retried. During an overload a service can run at 100% throughput and 0% goodput. That is exactly what happens below.

The example we follow

A real company has hundreds of services; that is too many to watch. So one small story runs through the whole page: one slow dependency turns a retry storm into an outage. Every trace on this page was produced by running a private reference simulation of this setup (apps, four shop-api instances with their threads, stock with its workers, a database with its connections, and every guard as a setting), with two random seeds, not worked out by hand. The example's numbers are smaller than the hook's so that the whole fleet fits on one line.

Basketly sells groceries online. Its shop-api serves product pages, carts and checkouts. It calls prices (always healthy here) and stock. stock answers every request with one query to its own database. On Friday at 20:00:00 the database's storage has a 20-second hiccup: every query takes ten times longer. At 20:00:20 it is healthy again.

Each Part adds one guard to the list above and replays the same 30 seconds: C1 timeouts and per-attempt deadlines (Part 2), C2 one retry layer (Part 3), C3 backoff with jitter (Part 4), C4 a retry budget (Part 5), C5 stock protects itself (Part 6), C6 circuit breakers (Part 7), C7 a bulkhead (Part 8) and C8 a brownout (Part 9). Each setting keeps all the guards before it.

The story in eleven beats: the site falls over and stays down (Part 1); bounded waits (Part 2); one place to retry (Part 3); jitter (Part 4); a retry budget (Part 5); stock protects itself (Part 6); breakers (Part 7); bulkheads (Part 8); brownouts (Part 9); why it stayed down (Part 10); one checkout, end to end (Part 11). Each Part shows only its own slice of events; the full table is in Part 13.

Words we use

The same 30 seconds, as shipped (C0)

Snapshot W1, 19:59:55: a normal Friday evening (seed 1; rates are per second)

Every tier is far below its limit. Little's law predicts the threads: 800 × 0.010 s + 1,260 × (0.010 + about 0.0042) s ≈ 8 + 18 = 26 busy on average (the run, time-averaged over 19:59:46 to 19:59:54: about 26; 28 at this instant).

Snapshot W2, 20:00:01: one second in (seed 1)

Why it can't drain

Why doesn't the site recover at 20:00:20, when the database is fast again? One inequality says it. The apps now send about 3 × 2,060 ≈ 6,180 attempts a second (each action times out twice and tries three times). How many can shop-api finish? It is limited by the database: 2,000 queries a second, and 1,260 of every 2,060 requests (61%) need one. So it can finish at most 2,000 ÷ (1,260 ÷ 2,060) ≈ 3,270 requests a second. 6,180 > 3,270: work arrives faster than it can be finished, so the queue grows by the difference whatever caused it. And every request that reaches the front has waited far longer than 2 s, so its app has already given up: the database runs at 100% on work nobody wants.

Snapshot W3, 20:00:25: the trigger is gone, the outage isn't

Synthesizing vector architecture diagram...

What to notice: the line that starts near 2,060 is goodput. It falls to zero within two seconds of the hiccup and stays there after the database recovers at second 20. The line that starts at zero is database work finished for nobody: about 200 a second during the hiccup, then about 2,000 a second, the database's full capacity, once it is healthy. The database is busy and useless. (Seed 1; the axis is not evenly spaced after second 5.)

---

Part 2. Timeouts and deadlines

Every call in C0 already had a timeout. They were simply wrong: too long, and set without looking at each other.

These timeouts are inverted: each callee is allowed to work longer than its caller will wait. shop-api may spend 3 s on an app attempt that is abandoned after 2 s, and stock may spend 15 s on a call that shop-api drops after 1 s. In this trace the database layer's 5 s never fires (a query waits about 0.6 s for a connection: 120 waiting ÷ 8 connections × 40 ms; under 0.8 s at the longest in the run), so the damage comes from the other two hops: stock finishing queries for calls shop-api has dropped, and shop-api serving a queue of requests whose apps have left.

A timeout from percentiles

A timeout should come from the dependency's measured latency, not from a guess. The method below is the AWS Builders' Library's ("Timeouts, retries, and backoff with jitter"):

textCopy
CHOOSE A TIMEOUT (per dependency; re-measure after every deploy)
  pick the false-timeout rate you accept              -> 0.1% of healthy calls
  read that percentile of healthy latency             -> P99.9
  pad it when P99.9 is close to P50,
      and for network distance                        -> about 2 x P99.9
  put connection setup and TLS inside the timer,
      or connect before the process takes traffic
  keep a separate, short connect timeout

Our simulation measured healthy stock calls, as shop-api sees them, over 60 s:

So C1 sets the shop-api → stock timeout to 60 ms, about 2 × P99.9: at most 0.1% of healthy calls could time out, and padding keeps a small latency rise from turning into a wave of timeouts.

Put connection setup inside the timer. The Builders' Library tells of a dependency called with a timeout of around 20 ms that timed out only right after deployments. The timer included setting up a new secure connection, which took longer than 20 ms, so new servers timed out on their first calls. The lasting fix was to open the connections when the process started, before it took traffic.

Connection timeout vs request timeout. A connection timeout bounds the TCP and TLS handshake: a host that doesn't answer at all should be given up on quickly. A request timeout bounds the wait for the answer and is the one set from percentiles.

Timeout vs deadline

• A timeout is a duration for one attempt: "wait 60 ms for this call".
• A deadline is a point in time for the whole request: "nobody wants this answer after 20:00:07.032".

Between machines the deadline travels as the time left, a duration, because clocks on different machines disagree. gRPC does exactly this: it turns the deadline into a timeout and subtracts the time already spent at each hop. (Timeouts that protect safety rather than capacity, such as leases, are a different job: see Leases, Fencing Tokens & Distributed Locks, Part 3 there.)

Each attempt's own deadline, cancelled when abandoned

C1 adds one rule at every hop: send the deadline of the attempt you are making, which is min(this attempt's timeout, the request's time left), minus a small margin. stock therefore hears "60 ms", never the whole 2 s. When the caller gives up on an attempt, it cancels it (an HTTP/2 stream reset or a gRPC cancellation), so the callee can stop. And every queue checks the deadline when it takes work off the queue.

Synthesizing vector architecture diagram...

What to notice: stock hears 60 ms, the deadline of the attempt shop-api is making, not the app's 2 s. When shop-api gives up, it cancels, and stock frees the connection instead of finishing a query nobody will read.

textCopy
SHOP-API CALLING STOCK (C1 onward)
  left = request.deadline - now
  if left <= 0: stop, the app has already given up
  budget = min(60 ms, left) - margin
  send the call with "deadline: budget" as a duration, not a clock time
  wait up to budget; if no reply: cancel the attempt, count a timeout

STOCK, AT EVERY QUEUE
  on arrival:  expires_at = my clock now + the deadline received
  at dequeue:  if now >= expires_at: drop it, never start the work
  run the query with statement timeout = expires_at - now
  on cancel from the caller: cancel the query, free the connection

Check the sum at every hop: the attempts, their timeouts and the backoff must fit inside the deadline above. For shop-api from C3 on: 2 attempts × 60 ms plus at most 25 ms of backoff is about 145 ms, well inside the app's 2.0 s.

The same 30 seconds, with bounded waits (C1)

Side row 10z: passing the whole deadline down, with the cancel lost. Same C1, but each hop sends the request's whole time left (up to 2 s) rather than the attempt's 60 ms, and the cancel never reaches stock (a proxy or client library that drops it). With the cancel working, sending the whole time left changes nothing here: the run recovers at 20:00:20, exactly like C1.

Nobody tells stock that shop-api gave up, and it believes it has up to 2 s, so it keeps working on attempts abandoned after 60 ms, and the stale-work loop of C0 comes back. The attempt's own deadline is the backstop for a lost cancel: it stops the work when the caller stops waiting, whether or not the cancel arrives. Envoy gets this right: its x-envoy-expected-rq-timeout-ms header carries the per-try timeout when one is set (the route timeout otherwise, or when hedging on per-try timeouts), never more than the route's time left.

Defaults that are too long

Why not 50 ms, and why not longer

This is the second question of the drill The Slow Recommendation Service That Took Down Checkout, on our numbers:

• Too short. With a 5 ms timeout on a healthy day, every call slower than 5 ms fails: about 31% of healthy stock calls (seeds 1 and 2: 30.7% and 30.5%), because P50 is 3.0 ms and P95 is 12.2 ms. Each false timeout becomes a retry, so a healthy dependency gets about a third more load, and some users fail for no reason.
• Too long. With a 5 s timeout, a hanging stock needs 1,260 calls a second × 5 s = 6,300 threads, and shop-api has 400.

A timeout is set from the healthy tail. "The dependency is plainly down" is a different signal, handled by a breaker or a concurrency limit (Parts 7 and 8).

---

Part 3. Retries: what, where and how many

In C0, one tap on "view cart" can become many database queries. The app tries 3 times; for each app attempt, shop-api tries stock 3 times; for each of those, stock tries its database 3 times.

Synthesizing vector architecture diagram...

What to notice: the attempts multiply, 3, then 9, then 27. This is the worst case: every attempt at every layer fails. In the C0 trace the database layer never times out, so the real product is smaller; the multiplication is still why the apps send about 6,180 attempts a second for 2,060 actions.

Which errors to retry

The AWS SDKs use the same three families (transient, throttling, non-retryable), and a throttle is not always a 429: DynamoDB's throttles are 400s that are "OK to retry".

Side effects: the same key on every retry

A checkout's reserve is a write. If its attempt times out, the reserve may or may not have happened: a timeout is an unknown, not a failure. Retry a side effect only with the same idempotency key, so the callee can return the first result instead of acting twice, and never fail over to a different provider on a timeout. Idempotency & Effectively-Once Processing owns the rules (Part 1 there for unknowns, Part 4 for how long to remember a key and the late original).

Event 11y. In C0, during the hiccup, about 3.8 checkouts a second get their reserve committed more than once by the simulator's database (about 7.5 extra commits a second): shop-api gave up on its attempt after 1 s and sent another, but stock was never told and ran both. (The app's own retries land behind the queue and commit a second time after 20:00:20.) With the idempotency key, stock returns the first result to the second. From C1 on, the run shows none, because an abandoned attempt is cancelled before it commits. The simulator's network adds no delay; in a real one a reply can still be lost after the commit, so the key stays.

When the server says when

Where to retry: one layer

Attempts multiply across layers: three layers of 3 attempts is 27 at the bottom, and the Builders' Library's example of five layers is 3⁵ = 243. The rule, from the Google SRE book: retry only in the layer immediately above the one that is failing, and let every other layer pass the failure up. When a service is overloaded, it says so with an "overloaded, don't retry" error, and its callers don't retry it.

textCopy
SHOP-API CALLING STOCK: the only layer that retries (C2)
  for attempt in 1, 2:
      reply = call stock with this attempt's deadline
      if reply is success: return it
      if reply is 503 "overloaded" or a permanent error: stop
      for a reserve: keep the same idempotency key
  return 503 + Retry-After to the app, marked "don't retry"

STOCK CALLING ITS DATABASE: one attempt, no retry
APP: retry once, only if it got no answer at all; on a 503 wait for Retry-After

Hidden retry layers

The layers you configure are not the only ones. Count these too:

---

Part 4. Backoff and jitter

At 20:00:20.000 the database recovers, and in one simulated instant 1,200 cart views fail at once. If each retries after a fixed 1 s, all 1,200 retries land together at 20:00:21.000: a second, self-made spike.

Synthesizing vector architecture diagram...

What to notice: with a fixed 1 s wait (the tall bar), all 1,200 retries fall in the one 10 ms slot starting at 1.0 s. With full jitter over 0 to 1 s (the short bars), each 10 ms slot gets about 12 (seed 1: never fewer than 5 or more than 25). One slot in ten is drawn.

Backoff, then jitter

Backoff makes each retry wait longer than the last, so a failing dependency gets fewer retries per second. Capped exponential backoff doubles the wait each time up to a cap. But clients that failed together still retry together. Jitter adds randomness so they don't. The AWS SDKs, and C3, use full jitter:

$$\text{wait before retry } n \;=\; \text{random}\big(0,\ \min(\text{cap},\ \text{base} \times 2^{n})\big)$$

C3 uses base 25 ms in shop-api (so the first retry waits 0 to 25 ms, inside the 2 s request) and base 1 s in the app.

What jitter does and doesn't do

Both are true, and they don't contradict each other. Jitter doesn't change how many retries failing clients may send: that is set by the attempt cap. When the dependency fails every call (as here, for 20 s), every allowed retry is sent, jittered or not. Under contention (clients competing for something that succeeds for some of them), jitter lets attempts succeed sooner, so fewer retries are needed: in the AWS Architecture Blog's simulation ("Exponential Backoff And Jitter", 2015) with 100 contending clients, jitter cut the number of calls by more than half.

Periodic work: a stable offset, not a fresh random wait

Replay P runs on a copy: 200 warehouse agents each re-read 50 stock counts every 60 s, 10,000 reads a minute, against a database with 740 reads a second to spare.

Synthesizing vector architecture diagram...

What to notice: the line that sits at 740 for the first 13.5 s of every minute, then drops to zero, is every agent starting at :00: 10,000 reads queue for 10,000 ÷ 740 ≈ 13.5 s. The line that never goes above 350 is stable offsets: each agent starts at hash(its ID) mod 60 s, the same second every minute, averaging 167 reads a second (seed 1; offsets are hashed, so some seconds get a few more agents).

Check your library's default

---

Part 5. Retry budgets and hedged requests

C3 still sends 1.9× the calls stock needs, because in a 20-second failure nearly every request uses its second attempt. A per-request cap ("at most 2 attempts") can't tell a blip, where retrying is cheap and helpful, from a real failure, where every retry is extra load. A budget can.

A cap per request vs a budget per client

• Per request: "at most N attempts". When the dependency fails everything, the load is N×. The SRE book's per-request budget of 3 attempts gives "a threefold increase" in the worst case.
• Per client: "retries at most 10% of my requests". When the dependency fails everything, the load is about 1.1×, and a blip that fails a few requests is still retried. The SRE book reports exactly this: a 10% per-client retry ratio reduces the growth "to just 1.1x".

textCopy
RETRY BUDGET, ONE PER SHOP-API INSTANCE (C4)
  tokens = 10                                  the bucket's size, the burst allowance
  on every first attempt:  tokens = min(10, tokens + 0.1)
  before any retry:        if tokens < 1: don't retry, fail now
                           else tokens = tokens - 1
  so retries stay near 10% of first attempts, plus a burst of 10

Real budgets

Budgets live in each client

Every one of these budgets is local: per process, per client instance or per proxy. The fleet's total is the sum.

Side row 13a. On a copy of C4 with 8 shop-api instances instead of 4, the run shows about the same 1,380 calls a second and 950 refusals. Our bucket earns per request, so the fleet's retries stay near 10% of traffic, however many instances share it. What does grow with the fleet is everything fixed per client: the burst (8 × 10 tokens instead of 4 × 10), the SDK's 500 tokens per client (about 35 transient retries each at 14 tokens: a thousand clients hold about 35,000 retries of burst), and any "N retries allowed" floor per proxy. Size those from the dependency's capacity ÷ the number of callers, or let the dependency protect itself (Part 6).

Hedging the tail

A hedged request is a retry sent early, for a different problem: one slow replica among healthy ones.

Synthesizing vector architecture diagram...

What to notice: the second call goes out only when the first is already slower than 95% of calls, to a different instance, for a read. The first answer wins and the loser is cancelled.

Replay H (on a copy, with a healthy database): each stock call independently stalls 250 ms with probability 1%, as if one instance were slow.

The first row has no timeout. With C1's 60 ms timeout and one retry instead, every stalled call becomes a timeout and a retry: P99.9 about 70 ms (P99 sits right on the 1% of stalled calls, so it swings between 30 and 50 ms by seed). Hedging after the P95 limits the extra load to about 5%, the same number as The Tail at Scale (Dean and Barroso), whose Google benchmark cut the 99.9th-percentile latency of reading 1,000 keys from 1,800 ms to 74 ms with just 2% more requests after a 10 ms hedge delay. (Their "tied requests" go further: both copies are sent, and each can cancel the other once it starts.)

The rules: hedge only idempotent reads, only after the P95, inside the same retry budget, and cancel the loser. During the hiccup the slowness is in the shared database, so every instance is slow and a hedge only adds load; with the budget empty, no hedges go out. A slow replica in a quorum read is the same case: Replication, Quorums & Read-Your-Writes.

---

Part 6. Backpressure: stockprotects itself

C4 still sends about 1,380 calls a second to a database that can do 200. Every caller setting so far has shaved repeats, but the 1,260 calls that are really needed are still 6× what the database can serve. Only stock knows how busy it is, so stock must be able to say no.

What backpressure is

A queue turns overload into delay: work waits instead of failing, and the backlog outlives the trigger (Part 1). Backpressure is the opposite reflex: the slow side's signal makes the fast side slow down or stop, instead of buffering without limit. The loops use it at every scale: the news feed's fanout workers slow down when cache latency rises (step 2.3 of the news feed loop), and the matching engine rejects a new order rather than queue a stale one (step 2.7 of the stock exchange loop). For a request/response service, the signal is a fast refusal: 503 with Retry-After.

Refuse early and cheaply

A refusal must cost far less than a success, or refusing becomes the new overload. So stock counts the requests inside it and refuses at the door, before any real work: before TLS and authentication where it can (the chat loop admits connections at TCP accept, before TLS: step 2.5 of the chat loop), and always before the database. C5's limit is 64 requests inside stock, and 80 for reserves, so checkouts still get in when views are refused.

Find the real queue

Where do requests actually wait? stock has 128 workers, and at most 80 requests are ever admitted, so a worker is always free: the worker queue never forms. The only real queue is the wait for one of the 8 database connections. So the other two tools act there:

• Bound its age (the CoDel variant in Maurer's "Fail at Scale", ACM Queue 2015): once the queue has not been empty for 100 ms (it is standing), drop every request that has waited more than 5 ms. A short burst still queues; a standing queue can't grow old.
• Serve newest first while it stands (adaptive LIFO, from the same article): the newest request is the one whose caller is most likely still waiting.
• Priority where requests wait: a reserve waiting for a connection goes before any view. Priority only at the admission limit helps little: it decides who gets in, not who gets a connection, and once inside, views and reserves would wait in the same line. (Google's SRE book gives requests one of four criticalities, carried with the request to every backend, for the same purpose.)

How many waits are even useful? Little's law again: during the hiccup the database finishes 200 queries a second, and a caller waits at most 60 ms, so only about 200 × 0.06 = 12 requests can usefully wait. A fixed limit of 64 is far above that; it helps here only because priority and the age limit act in the connection queue. An adaptive limit (Part 8) would find a number near 12 on its own.

Synthesizing vector architecture diagram...

What to notice: the refusals come from two places, the admission check at the door and the age limit in the connection queue, and both go back to shop-api as a fast 503: that arrow is the backpressure. Priority and the age limit sit in the connection queue, the only place where requests actually wait.

textCopy
STOCK (C5)
  on a call:
      limit = 80 for a reserve, else 64
      if calls inside stock >= limit: reply 503 + Retry-After   before any real work
      take a worker and join the connection wait queue
  when a connection frees:
      if the queue has been non-empty for 100 ms:
          reply 503 to every waiting call older than 5 ms
          pick from the newest end                               last in, first out
      else:
          pick from the oldest end                               first in, first out
      a waiting reserve is picked before any view
      if its deadline has passed: drop it
      else run the query with statement timeout = its time left

The same 30 seconds, with stockprotecting itself (C5)

Side row 17x: stock protecting itself from as-shipped callers. The same stock guards on C0, whose apps and shop-api retry everything at once, including 503s:

Server-side protection works even against badly behaved callers, for one reason: each refusal costs microseconds, so a checkout refused at the door soon gets through on one of its many retries. If a refusal cost as much as a query (TLS, authentication, a database read), those 7,300 refusals a second would be the new overload. That is why the refusal must be early.

Will it ever drain?

A backlog empties only at what you can finish minus what keeps arriving. With the retries of C0, arrivals (6,180 a second) exceed what can be finished (3,270), so it never empties (Part 1).

Side row 17y. A copy of C0 with no retries anywhere: arrivals are 2,060 a second, below 3,270. At 20:00:20 shop-api holds about 28,000 queued requests, and stock about 4,100 stale calls of its own. stock works those off first (by 20:00:25); then shop-api's queue falls from 28,700 at 20:00:25 to 3,900 at 20:00:45, about 1,240 a second, close to the 3,270 − 2,060 ≈ 1,210 the arithmetic predicts. It is empty at 20:00:48 to 20:00:49. The Builders' Library ("Avoiding insurmountable queue backlogs") puts it in one line: an hour-long outage needs double the capacity for another hour to catch up.

Buffers you pull from (an SQS queue, a stream) are backpressure by design: the consumer takes work at its own pace, and the producer never waits. The danger moves to the backlog's age, so alarm on the age of the oldest message, not only on how many there are. Consumer lag on a change stream is the same backlog (Change Streams & the Transactional Outbox). Visibility timeouts, redelivery and dead-letter queues belong to a later loop primitive, Queues & Delivery Semantics.

Queues in load balancers. The Classic Load Balancer queued excess requests in a surge queue of up to 1,024; a request that waited there reached the server with no idea how old it was. The Builders' Library ("Using load shedding to avoid overload") recommends failing fast instead of queueing, and says the Application Load Balancer rejects excess traffic.

503, 429 or backpressure?

Lowering every tenant's rate limit is not overload protection: when the backend is slow and nobody is over their limit (step 3.4 of the rate limiter loop), only shedding by the service's own health helps.

---

Part 7. Circuit breakers

With C5, about 157 cart views a second get their stock badges and more than 1,000 get an error, while shop-api keeps sending about 1,400 calls a second that mostly fail. A circuit breaker notices that a dependency is plainly failing, stops calling it for a while, and answers at once instead: with a fallback, or with a fast error.

The three states

Synthesizing vector architecture diagram...

What to notice: every arrow uses the example's numbers (C6). The note is the flap of event 15f: the probes test the dependency at a load far lighter than the real one.

textCopy
BREAKER, one per shop-api instance and per operation: stock.view, stock.reserve (C6)
  before a call:
      if OPEN and 5 s have passed: go HALF_OPEN (0 probes sent, 0 passed)
      if OPEN: refuse now -> fallback for a view, fast 503 for a reserve
      if HALF_OPEN: if 3 probes already sent: refuse now
                    else send this call as a probe, with a 100 ms timeout
  after a call:
      a probe: failure -> OPEN; success -> if 3 passed: CLOSED, clear the window
      in CLOSED: add the outcome to the last-20 window
                 if the window is full and at least half failed: OPEN

Count window or time window

A breaker decides on recent calls. A count window looks at the last N calls: C6 uses the last 20 per instance and operation, as resilience4j does by default (COUNT_BASED, though its default size is 100). A time window looks at the last T seconds. With a sudden total failure, a time window full of healthy history is slow to trip: a 10 s window holds about 315 × 10 ≈ 3,000 healthy calls per instance, and must see failures outnumber them before it opens, so the time to open is roughly the window × the threshold. A window of the last 20 calls trips within the first second. (Side row 15t in Part 13 runs the 10 s time window: about 345 stock calls a second get through during the hiccup instead of about 80.)

The same 30 seconds, with breakers (C6)

Probes and their timeout

The probe's timeout must sit above the healthy P99, or a recovered dependency can never pass it. Shopify's Semian names it half_open_resource_timeout, and the half-open call is a real request: for payments, a real buyer's payment, so a failed probe is an unknown to resolve like any other timeout (step 3.1 of the Shopify case study; the loop's fact-check caught a probe timeout set below the healthy 1.5 s call). The same step shows the price of probing: while a dependency stays down, each open circuit lets one real request wait up to h (the probe timeout) every open period e, about h ÷ e of a thread per failing resource. Shopify's published example multiplies that across 42 Redis instances, all down, on a 2-thread worker: "an extra utilization of 263%" with a 0.25 s half-open timeout and a 2 s error_timeout (0.25 × 42 ÷ (2 × 2) ≈ 2.63), and 4% after moving to a 50 ms half-open timeout and a 30 s error_timeout.

Key it by what fails separately

Key a breaker by dependency and operation, and further (tenant, country, route) wherever failures differ: Shopify keys its payment circuits by the merchant's country, so one country's outage doesn't close checkout elsewhere. Our view breaker may open; the reserve breaker must not open because of views. Decide in advance what each open breaker returns: a fallback for an optional call (Part 9), a fast 503 for a required one. And remember the state is per process: Semian's is explicitly not shared across workers, so each of our 4 instances needs its own evidence before it opens.

Per host, per dependency, or a cap?

Breaker or a retry budget only, on equal terms

The AWS Builders' Library prefers limiting retries locally with a token bucket over breakers: breakers add a mode that is hard to test and can add significant time to recovery. On this slow-but-alive database, with everything else equal (both runs have stock's protections and the brownout fallback of Part 9, and neither has the bulkhead):

Over five seeds, checkouts are about the same with and without breakers (about 44 without; 43 to 47 with, depending on details: 43 as above, 45 if stock's 503s don't count as breaker failures, 47 in C8 with the bulkhead). Adding the bulkhead without breakers is worse for checkouts (about 34), because one gate shared by both operations starves reserves when nothing stops the views. The breakers' real gain is 95% less load on a struggling dependency; their cost is about 150 cart views a second shown without badges that could have had them, a second of slower recovery and a mode that flaps. So: a breaker where a fallback exists or the dependency is plainly down; a retry budget always.

Drill 23, the second question (The Slow Recommendation Service That Took Down Checkout): why not 50 ms timeouts everywhere instead of a breaker? A timeout is set from the healthy tail; below it, healthy calls fail and turn into retries (Part 2). And a timeout still sends every call: each one to a dead dependency costs a thread for the full timeout and adds load. A breaker stops sending, gives the dependency room to recover, and answers the caller at once. You need both: the timeout bounds one call, the breaker bounds the calls you keep making.

---

Part 8. Bulkheads and concurrency limits

Go back to event 3: in C0, product pages waited because cart views and checkouts held all 400 threads. The pool was shared. A bulkhead (named after a ship's watertight walls) caps how many threads, connections or requests one dependency may hold, so a slow dependency can sink only its own compartment.

Synthesizing vector architecture diagram...

What to notice: inside the "shop-api instance: 100 threads" panel, only calls to stock pass the gate of 12; calls to prices go straight through. However slow stock gets, it can hold at most 12 of the 100 threads, and product pages always have the rest.

Size a cap with Little's law

Size = calls a second × the P99 latency × headroom. For stock: 315 calls a second per instance × 0.0187 s × 2 ≈ 11.8, so 12 per instance (48 across the fleet). With C1's 60 ms timeout, even if every call waited the full timeout, an instance would hold at most 315 × 0.06 ≈ 19 calls. A timeout bounds each call; the cap bounds how many.

A backstop, and when it saves you

Why does 40 work so well and 60 not? With 1 s timeouts, 40 × 4 = 160 calls in flight, served at the slow database's 200 a second, take about 160 ÷ 200 = 0.8 s each (Little's law): inside the 1 s timeout. At 60 per instance, 240 in flight take about 1.2 s each: every call waits past its 1 s, and all of them fail. Either way the cap protects the other traffic (product pages) and ends the outage at 20:00:20, which C0 never does. That is a bulkhead's job: the backstop for a timeout that turns out to be wrong.

(In the recommended order, the bulkhead comes after the breakers. Placed before them, on C5, a cap of 12 refuses about 290 to 300 calls a second, but checkouts fall from about 44 to about 33, because the one cap is shared by views and reserves. A cap per operation avoids that.)

Pools and semaphores

A semaphore caps how many of the caller's own threads may be inside calls to one dependency. A separate thread pool per dependency does the same with its own threads, and lets the caller walk away from a blocked call, at the price of more threads and context switches. Neither can end a call that is already blocked: that is what the timeout beside it is for.

Caps multiply with the fleet

A cap is per instance, so the fleet's total is cap × instances. Replay A scales shop-api from 4 to 8 instances at 20:00:10, the moment an autoscaler might react. On C7 it changes nothing visible (the breakers keep 2 to 4 calls in flight; recovered at 20:00:21), but the fleet-wide cap has doubled from 48 to 96 at exactly the moment stock is weakest. On C0 it doubles the threads waiting on stock (400 → 800): at 20:00:15 about 5,200 calls are inside stock, against 3,100 to 3,200 without the extra instances, and goodput stays at zero. Size caps from the dependency's capacity ÷ the number of callers, and let the dependency enforce its own limit too (Part 6).

Limits that adapt

A fixed cap goes stale as deploys and scaling change the system. Two adaptive forms, one sentence each:

• At the client, AIMD (additive increase, multiplicative decrease, as in TCP): add 1 to the limit after each healthy interval; halve it on timeouts, 429/503 or latency above a target, at most once per interval. The job scheduler does this per target (step 2.6 of the job scheduler loop).
• At the server, latency-based: Netflix's concurrency-limits library estimates the queue from how far current latency is above the minimum (its Vegas limit estimates the queue as limit × (1 − minRTT ÷ sampleRTT), adding 1 to the limit when that queue is small and subtracting 1 when it is large) and refuses the excess at once. For our stock it would cut the limit sharply during the hiccup; the page doesn't simulate where it settles.

Managed bulkheads on AWS: Lambda reserved concurrency is both the maximum and the minimum concurrency of one function, carved out of the account's Regional pool (1,000 by default, shared by every function); an SQS event source's maximum concurrency caps how many invocations that one queue can drive, and should be set no higher than the function's reserved concurrency.

Drill 23, the first question (The Slow Recommendation Service That Took Down Checkout): why does a gateway run out of threads when an optional service slows from 30 ms to 5 s? Little's law: threads in use = rate × time held. At 30 ms the calls hold a few threads; at 5 s they need about 167 times as many, and in one shared pool they take every thread, so requests that never call the slow service wait too. A per-dependency cap keeps the rest of the pool free.

---

Part 9. Brownouts and fallbacks

After C7, about 1,190 cart views a second fail fast: the breaker is open or the gate is full, and the view has nowhere to go. But a cart view needs its items and prices; the stock badges are optional. A brownout drops the optional part and serves the rest.

Synthesizing vector architecture diagram...

What to notice: the same five kinds of failure lead to two different answers, decided in advance per operation: a degraded cart page for the optional call, a fast, honest refusal for the required one.

Snapshot W4, 20:00:05: in the hiccup, at a flap (C8, seed 1; rates are per second)

Snapshot W5, 20:00:21: recovered (C8, seed 1)

Three kinds of fallback

Netflix's API (step 2.3 of the Netflix case study) names three, decided per call in advance:

Fallbacks that never run don't work. Exercise them on purpose (step 2.5 of the Netflix case study: failures surprise us in production). Two things are never shed: the load balancer's health checks, because a server that ignores them is taken out of service just when every server is needed; and emergency paths, which the ride-sharing loop exempts from shedding (step 3.5 of the ride-sharing loop).

Why checkouts top out near 50

Even with no queue at all, a reserve attempt has 60 ms, and during the hiccup a query takes 40 ms on average, with an exponential spread. The chance that one query takes longer than 60 ms is e^(−60 ÷ 40) = e^(−1.5) ≈ 22%. The run agrees: 22% and 23% of reserve attempts timed out. A second attempt helps (at best 1 − 0.22² ≈ 95% of checkouts would succeed), but the age limit, the shared gate and an occasional open reserve breaker take their share, so about 44 to 50 of 60 get through. To do better, give the checkout a longer attempt timeout (a checkout is worth more than a badge), or let fewer views compete for the 8 connections.

Shedding and autoscaling

Shedding hides the signal autoscaling waits for. The Builders' Library warns that a service shedding at a CPU target keeps its CPU at that target, so CPU-based scaling never fires; and fast refusals make the median latency look excellent while successful requests are slow. So scale and alarm on offered load, shed counts, breaker opens and budget refusals, and report latency of successes separately. Shedding also doesn't replace headroom: Amazon services keep enough spare capacity to handle an Availability Zone failure without adding capacity, and shedding can hide how close a fleet is to that limit.

---

Part 10. Why it stayed down: metastable failures

Look at snapshot W3 again: at 20:00:25 the database is healthy, busy at full capacity, and serving nobody. The trigger lasted 20 seconds; C0's outage would have lasted until someone intervened. This Part names that kind of failure, the loops that sustain it, and the ways out.

Stable, vulnerable, metastable

Bronson and colleagues ("Metastable Failures in Distributed Systems", HotOS 2021) describe three states:

Synthesizing vector architecture diagram...

What to notice: nothing goes wrong in the vulnerable state; it only needs a trigger. And there is no arrow from Metastable back to Vulnerable when the trigger ends: the only way out is a push from outside.

• Stable: even the worst amplification the system can produce (every retry, every stale request) still fits in its capacity.
• Vulnerable: normal load is fine, but the amplified load would not fit. Basketly on Friday evening is vulnerable: 2,060 actions a second is fine, but C0's 3 × 2,060 = 6,180 attempts a second is not.
• Metastable: a trigger pushed it over, and a sustaining effect keeps it there after the trigger is gone.

Systems run in the vulnerable state because it is cheaper: higher utilization, fewer servers. The job of the guards on this page is to make the amplification small enough, or breakable enough, that the vulnerable state is safe.

The loops that kept it down

1. Retries (event 4): about 6,180 attempts a second against about 3,270 a second that can be finished.
2. Stale work (event 21, below): even with no retries at all, shop-api served its queue first in, first out, for callers who had left, and recovered only 25 s after the trigger ended.
3. A cache that can't refill, the look-aside cache example in Bronson's paper (its second, section 2.2): behind a look-aside cache with 90% hits, the database sees a tenth of the reads. Lose the cache while the system is vulnerable and the database sees ten times as many; the reads time out, nothing gets back into the cache, and the system stays down. Cold caches and stampedes belong to Caching & Invalidation (Parts 5 and 8 there); coalescing identical reads is in Sharding, Hot Keys & Rebalancing (Part 4 there).

Synthesizing vector architecture diagram...

What to notice: three lines, seed 1, every other second. The line near 2,060 all the way across is C8, counting cart views served without badges; the line that sits near 850 until second 20 and then joins it is C8 counting only full answers (the gap is the 1,190 degraded cart views a second). The line at zero from second 2 onward is C0: it never comes back.

Getting out

At 20:02:00, two minutes into C0's outage, with about 403,000 requests queued in shop-api, two operator actions were tried on copies:

Getting out needs two things together: load below capacity, and the stale backlog thrown away. Shedding new arrivals does the first but not the second; a flush does the second, and the load then usually fits. It isn't guaranteed while the as-shipped retries remain: every flushed attempt times out 2 s later and is retried at once, and that wave alone can tip the site straight back. So flush and stop the retries (or switch on deadlines) together; and the next trigger will start it again if nothing else changes. Dropping expired work by deadline (C1) is the same flush, done continuously.

Why "add capacity" didn't help

Replay A (Part 8) doubled shop-api at 20:00:10: on C0 goodput stayed at zero, and about 60% more calls were waiting inside stock than without the extra instances. More callers put more pressure on the slow dependency, and capacity that arrives in minutes can't beat a storm that arrives in seconds. The loops' fact-checks keep catching "autoscale" as the fix (for example, step 2.6 of the mobile stock-trading loop, the opening bell).

Every guard on equal terms

What each guard is:

What the run shows, each guard alone on the as-shipped settings (seed 1):

Four guards prevent it on their own; only two end it once it has started, because only they stop shop-api's threads from being consumed by the backlog. That is why the page stacks guards: each layer should be able to break the loop by itself.

---

Part 11. End to end through the layers

With C8, every hop bounds time, repeats and concurrency. Follow one checkout and one cart view through all of them at 20:00:07, in the middle of the hiccup (seed 1, event 23).

Synthesizing vector architecture diagram...

What to notice: the deadline shrinks at every hop, 2.0 s at the app, 60 ms at stock, whatever is left at the database. The highlighted boxes, shop-api and stock, are the two services that refuse work: one on behalf of its callers, one on its own behalf.

The trace (seed 1, C8):

The deadline at each hop

Which guard acts where

Sizing for an Availability Zone loss

stock's 128 workers run in two Availability Zones, 64 in each. Lose one zone during the hiccup and 64 remain. On a copy of C8 with 64 workers, the run is unchanged: checkouts 44 to 50 a second, cart views degraded, back at 20:00:21. At most 80 calls are ever inside stock, and the 8 database connections, not the workers, are the limit; on a healthy day stock needs only about 1,260 × 0.005 ≈ 6 workers. With the admission limits also halved (32, reserves 40), checkouts are 47 to 50. So stock can lose a zone with views degraded and still serve every checkout the database can. The database's own zone failure is a different mechanism: see Replication, Quorums & Read-Your-Writes.

---

Part 12. On AWS

Every guard on this page exists somewhere in AWS, often switched on by default, sometimes with a default you should change. Three rules carry over: count every layer's retries, remember which limits are per client and which are per account, and know that you still choose the deadlines, the one retry layer, the priorities and the fallbacks.

Managed services that use it

The AWS SDKs (not a service, so not a chip) are the retry layer most applications already have:

The updated behaviour can be switched on now with AWS_NEW_RETRIES_2026=true and becomes the default in November 2026 (AWS, 20 May 2026), which also moves the legacy-mode SDKs to standard mode.

Running it yourself

Look-alikes that are not this mechanism

---

Part 13. What you've learned

Back to the home page

An optional star rating took a home page down because waiting held threads (Little's law: 3,333 a second × 5 s = 16,665 threads needed from a pool of 1,000) and failure multiplied work (retries at every layer, and answers finished for callers who had left). Basketly's 20-second hiccup did the same, and C0 never came back. With every guard in place (C8), the same 30 seconds kept product pages whole, served about 1,190 cart views a second without badges, confirmed 44 to 50 of 60 checkouts a second, and was back to normal one second after the database. Here is what each piece did:

• Timeouts from percentiles and per-attempt deadlines (Part 2) bounded how long anyone waited, and dropped stale work instead of serving it: C1 alone recovered at 20:00:20.
• One retry layer (Part 3) cut the calls reaching stock from 5.4× to 1.9×; the idempotency key made the reserve's retries safe.
• Jitter (Part 4) spread retries in time without removing any; stable offsets kept periodic work flat and fresh.
• A retry budget (Part 5) brought retries down to about 1.1×.
• stock protecting itself (Part 6) refused early and gave its 200 queries a second to reserves and to fresh requests: checkouts from 9 to 44 a second.
• Breakers per operation (Part 7) cut the load on stock by 95%, at the price of a flap every 5 s.
• A bulkhead (Part 8) was a backstop on the main line, and the thing that saved the site when the timeout was wrong.
• A brownout (Part 9) turned 1,190 errors a second into carts without badges.

The snapshots, side by side

What it costs

1. False timeouts on a healthy day (at most 0.1% of stock calls with a 60 ms timeout), each one a retry.
2. Requests refused while some capacity is left: admission limits, age limits, gates and budgets all refuse work that might have succeeded.
3. A degraded page: 1,190 carts a second without badges for 20 s, and some customers who reach checkout and find an item gone.
4. Tuning per dependency: timeouts, caps, windows and probe timeouts come from measurements, and go stale after deploys.
5. Counters that don't know the fleet's size: breakers, budgets and caps are per process, so fixed allowances grow with every instance you add.
6. Discipline: deadlines and cancellation at every hop, idempotency keys on every retried write, fallbacks that are exercised.

The whole story, event by event

Side rows and side replays are marked "(side)". Seeds 1 and 2 unless stated.

The cheat card

Failure checklist

• Is every timeout set from a measured percentile, with connection setup inside it or done before taking traffic?
• Does every hop send the deadline of the attempt it is making, cancel what it abandons, and drop expired work at dequeue?
• Is exactly one layer retrying each dependency, and have you counted the proxies, SDKs, CDNs and managed runtimes that also retry?
• Does every retried side effect carry the same idempotency key, and is a timeout treated as an unknown?
• Do all retries use jitter, and does periodic work use stable offsets?
• Is there a retry budget, and is any fixed per-client allowance sized for the whole fleet?
• Can each dependency refuse work before its expensive part, with 503 and Retry-After, and do callers honour it?
• Do priority and age limits sit in the queue where requests actually wait?
• Is each breaker keyed by operation, with a probe timeout above the healthy P99?
• Does every dependency have a cap sized by rate × P99 × headroom, and is it checked against the dependency's capacity ÷ the fleet?
• Does every optional call have a tested fallback, and are health checks and emergency paths exempt from shedding?
• Do alarms and scaling use offered load, shed counts, breaker opens and queue age, not only CPU and median latency?

Think-first drills

Interview questions

Where to go next

• Idempotency & Effectively-Once Processing: keys, the retry horizon and resolving unknowns, which Part 3 only recaps (Parts 1 and 4 there).
• Sharding, Hot Keys & Rebalancing: per-key admission with 429 and Retry-After, single-flight and coalescing (Parts 3 and 4 there); this page handles a whole dependency, that one a single hot key.
• Multi-Region Failover: the Region-level version: client backoff after a routing flip (Part 5 there) and cells as a blast-radius unit (Part 9 there).
• Leases, Fencing Tokens & Distributed Locks: timeouts that protect safety rather than capacity (Part 3 there).
• Change Streams & the Transactional Outbox: consumer lag is a backlog with a drain time.
• Replication, Quorums & Read-Your-Writes: the slow replica that hedging works around.
• Caching & Invalidation: stampedes (Part 5 there) and the cold cache that can keep a database down (Part 8 there), the third loop of Part 10.
• Coming later: Queues & Delivery Semantics (visibility timeouts, redelivery, dead-letter queues) and Rate Limiting Algorithms (token buckets, sliding windows, distributed counters).
• Drill: The Slow Recommendation Service That Took Down Checkout, both questions answered in Parts 7 and 8.
• Background: Circuit Breaker, Bulkhead & Fault Tolerance Patterns, the broader primitive.
• Loops that rely on this page: the key-value store (R2.8 and R2.9), the rate limiter (steps 1.4, 1.5, 2.4, 3.4), the news feed (step 2.3), chat (steps 1.6, 2.5), notifications (steps 1.2, 2.1, 2.4, 3.3), the S3-like object store (steps 2.4, 3.6), search autocomplete (step 2.4), the web crawler (step 3.4), the job scheduler (steps 1.4, 2.3, 2.6), payments (steps 1.2, 2.4), the stock exchange (step 2.7), hotel reservations (step 2.2), ride-sharing (step 3.5), the offline-first news feed (step 2.5 and R2.8), the mobile stock-trading app (steps 2.6, 3.2), the paging library (R2.8), and the Netflix (steps 2.3, 2.5), Uber (step 1.3) and Shopify (steps 2.3, 3.1, 3.4) case studies.