An optional widget exhausts the checkout pool

Recommendations go slow in a hypothetical shop. Checkout itself is healthy, but both routes borrow connections from the same pool. Recommendations fills the pool; checkout starts failing too.

You have a dependency problem and a resource-sharing problem. A timeout alone addresses only part of either.

Give the failure a boundary

A bulkhead allocates separate pools or semaphores to independent work. A concurrency limit bounds in-flight requests. A circuit breaker decides whether to attempt calls based on recent health. These can cooperate, but none supplies unlimited capacity.

pseudocodeCopy
on request arrival:
    reject if deadline expired
    try acquire route/dependency permit
    if permit unavailable:
        reject or enter a bounded queue
        recheck deadline before starting queued work
    try:
        if breaker refuses:
            return chosen fallback
        make bounded call
        record outcome
    finally:
        release permit

The queue must be bounded by both length and useful age. Move admission before expensive parsing, database acquisition, or other scarce work where feasible. A queue hidden inside a connection pool still consumes your deadline budget even if the application queue looks empty.

Test what the breaker actually learns

A common breaker begins closed, opens after enough failures in a defined observation window, and later permits a limited number of half-open probes. Successful probes can close it; failed probes reopen it. Define minimum sample count, failure classification, probe timeout, and probe concurrency.

An unrealistically short probe timeout can keep a recovered but moderately slow service permanently excluded. A synchronized fleet of breakers can stampede during half-open recovery. Key breakers and pools by meaningful failure boundaries: one tenant's bad requests should not necessarily block every caller to a healthy service.

Static caps are easier to reason about. Adaptive controllers infer queuing from latency changes and adjust concurrency, but need a healthy baseline and stable feedback. A workload change can resemble congestion. Always retain hard safety limits and test controller behavior during recovery, not just steady load.

An optional response buys essential capacity

Define degradation before the incident. A product page may serve without recommendations; a charge authorization cannot be replaced with “probably approved.” Cached fallback has its own freshness policy and cold-start load. Sending fallback traffic to another overloaded dependency can enlarge the incident.

Suppose a worker can finish 800 jobs/second, new admitted jobs arrive at 600/second, and backlog is 20,000. In a simplified constant-rate model, drain time is 100 seconds. If retries add 250/second, the backlog grows. You must reduce admitted work below capacity, including retries, to recover.

This is why a system can remain unhealthy after the original trigger disappears. Retries, cold caches, and growing queues sustain the overload. Discard expired work, cap refill/retry traffic, reduce optional work, and recover gradually. Autoscaling helps only when the real bottleneck and startup time permit it.

Watch each boundary

Inspect per-route goodput, in-flight work, permit rejection, queue age, pool waits, breaker transitions, probe results, and downstream latency. Average CPU can look fine while a connection pool or one hot shard is saturated. Per-instance caps multiply with fleet size; a ten-request semaphore on a hundred instances is a thousand potential downstream requests.

Mental model: protect useful work by allocating finite resources before the failure consumes them. Breakers predict; limits enforce; bulkheads contain.

Transfer: one zone disappears

---

Source trail: 14-api-gateway-and-reverse-proxy, 22-circuit-breaker-bulkhead-and-fault-tolerance-patterns, 10-retries-timeouts-backpressure-and-load-shedding. See the source map for consolidation and reference-retained material.