The Flight Search That Stopped for One Slow Airline
The Flight Search That Stopped for One Slow Airline
This situation follows the lesson Retries, Timeouts, Backpressure & Load Shedding. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Farewise is a flight-search website. A traveller types a route and a date, and Farewise shows the fares of 12 airlines on one page. Farewise has no fares of its own. For every search, its search service calls the public API (application programming interface) of each airline, one call per airline, all 12 at the same time, and waits for all 12 answers. This pattern, one request that becomes many parallel calls, is called a fan-out. The search result is ready when the slowest of the 12 answers arrives, so the search takes as long as its slowest call.
The search service runs on 8 identical instances. Each instance makes its airline calls on a pool of 400 outbound threads, shared by all 12 airlines. A call holds one thread from the moment it is sent until the answer arrives. When every thread is busy, a new call waits in a queue, first in, first out, with no limit on the queue's length.
| Setting | Value |
|---|---|
| Searches | 240 a second across the fleet, arriving at random; 30 a second per instance |
| Airline calls | 12 per search: 360 a second per instance, 2,880 a second across the fleet |
| Outbound threads | 400 per instance, one shared pool |
| A healthy airline call, measured over one week | Average 0.29 s; P50 (median) 0.25 s; P99 0.9 s; P99.9 1.4 s. Assumption: all 12 airlines have the same healthy latency, and it follows a lognormal distribution (a distribution with a long tail of slow calls: most calls finish near the median, a few take much longer) with these values |
| The airline call, as shipped | Timeout 30 s, 1 attempt, no cap on calls in flight, no circuit breaker |
| The search result, as shipped | Waits for all 12 answers. If any airline call fails, the whole search fails. The website shows "search failed, please try again" when a result takes longer than 30 s |
| The incident | On Tuesday at 14:00:00, one airline, Skyline Air, deploys a change and every call to it takes 20 s (assumption: exactly 20 s, a constant, to keep the arithmetic simple). At 14:03:00 Skyline Air rolls the change back and its calls are healthy again. The other 11 airlines are healthy the whole time |
Synthesizing vector architecture diagram...
What happened. From 14:00:00, every search took longer than 20 s, because each search waited for its Skyline Air call. About 10 seconds after 14:00:00, every outbound thread on every instance was busy, and from then on every new call, to the 11 healthy airlines too, waited in the queue, so each search took longer than the one before it. Within a minute the website showed "search failed" for nearly every traveller, although 11 of the 12 airlines answered in 0.29 s the whole time. After 14:03:00 the site needed more than half a minute to come back.
Your task
Explain, with Little's law, why every outbound thread was busy about 10 seconds after Skyline Air slowed and why no search finished in time after that, and then design the guards for the airline calls: the timeout per airline call, the cap on calls in flight per airline, the circuit breaker (what it is keyed by, its window, and the timeout of its probes), and what the search page shows while Skyline Air is slow. A complete answer has the arithmetic for the time until the pool is full, for how fast the queue of waiting calls grows as shipped and how long it takes to drain after Skyline Air's rollback, for the timeout, for the cap and for the time until the breaker opens, and it names one failure the new design still has: the one that appears when several airlines are slow at the same time.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Flight Search That Stopped for One Slow Airline
This situation follows the lesson Retries, Timeouts, Backpressure & Load Shedding. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Farewise is a flight-search website. A traveller types a route and a date, and Farewise shows the fares of 12 airlines on one page. Farewise has no fares of its own. For every search, its search service calls the public API (application programming interface) of each airline, one call per airline, all 12 at the same time, and waits for all 12 answers. This pattern, one request that becomes many parallel calls, is called a fan-out. The search result is ready when the slowest of the 12 answers arrives, so the search takes as long as its slowest call.
The search service runs on 8 identical instances. Each instance makes its airline calls on a pool of 400 outbound threads, shared by all 12 airlines. A call holds one thread from the moment it is sent until the answer arrives. When every thread is busy, a new call waits in a queue, first in, first out, with no limit on the queue's length.
| Setting | Value |
|---|---|
| Searches | 240 a second across the fleet, arriving at random; 30 a second per instance |
| Airline calls | 12 per search: 360 a second per instance, 2,880 a second across the fleet |
| Outbound threads | 400 per instance, one shared pool |
| A healthy airline call, measured over one week | Average 0.29 s; P50 (median) 0.25 s; P99 0.9 s; P99.9 1.4 s. Assumption: all 12 airlines have the same healthy latency, and it follows a lognormal distribution (a distribution with a long tail of slow calls: most calls finish near the median, a few take much longer) with these values |
| The airline call, as shipped | Timeout 30 s, 1 attempt, no cap on calls in flight, no circuit breaker |
| The search result, as shipped | Waits for all 12 answers. If any airline call fails, the whole search fails. The website shows "search failed, please try again" when a result takes longer than 30 s |
| The incident | On Tuesday at 14:00:00, one airline, Skyline Air, deploys a change and every call to it takes 20 s (assumption: exactly 20 s, a constant, to keep the arithmetic simple). At 14:03:00 Skyline Air rolls the change back and its calls are healthy again. The other 11 airlines are healthy the whole time |
Synthesizing vector architecture diagram...
What happened. From 14:00:00, every search took longer than 20 s, because each search waited for its Skyline Air call. About 10 seconds after 14:00:00, every outbound thread on every instance was busy, and from then on every new call, to the 11 healthy airlines too, waited in the queue, so each search took longer than the one before it. Within a minute the website showed "search failed" for nearly every traveller, although 11 of the 12 airlines answered in 0.29 s the whole time. After 14:03:00 the site needed more than half a minute to come back.
Your task
Explain, with Little's law, why every outbound thread was busy about 10 seconds after Skyline Air slowed and why no search finished in time after that, and then design the guards for the airline calls: the timeout per airline call, the cap on calls in flight per airline, the circuit breaker (what it is keyed by, its window, and the timeout of its probes), and what the search page shows while Skyline Air is slow. A complete answer has the arithmetic for the time until the pool is full, for how fast the queue of waiting calls grows as shipped and how long it takes to drain after Skyline Air's rollback, for the timeout, for the cap and for the time until the breaker opens, and it names one failure the new design still has: the one that appears when several airlines are slow at the same time.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).