Design Google Maps and a Routing Engine
This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.
| Round 1: Mid-level | Round 2: Senior | Round 3: Architect | |
|---|---|---|---|
| Story | A delivery company's map and route planner for the US | A consumer maps app for North America: live traffic, ETAs, rerouting | The whole planet: continents, offline maps, walking, cycling and transit, map data from many sources |
| Level (Amazon) | SDE II (L5) | Senior SDE (L6) | Principal (L7) |
| Volume | ~24M-node road graph; 2M routes/day, 120/s peak; 54M tiles/day, 2,100/s peak | 20M phones sharing speed: 4M pings/s (6M peak); 50M routes/day plus reroutes, 10,000 route computations/s planned; 1B tiles/day, 50K/s peak | 415M-edge planet graph in 3 regions; 12M pings/s; 4B tiles/day, 200K/s peak |
| Footprint | 1 region, 3 AZs | 1 region, 3 AZs, plus the CDN | 3 regions plus the CDN; offline packs on phones |
| Targets | Tile P99 < 100 ms; route P99 < 500 ms; 99.9% | Route P99 < 25 ms server time; traffic reflected within 30 s; 99.95% | Closures everywhere within seconds; correct offline routes; 99.99% per region |
| Reading time | ~35 min | ~40 min | ~45 min |
You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.
Loop Opener: What Is a Maps Engine?
You Already Know One: a Paper Map, a Route Planner and a Traffic Report
Before phones, a road trip needed three things: a paper map to see where you are, a route planner (a person with the map and a pencil) to pick the roads, and the traffic report on the radio to tell you which roads to avoid right now. A maps app is those three things in one screen, and behind the screen they are three very different systems.
| Thing | In the road-trip picture | In our system |
|---|---|---|
| The paper map | A big picture of the world, printed once, used by everyone | Map tiles: the world cut into small square pictures (or drawing instructions), made in advance and cached everywhere |
| The route planner | Finds the cheapest way through the roads | Routing: a shortest-path search on a graph of roads, answered in milliseconds |
| The traffic report | Says how fast each road is moving right now | Live traffic: millions of phones reporting their speed, turned into the current cost of every road |
A few words we'll use all the way through:
- A graph is a set of points joined by lines. For roads, a node is an intersection (or a point where a road changes), an edge is a piece of road between two nodes, and an edge's weight is what it costs to drive it, usually seconds of travel time.
- A tile is one square of the map at one zoom level. At zoom 0 one tile shows the whole world; each zoom level splits every tile into 4.
- An ETA (estimated time of arrival) is our prediction of when the driver arrives.
Synthesizing vector architecture diagram...
Three systems with three different rhythms. Tiles change once a week, the road graph's shape changes once a week, but the road graph's weights change every few seconds. Most of the difficulty lives on the arrow from traffic to routing.
What Makes It Hard
- The graph is huge. A single country has tens of millions of road edges; the planet has hundreds of millions. A textbook search explores millions of them per query.
- Answers must be fast. A driver who misses a turn needs a new route before the next intersection, so a route has to come back in milliseconds, not seconds.
- The cost of every road keeps changing. The fastest shortest-path technique (preprocessing the graph so queries skip most of it) assumes the weights are fixed. Traffic changes them every minute.
- The input is noisy. GPS positions wander 5 to 15 meters or more in cities, enough to jump to the parallel road.
The Question the Whole Loop Answers
How do we find the fastest route in milliseconds on a graph whose weights change every minute?
The answer gets sharper every round:
- Round 1: pre-built vector tiles on a CDN, and a road graph held in memory with a preprocessing step (contraction hierarchies) that makes queries skip almost all of it.
- Round 2: a telemetry pipeline that turns 4 million GPS pings a second into road speeds, and a preprocessing technique that splits the graph's shape from its weights, so new weights apply in seconds.
- Round 3: the planet in regional pieces, offline maps, transit, map data from many sources, and closures that reach everyone within seconds.
The proximity service loop covers the other half of location systems: finding things near a point. Here we care about paths between points.
Round 1 · Mid-level · "Maps and Directions for One Country"
~35 min · SDE II (L5) · 1 region, 3 AZs · ~24M-node road graph · 2M routes/day, 120/s peak · 54M tiles/day, 2,100/s peak · tile P99 < 100 ms · route P99 < 500 ms · 99.9%
R1.1 Establish Design Scope
The interviewer says: "We're a delivery company. Our drivers and our planners need a map and turn-by-turn routes between stops. Today we pay a vendor per request and the bill is growing. Design our own." We ask before we draw.
| We ask | Interviewer answers | What it changes in the design |
|---|---|---|
| Which country, and how big is the road network? | The US. | We need a size. The standard public benchmark graph of the US road network (DIMACS) has about 23.9M nodes and 58.3M directed edges; we size from it (R1.7) and note that a graph built from today's OpenStreetMap is more detailed. |
| Driving only? | Yes: vans on public roads. | One travel mode, one cost function (travel time). |
| Do routes need live traffic? | Not yet. Use typical speeds per road type. | Weights are fixed between map releases. That's what makes heavy preprocessing possible this round. |
| Must we respect turn restrictions? | Yes. A van making an illegal left turn is a fine and a safety problem. | The graph must model turns (step 1.5). |
| How do drivers see the map? | In our app, on phones. | We send vector tiles and the phone draws them (step 1.1). |
| Offline use? | Not yet. Drivers have data plans. | Everything is served online this round. |
| Where do addresses become coordinates? | The dispatch system already stores each stop's latitude and longitude. | No geocoding service this round: requests arrive as coordinates. |
Out of scope for this round:
- Live traffic and ETAs that react to it.
- Rerouting drivers during a trip.
- Any country but the US, and any mode but driving.
R1.2 Functional Requirements, Derived Step by Step
| Phrase from the problem | Requirement |
|---|---|
| "Drivers need a map" | Serve map tiles for any zoom level and any place in the country |
| "Routes between stops" | Given an origin and a destination, return the fastest legal driving route |
| "Turn-by-turn" | Return the route as steps (turn left onto Main St) and a line to draw on the map |
| "Legal" | Never route through a banned turn or the wrong way down a one-way street |
Not yet: live traffic, rerouting, other countries, other modes, offline.
R1.3 Non-Functional Requirements: the Questions
- Tile latency. P99 under 100 ms. The map must follow the driver's thumb without blank squares.
- Route latency. P99 under 500 ms server time. A planner clicking through stops notices anything slower.
- Correctness. Every returned route is legal (turns, one-ways) and the fastest under our weights. A slightly slow answer is a bug report; an illegal route is an incident.
- Availability. 99.9%, about 43.8 minutes a month. If routing is down, drivers fall back to the last route their app downloaded, so this is a fair target for Round 1.
R1.4 The API
Get a map tile
A tile is addressed by three numbers, z/x/y:
zis the zoom level. At zoomzthe world is a grid of tiles.xis the column, counting east from longitude −180°.yis the row, counting south from the top edge of the map.
The map uses the Web Mercator projection (the one almost every web map uses). It stretches the globe onto a square, which only works up to about 85.05° north and south, so the poles are cut off. The formulas for a point at latitude and longitude :
Worked example: San Francisco City Hall, 37.7749° N, 122.4194° W, at zoom 14 ( tiles across):
- ; ; ; ;
So the tile is 14/2620/6332. Its parent at zoom 13 is 13/1310/3166 (halve both and round down), and its four children at zoom 15 are 15/5240/12664, 15/5241/12664, 15/5240/12665 and 15/5241/12665.
httpGET /v1/tiles/r2026-09-20/14/2620/6332.mvt HTTP/1.1 Host: tiles.maps.example Accept-Encoding: gzip
httpHTTP/1.1 200 OK Content-Type: application/vnd.mapbox-vector-tile Content-Encoding: gzip Cache-Control: public, max-age=31536000, immutable ETag: "9f2c61d0"
.mvtis a Mapbox Vector Tile: an open specification for a tile encoded as Protocol Buffers. It holds geometry (roads, buildings, water as points, lines and polygons in tile-local coordinates, 4,096 units across by default) plus attributes (road class, name). The phone's GPU draws it.r2026-09-20is the map release. Putting it in the path makes every tile URL immutable: a new release gets new URLs, so caches can keep a tile for a year and never serve a stale one (step 1.1). Because it's in the path, the CDN's cache key includes it for free.- The app learns the current release from a small style document (
GET /v1/style.json, cached for 60 seconds), which also tells it how to color each road class.
Compute a route
httpPOST /v1/routes HTTP/1.1 Host: api.maps.example Authorization: Bearer <access token> Content-Type: application/json { "origin": { "lat": 37.774929, "lon": -122.419416 }, "destination": { "lat": 37.338208, "lon": -121.886329 }, "mode": "DRIVE", "avoid": ["FERRIES"], "steps": true }
httpHTTP/1.1 200 OK Content-Type: application/json { "route_id": "rt_01J8Z3", "map_release": "r2026-09-20", "distance_m": 77840, "duration_s": 2940, "polyline": "i|peFj`ejV...", "legs": [ { "steps": [ { "i": 0, "maneuver": "DEPART", "instruction": "Head southeast on Market St", "distance_m": 450, "duration_s": 65 }, { "i": 1, "maneuver": "RAMP_LEFT", "instruction": "Take the ramp onto US-101 S", "distance_m": 72400, "duration_s": 2600 } ] } ] }
| Field | Why it's there |
|---|---|
polyline | The route's line as an encoded polyline: each latitude and longitude, rounded to 5 decimal places, stored as a small difference from the previous point and packed into text characters. A point costs about 4 to 10 characters instead of about 22 as JSON numbers |
duration_s | The sum of the edge weights along the route. In Round 1 these are typical speeds, not live traffic |
steps | What the driver hears: one entry per maneuver, built from the edges' names and the angle of each turn |
map_release | Which graph answered. Useful when a driver reports a wrong route after we ship a new map |
Recap
- Tiles are addressed
z/x/yin Web Mercator, and the release sits in the URL so tiles are immutable. - One route endpoint returns a line, a duration and steps.
- About 24M nodes and 58M edges; 120 routes and 2,100 tiles a second at peak (R1.7).
Let's build it, starting with the first thing a team usually tries.
R1.5 Design Evolution: From Rendering on Request to a Graph in Memory
Every step follows the same pattern: a problem, your turn to think, the answer, and what it costs us. The cost is usually the next problem.
Step 1.0: The Baseline
A team's first version: a web server that draws a map image on every request from the map data in PostgreSQL (with the PostGIS extension), and a route endpoint that runs a recursive SQL query over a road_edges table.
Synthesizing vector architecture diagram...
Everything goes to one database: every map view draws a picture, and every route walks the road table row by row.
What's good about it: one database, one server, and it works for a demo city.
What it costs us: drawing a map image takes tens to hundreds of milliseconds of CPU, and every driver panning the map asks for it again. A recursive query explores the graph one join at a time through indexes on disk. Both get slower as the country gets bigger.
Step 1.1: "Rendering Maps on Every Request Is Slow"
The problem: at 2,100 map requests a second, the render servers are at 100% CPU and a map view takes a second to fill in. Most requests are for the same city centers the drivers were looking at five minutes ago. What would you do?
Primitive: Distributed Cache Patterns and Eviction · Drill: The image service that made every origin server sweat (both questions are answered here and in R1.9: how long viewers keep seeing old content, and why a CDN beats scaling the origin)
How long until drivers stop seeing an old map? With release-versioned URLs, at most one style reload: 60 seconds after we publish a new style.json, apps ask for the new release's URLs, and the old tiles simply stop being requested. There is nothing to invalidate. If we had used unversioned URLs with a one-year cache time, drivers could see the old map for up to a year, unless we sent a CloudFront invalidation (which takes minutes to reach all edges, and costs $0.005 per path beyond the first 1,000 paths a month).
Why a CDN instead of a bigger origin? Three reasons, with numbers from R1.7:
| Scale the origin (render servers or S3 behind a load balancer) | CloudFront in front of S3 (our choice) | |
|---|---|---|
| Latency for a driver 3,000 km from the region | 60–80 ms of round trip before the first byte, every time | A few ms to the nearest edge on a hit |
| Load on our side at 2,100 tiles/s | 2,100/s | About 105/s of misses at a 95% hit ratio |
| Price of the bytes | S3 or EC2 internet egress in us-east-1 starts at $0.09/GB | CloudFront starts at $0.085/GB, and S3-to-CloudFront transfer is free |
The CDN wins on all three: it moves the bytes closer, does 95% of the work for us, and costs no more per byte.
Step 1.2: "Route Search in SQL Takes Seconds"
The problem: a route from San Francisco to San Jose takes 4 seconds with the recursive SQL query, and a cross-country route times out. The query explores the graph one join at a time, and every join is an index lookup. What would you do?
The graph in memory
| Node record field | Type | Bytes | Why |
|---|---|---|---|
lat | int32, degrees × 10⁷ | 4 | Position, to about 1 cm. A float32 would only be exact to about 1 m at these magnitudes |
lon | int32, degrees × 10⁷ | 4 | Position |
first_edge | uint32 | 4 | Index of this node's first outgoing edge; up to 4.29 billion edges |
ch_rank | uint32 | 4 | The node's importance, used from step 1.4 on |
| Total | 16 | The node's ID is its position in the array, so it isn't stored |
| Edge record field | Type | Bytes | Why |
|---|---|---|---|
target | uint32 | 4 | The node this edge leads to |
weight_ds | uint32 | 4 | Travel time in tenths of a second: the cost the search adds |
length_m | uint32 | 4 | Length in meters, summed into the route's distance |
road_class | uint8 | 1 | Motorway, primary, residential ... used for steps and styling |
flags | uint8 | 1 | Toll, ferry, private, shortcut (step 1.4) |
| padding | 2 | Keeps records 16 bytes, so exactly 4 edges share one 64-byte CPU cache line | |
| Total | 16 |
Road names, shapes (the bends between two nodes) and turn instructions live in separate parallel arrays, indexed by edge number. The search never touches them; only the final path does. Keeping the hot fields small is what keeps the search fast.
Dijkstra's algorithm on a 6-node example
Dijkstra keeps a tentative cost for every node it has reached, and a priority queue ordered by that cost. It repeatedly takes the cheapest node, marks it settled (its cost is now final), and relaxes its edges: if going through it makes a neighbor cheaper, it lowers the neighbor's cost.
Synthesizing vector architecture diagram...
Six intersections; each edge is a two-way road with its travel time in minutes. We want the fastest route from A to F.
| Step | Settle (cost) | Relaxations | Queue after the step |
|---|---|---|---|
| 1 | A (0) | B = 4, C = 2, E = 3 | C 2, E 3, B 4 |
| 2 | C (2) | B: 2 + 1 = 3 < 4, lowered to 3; D = 2 + 8 = 10 | E 3, B 3, D 10 |
| 3 | E (3) | nothing new (A is settled) | B 3, D 10 |
| 4 | B (3) | D: 3 + 5 = 8 < 10, lowered to 8 | D 8 |
| 5 | D (8) | F = 8 + 2 = 10 | F 10 |
| 6 | F (10) | target reached, stop |
The route is A → C → B → D → F, 2 + 1 + 5 + 2 = 10 minutes. Following the "came from" pointers back from F gives it. Note that the direct road A–B (4) lost to the detour through C (3), and that Dijkstra settled E even though E leads nowhere useful: it settles everything closer than the target, in every direction.
textDijkstra(source, target): cost[source] = 0; push (source, 0) onto the queue while the queue is not empty: (u, c) = pop the entry with the lowest c if c > cost[u]: skip it -- a stale entry; u was lowered later if u == target: stop for each edge (u -> v, w) in the slice edges[first_edge[u] .. first_edge[u+1]): if c + w < cost[v]: cost[v] = c + w; parent[v] = u push (v, c + w) -- push a new entry; don't edit old ones
The queue holds fixed (node, cost) entries. When a node gets cheaper we push a new entry and skip the stale one when it comes out. Editing an entry's cost in place without re-ordering the heap is a classic bug that returns wrong routes.
Step 1.3: "Dijkstra Explores the Whole Country"
The problem: in memory, the San Francisco to San Jose route now takes a few hundred milliseconds, but San Francisco to Chicago takes over 2 seconds. Dijkstra settles every node closer than the destination, in every direction, including all of Oregon for a trip that goes east. What would you do?
A* on the same 6-node graph. Our guesses (straight line to F divided by top speed, in minutes): A 9, B 7, C 8, D 2, E 11, F 0. Each is at most the true remaining cost (A's true cost is 10, E's is 13).
| Step | Settle (f = g + h) | Relaxations | Queue after the step (by f) |
|---|---|---|---|
| 1 | A (0 + 9 = 9) | B: g 4, f 11 · C: g 2, f 10 · E: g 3, f 14 | C 10, B 11, E 14 |
| 2 | C (2 + 8 = 10) | B: g 3, f 10 (lowered) · D: g 10, f 12 | B 10, D 12, E 14 |
| 3 | B (3 + 7 = 10) | D: g 8, f 10 (lowered) | D 10, E 14 |
| 4 | D (8 + 2 = 10) | F: g 10, f 10 | F 10, E 14 |
| 5 | F (10 + 0 = 10) | target reached, stop |
Same route, same 10 minutes, and E is never settled: its guess says it points the wrong way. On a real map, E is Oregon.
Step 1.4: "Long Routes Still Take Hundreds of Milliseconds"
The problem: even bidirectional A* explores hundreds of thousands of nodes for a cross-country route. Planners are complaining, and the fleet will double next year. What would you do?
CH on the 6-node graph. We contract in the order C, E, F, A, B, D (ranks 1 to 6).
| Contract | Pairs of remaining neighbors | Shortcut? |
|---|---|---|
| C (rank 1) | A–B via C = 2 + 1 = 3; no other path from A to B costs 3 or less (the direct road is 4) | Yes: shortcut A–B, weight 3, via C |
| A–D via C = 2 + 8 = 10; witness A–B–D = 4 + 5 = 9 | No | |
| B–D via C = 1 + 8 = 9; witness: the direct road B–D = 5 | No | |
| E (rank 2) | Only one neighbor (A) | No |
| F (rank 3) | Only one neighbor (D) | No |
| A (rank 4) | Remaining neighbors: only B | No |
| B, D (ranks 5, 6) | Nothing left to bypass | No |
Synthesizing vector architecture diagram...
Bottom to top by rank. The forward search from A climbs A → B → D using the shortcut; the backward search from F climbs F → D. They meet at D, the highest node on the route.
The query from A to F:
- Forward, upward from A (rank 4): A's edges to C (rank 1) and E (rank 2) go down, so they're ignored. The shortcut to B (rank 5) goes up: B = 3. From B, the edge to D (rank 6) goes up: D = 3 + 5 = 8.
- Backward, upward from F (rank 3): the edge to D (rank 6) goes up: D = 2.
- Meet at D: 8 + 2 = 10. Same answer as Dijkstra.
- Unpack the shortcut A–B into A–C–B, giving A → C → B → D → F.
The two searches touched 4 nodes (A, B, D, F) instead of 6. On a country, the difference is 280 against 9.3 million.
textCH_query(s, t): run two Dijkstra searches, forward from s and backward from t, alternating each search relaxes only edges (u -> v) with rank[v] > rank[u] whenever a node has a cost from both sides: best = min(best, fwd[v] + bwd[v]), meet = v stop a side when its cheapest queue entry is >= best; stop when both sides stopped path = fwd path s..meet + reversed bwd path meet..t replace every shortcut on the path by its two halves, recursively -- "unpacking"
Step 1.5: "The Route Turned Left Where It's Illegal"
The problem: at intersection B, a sign bans the turn from the road coming from C onto the road to D. Our route A → C → B → D → F makes exactly that turn. A driver got a ticket following it. What would you do?
Synthesizing vector architecture diagram...
In the node-based graph, the ban at B can't be expressed: B is one node. In the edge-based graph, the ban is a missing edge from "road C to B" to "road B to D".
Step 1.6: "Where Does the User's Tap Land on the Road?"
The problem: a stop's coordinates are the middle of a warehouse roof. The route has to start on a road. The first version picks the nearest graph node, which is sometimes 800 m away at the far end of a long highway segment, or across a river. What would you do?
Primitive: Geospatial Indexing: Geohash, Quadtree, S2
Round 1 Step Summary
| Step | Problem | Component | What it costs us |
|---|---|---|---|
| 1.0 | (baseline) | Render on request; route in SQL | Slow maps, slower routes |
| 1.1 | Slow map rendering | Vector tiles per release in S3, served by CloudFront | A tile build per release; a miss burst after it |
| 1.2 | Route search in SQL | Forward-star graph in RAM; Dijkstra | RAM per server; load time |
| 1.3 | Dijkstra explores everything | A*, bidirectional search, landmarks | Still too slow for long routes |
| 1.4 | Long routes too slow | Contraction hierarchies | Preprocessing; assumes fixed weights |
| 1.5 | Illegal turns | Edge-based graph | About 5 times the memory |
| 1.6 | Where the route starts | Snap to the nearest edge with a spatial index | An index per server |
R1.6 Architecture v1
Synthesizing vector architecture diagram...
Two paths that share nothing at request time: tiles come from the CDN, routes from servers that hold the whole graph in memory. The only link between them is the release number.
The pieces:
- Build pipeline: a weekly job takes the map data, validates it (below), builds tiles (zoom 0–14) and the routing graph (edge-based, with CH), and writes both to S3 under the release number. It runs on one large instance for about an hour (R1.7).
- CloudFront with Origin Shield: Origin Shield is one extra cache layer in front of S3, so that a miss in many edge locations turns into one request to S3.
- Routing servers: stateless. Each holds the whole graph. The ALB spreads requests across them. Any server can answer any route.
- No database on the request path. Both tiles and routes are served from immutable, versioned files.
Validation before a release goes out (a bad map is worse than an old map):
| Check | Fails the release when |
|---|---|
| Size | Node or edge count differs from last week by more than 2% |
| Connectivity | The largest connected component shrinks (a broken bridge in the data disconnects a city) |
| Route regression | Of 10,000 fixed origin-destination pairs, more than 0.5% change duration by more than 20%, or any becomes unreachable |
| Turn restrictions | The count drops by more than 1% |
| Tiles | A sample of tiles fails to decode, or any tile exceeds 500 KB |
Trace 1: a tile request
Synthesizing vector architecture diagram...
Most requests end at the edge. The shield turns many edges' misses into one read from S3.
Trace 2: a route request
Synthesizing vector architecture diagram...
Everything happens in one process, in memory. The server time is a few milliseconds against a 500 ms target.
R1.7 Numbers
Traffic
| Quantity | Math | Value |
|---|---|---|
| Routes per day | 20,000 drivers × 100 stops | 2,000,000 |
| Average during the working day | 2,000,000 ÷ 36,000 s (10 hours) | ≈ 55.6/s |
| Peak | assume 2× the working-day average (morning planning) = 111/s | plan for 120/s |
| Tiles per day, drivers | 20,000 × 600 min × 2 new tiles/min (the phone caches its area, so it fetches only new ground) | 24M |
| Tiles per day, customers tracking a delivery | 2M deliveries × 1.5 views × 10 tiles | 30M |
| Tiles per day | 24M + 30M | 54M |
| Tile rate | 54M ÷ 36,000 s = 1,500/s; peak assume 1.4× | 2,100/s peak |
| Tile misses at the edge | 2,100 × (1 − 0.95), assuming a 95% hit ratio (a modest fleet, lots of rural tiles) | ≈ 105/s peak |
Tiles and storage
| Quantity | Math | Value |
|---|---|---|
| Tile width at zoom 14 | 40,075 km ÷ 16,384 at the equator; × cos 38° ≈ 0.788 | ≈ 2.45 km; ≈ 1.93 km at 38° N |
| Zoom-14 tiles over the lower 48 states | box from 24.5° to 49.4° N, 125° to 66.9° W: ≈ 2,644 columns × 1,443 rows | ≈ 3.81M |
| All zoom levels 0–14 | each level has a quarter of the tiles of the next: × (1 + 1/4 + 1/16 + ...) ≈ × 4/3 | ≈ 5.1M tiles |
| Storage per release | rough: most tiles are nearly empty countryside; city tiles are tens to hundreds of KB. A published full-planet build of zoom 0–14 is about 99 GB, so a US-only build of a few to about 10 GB is a fair planning figure | ≈ 10 GB |
Graph memory
| Part | Math | Size |
|---|---|---|
| Node array (node-based) | 23.9M × 16 B | 0.38 GB |
| Edge array (node-based) | 58.3M × 16 B | 0.93 GB |
| Plain graph | 1.3 GB | |
| Edge-based graph + CH shortcuts + unpacking data (replaces the plain graph above) | scaled from the published Western Europe figure: 3.14 GiB × (58.3M ÷ 42.5M edges) | ≈ 4.3 GiB (4.6 GB) |
| Shapes, names and turn instructions | estimate | ≈ 2 GB |
| Snapping index | estimate | ≈ 0.5 GB |
| Per routing server (edge-based graph + shapes + index: 4.6 + 2 + 0.5) | ≈ 7 GB |
An r7g.large has 16 GiB, so the graph fits with room for the process and the operating system. It's RAM, not CPU cache: a CPU's L3 cache holds tens of megabytes, a thousand times less. What makes the search fast is that each step touches few, nearby memory locations.
Routing CPU. We plan on about 5 ms of CPU per route request (snap, CH query, unpacking, shapes, steps, JSON), an estimate to confirm by load test. At 120/s that's 0.6 vCPU busy. Three r7g.large servers (2 vCPUs each) run at 10% busy, and after losing an AZ the remaining two run at 15%. We have three for availability, not for load.
Preprocessing. The published edge-based CH on Western Europe took about 23 minutes on 12 cores. Our graph is about 1.4 times larger, so we plan on about 30–40 minutes on one 16-vCPU instance, plus about an hour for the tiles, once a week.
Monthly cost (us-east-1 on-demand list prices, 730 hours, 30.4 days)
| Item | Math | Monthly |
|---|---|---|
Routing servers, 3 × r7g.large | 3 × $0.1071/h × 730 h | ≈ $235 |
| ALB | $0.0225/h × 730 h + a few capacity units | ≈ $25 |
| CloudFront data out | 54M × 30.4 = 1.64B tiles × 25 KB ≈ 41 TB: 10 TB × $0.085 + 31 TB × $0.080 per GB | ≈ $3,333 |
| CloudFront HTTPS requests | 1.64B ÷ 10,000 × $0.0100 | ≈ $1,642 |
| Origin Shield requests | 5% of 1.64B = 82M ÷ 10,000 × $0.0075 | ≈ $62 |
| S3 GETs and storage | 82M × $0.0004 per 1,000 + ~10 GB per release | ≈ $34 |
| Weekly build, one 16-vCPU instance for ~2 h | ≈ $20 | |
| CloudWatch | ≈ $50 | |
| Total | ≈ $5,400/month |
Almost all of it is the CDN, and a third of the CDN bill is the per-request charge, not the bytes. The routing servers cost $235. (CloudFront's always-free allowance of 1 TB and 10 million requests a month would take about $95 off; we leave it out.)
R1.8 Trade-Offs
Search techniques (published Western Europe figures, single core, 18.0M nodes; illustrative for our graph, not promises)
| Dijkstra | Bidirectional Dijkstra | A* / ALT (landmarks) | Contraction hierarchies (our choice) | |
|---|---|---|---|---|
| Nodes settled per long query | ~9.3M | ~4.9M | Far fewer; depends on the landmarks | ~280 |
| Query time | ~2.2 s | ~1.2 s | Tens of ms or more (rough) | ~0.1 ms (0.2–0.3 ms with turns) |
| Preprocessing | None | None | Minutes (distances to 16–32 landmarks) | ~5 min; ~23 min with turns on 12 cores |
| Weight changes | Free | Free | Free while weights don't drop below the landmark distances | Rebuild (Round 2's problem) |
| Right when | Tiny graphs, one-off analysis | Small graphs | Weights change and memory is tight | Fixed weights, fast queries |
Vector vs raster tiles
| Raster (PNG images) | Vector (MVT, our choice) | |
|---|---|---|
| Who draws | Our servers, once per tile per style | The phone's GPU, every frame |
| Zoom levels to build | Every level up to the deepest (each deeper level has 4 times the tiles) | Up to 14; deeper levels are overzoomed |
| Changing a color or adding night mode | Rebuild and re-cache every tile | Change the style document |
| Rotation, tilt, smooth zoom | Blurry or impossible | Natural |
| Cost | Heavier files for most map content | Needs a capable client renderer |
Graph in memory vs a graph database
| Graph database | Forward-star arrays in RAM (our choice) | |
|---|---|---|
| Layout | Records with pointers and headers | Two flat arrays, 16 bytes per node and per edge |
| One edge step | A pointer chase, often a cache miss | The next 16 bytes, often in the same cache line |
| Preprocessing like CH | Not supported | Built by our pipeline |
| Flexible queries | Excellent | Only what we coded |
| Updating one road | A write | A new release (fine: the map changes weekly) |
R1.9 Failure Modes
| Failure | What you'd see | How the design responds |
|---|---|---|
| A routing server restarts | One of three servers out for a few minutes | The ALB stops sending it traffic when its health check fails. The server downloads about 7 GB from S3 (roughly 10 seconds at 1 GB/s with parallel ranged reads; a rough figure), maps the arrays into memory and warms them with a few thousand canned queries before reporting healthy. It only passes the health check once warm, so no user gets a cold, slow answer. Two servers carry the load meanwhile. |
| A bad map release (a bridge missing from the data) | Route regression checks fail in the pipeline | The release never reaches S3's "current" pointer. If a problem slips through, we roll back by pointing the routing fleet and style.json back at the previous release number; the old files are still in S3. Routing servers switch releases blue/green: new instances load the new graph, pass canary queries, then take traffic. |
| A miss storm after a tile release | Hit ratio drops, S3 GETs jump for an hour | Every tile URL is new, so every edge misses once per tile. CloudFront collapses simultaneous requests for the same object, Origin Shield turns misses from many edges into one S3 read, and S3 serves thousands of reads a second per key prefix. We also roll the new release out gradually: style.json sends 10% of apps to the new release, then 50%, then all, over a few hours. |
| Tile P99 above 100 ms | CloudWatch shows slow requests | With a 5% miss rate, the slowest 1% of requests are all misses, so the P99 is the miss path. Origin Shield in the same region as the bucket keeps a miss to one extra hop; if it's still slow, we look for tiles that are too big (the release check rejects any over 500 KB). |
| An AZ fails | One routing server gone | The other two run at 15% CPU. CloudFront and S3 are unaffected. |
R1.10 Pillar Check
| Pillar | What Round 1 covers |
|---|---|
| Reliability | Stateless routing servers in 3 AZs behind an ALB; releases validated and switched blue/green, with instant rollback to the previous release number REL 8 · REL 10 · REL 11 |
| Performance Efficiency | The CDN serves 95% of tiles from the edge; the graph is laid out for the CPU (16-byte records, Hilbert order) and preprocessed so a query settles hundreds of nodes, not millions PERF 1 · PERF 3 |
| Security | Tiles are public; the route API requires a token; routing servers can only read the graph bucket; TLS from app to CloudFront and ALB SEC 3 · SEC 9 |
| Cost Optimization | About $5,400 a month, almost all CDN; Graviton memory-optimized instances sized to the graph, not oversized for CPU COST 5 · COST 6 |
| Operational Excellence | Light this round: release checks with clear pass and fail rules; alarms on route latency, 5xx rate and tile hit ratio OPS 8 |
| Sustainability | Skipped this round: three small instances and a CDN. |
R1.11 Round 1 Rubric and Follow-Ups
What a strong mid-level (L5) answer shows
- Separates the map (static, cacheable) from routing (a search) from the start.
- Explains
z/x/ytiles and why vector tiles beat images, and makes tile URLs immutable with a release number. - Puts the graph in memory as flat arrays and explains why, with a size estimate.
- Runs Dijkstra correctly on a small example, and explains how A*, bidirectional search and CH each cut the work.
- Handles turn restrictions in the graph, not after the search, and snaps to edges, not nodes.
- Says which assumption CH depends on (fixed weights).
Follow-up questions
-
"Our planners want a 100 × 100 table of travel times between all stops of one van, to optimize the stop order. Do we run 10,000 route queries?" Answer: no. CH has a many-to-many method: run one upward backward search from each of the 100 targets and store, at every node reached, "target j is this far". Then run one upward forward search from each of the 100 sources and, at every node reached, combine with the stored entries. That's 200 small searches instead of 10,000, and only distances, no shapes. Only the final chosen route needs its geometry.
-
"Why zoom 14 and not zoom 18 for the tiles?" Answer: each zoom level has 4 times the tiles of the one before, so building to 18 means times the zoom-14 count: about a billion tiles for the US instead of about 4 million. Vector tiles carry exact shapes, so the phone can draw zoom 18 from zoom-14 data. We'd only go deeper if zoom-14 tiles got too big in dense cities, and we can drop small details from low zooms to keep sizes in check.
-
"Could the routing server read the graph from disk instead of RAM?" Answer: memory-mapped from a local NVMe disk, yes, once the operating system has cached it; that's a common way to make startup instant. Reading it from network storage per query, no: each search step would wait on I/O. The working set is the whole graph because routes can go anywhere, so it has to live in RAM either way.
Interview gotchas from this round's wrong answers
| Gotcha | Why it's wrong |
|---|---|
| "Shortest path in SQL" | A recursive query walks the graph one indexed join at a time; seconds per route, and one busy planner can saturate the database. |
| "The graph fits in the L3 cache" | L3 holds tens of megabytes; the graph is gigabytes. It fits in RAM, and the layout keeps each step's memory reads few and nearby. |
| "Cache all routes" | Trillions of pairs, and stops rarely repeat exactly. |
| "Snap to the nearest node" | Nodes are sparse on long roads and may be unreachable (a motorway overhead). Snap to edges. |
| "Raster tiles, render per request" | Redraws the same pictures forever; styling changes mean re-rendering everything. |
| "CH handles traffic by updating weights" | CH left out shortcuts based on this release's weights; change them and answers can be wrong. |
Round 2 · Senior · "Live Traffic From 20M Drivers, and Fast ETAs"
~40 min · Senior SDE (L6) · 1 region (us-east-1), 3 AZs, plus the CDN · North America · 4M pings/s (6M peak) · 50M routes/day, 10,000 route computations/s planned · 1B tiles/day, 50K/s peak · route P99 < 25 ms server time · traffic reflected within 30 s · 99.95%
R2.0 Where We Left Off
This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.
Round 1 in 60 seconds. "A delivery company needed maps and routes for the US: a graph of about 24M nodes and 58M directed edges, 120 routes and 2,100 tiles a second at peak. Tiles are built once per weekly release as vector tiles up to zoom 14, stored in S3 under the release number and served by CloudFront, so URLs are immutable and nothing is ever invalidated; the phone draws them and overzooms past 14. Routing servers hold the whole graph in RAM as flat forward-star arrays, 16 bytes per node and per edge, renumbered along a Hilbert curve. Dijkstra settles millions of nodes, A* and bidirectional search help a little, and contraction hierarchies, which add shortcuts so a query only climbs toward important roads from both ends, cut a query to a few hundred nodes and well under a millisecond. Turn restrictions live in an edge-based graph, about 5 times the memory, and routes start on the nearest edge from a spatial index. Each server needs about 7 GB; three
r7g.largeservers and the CDN cost about $5,400 a month. The open cost: CH assumes fixed weights, and we have no traffic."
Architecture v1, compact
Synthesizing vector architecture diagram...
Round 1 in one picture: static tiles on a CDN, a static graph in memory, both rebuilt weekly.
Round 1 step summary
| Step | Problem | Component | What it costs us |
|---|---|---|---|
| 1.1 | Slow map rendering | Vector tiles per release on CloudFront | A build per release |
| 1.2 | Routes in SQL | Forward-star graph in RAM; Dijkstra | RAM per server |
| 1.3 | Dijkstra explores everything | A*, bidirectional, landmarks | Still slow for long routes |
| 1.4 | Long routes | Contraction hierarchies | Assumes fixed weights |
| 1.5 | Illegal turns | Edge-based graph | About 5 times the memory |
| 1.6 | Where routes start | Snap to the nearest edge | A spatial index per server |
Open costs: no live traffic, no ETA beyond typical speeds, nobody gets rerouted, and CH can't take new weights without a rebuild.
R2.1 The Scope Raise
Interviewer: "The routing engine worked, so now it's our consumer maps app for North America. At a busy hour about 20 million phones have the app open in a car and share their location every 5 seconds; the busiest hours of the year reach 30 million. Routes and ETAs have to reflect traffic within about 30 seconds of it happening. When a jam or a closure appears, drivers already on their way must be rerouted. GPS is noisy in cities. We compute 50 million routes a day, with sharp rush hours, and a route has to come back in under 25 milliseconds."
We ask back, and say what each answer changes.
| We ask | Interviewer answers | What it changes in the design |
|---|---|---|
| Are all 20M phones navigating? | No. About 2 million are in active navigation at the evening peak; the rest have the app open or share speed in the background. | Two different loads: 4M pings/s of telemetry from everyone (6M at the yearly peak), and about 2.1M navigation sessions that can be rerouted (R2.6). |
| What's in a ping, and can we trust it? | Position, speed, heading, accuracy and a timestamp. Positions are often off by 5 to 15 m, much more between tall buildings. | We can't just snap each ping to the nearest road (step 2.2). |
| What does "reflect traffic within 30 s" mean? | From a car slowing down to routes and ETAs using the new speed, 30 seconds at most. | A timing budget across every stage (step 2.3). |
| What is "under 25 ms"? | P99 server time for routes up to about 1,000 km. | Rules out ALT-style searches without preprocessing; keeps a preprocessed graph. |
| How spiky are routes? | About 10% of a day's trips start in the busiest hour. Rerouting adds to that. | Plan for 10,000 route computations/s (R2.6). |
| Do we still build our own tiles? | Yes, now about 1 billion tile requests a day, 50,000 a second at peak, plus a traffic color layer. | Same CDN design, bigger bill, and a short-lived traffic layer. |
| Can location data be kept? | Only as long as needed, and never tied to a person. | Pseudonymous device IDs, raw pings not archived, observations kept 30 days (R2.6). |
Scope change
| Round 1 | Round 2 | |
|---|---|---|
| Region | US, driving | North America, driving |
| Graph | ~24M nodes, 58M edges, fixed weights | ~70M nodes, 172M directed edges, weights change every few seconds |
| Routes | 120/s peak | 10,000 computations/s planned (new routes + reroutes) |
| Telemetry | none | 4M pings/s, 6M at the yearly peak |
| Tiles | 2,100/s peak | 50,000/s peak, plus a traffic layer |
| Latency | route P99 < 500 ms | route P99 < 25 ms; traffic within 30 s |
| Availability | 99.9% | 99.95% (about 21.9 minutes a month) |
It's tempting to promise "five nines" (5.26 minutes of downtime a year, about 26 seconds a month). We don't. One bad weekly release or one regional event would spend a year's budget, and the phone already carries its route through short outages. 99.95% for routing, with tiles on a CDN, is the honest target.
R2.2 What Breaks in the Round 1 Design
| Round 1 choice | What breaks at the new scope |
|---|---|
| CH built once per release | Weights now change every few seconds. CH's left-out shortcuts were chosen for the old weights, so updated weights can give wrong answers, and a rebuild takes minutes to hours. |
| No telemetry path | 4M pings a second have nowhere to go, and nothing turns them into road speeds. |
| Snap to the nearest road | Noisy pings jump to the parallel road or the overpass; speeds get assigned to the wrong roads. |
| Everyone gets the fastest route | Send 3,000 cars down the same quiet side street and it jams worse than the motorway did. |
| Routes are fire-and-forget | A bridge closes; 15,000 drivers heading for it never hear about it. |
| Typical speeds for ETAs | Wrong every rush hour. |
R2.3 New Requirements and API Additions
The telemetry ping. Phones keep one long-lived TLS connection to our ingest fleet and send a small binary frame every 5 seconds. JSON over HTTPS would cost several hundred bytes per ping in headers alone, times 4 million a second.
| Field | Bytes | Encoding | Why |
|---|---|---|---|
device_pseudonym | 8 | random 64-bit ID, rotated every 24 hours | Links one device's pings for a day, so we can map-match its path, without identifying the person |
seq | 4 | uint32, +1 per ping | Detects gaps and duplicates |
ts | 4 | uint32, seconds since 1970 | When the fix was taken |
ts_ms | 2 | uint16, 0–999 | Sub-second part |
lat, lon | 4 + 4 | int32, degrees × 10⁷ | About 1 cm resolution |
speed | 2 | uint16, cm/s | GPS speed (from the Doppler shift, often better than position) |
heading | 2 | uint16, 0.01° | Direction of travel |
accuracy | 1 | uint8, meters | The phone's own error estimate |
flags | 1 | bits | Navigating or not, charging, mock location |
| Total | 32 | About 100 bytes on the wire with TLS and TCP/IP overhead |
The server answers every ping with a 10-byte ACK frame: the acked seq, a notice flag, and a route_version when a reroute is waiting (step 2.6).
Start navigation
httpPOST /v1/navigation/sessions HTTP/1.1 Host: api.maps.example Authorization: Bearer <access token> Content-Type: application/json { "route_id": "rt_01J8Z3", "device_pseudonym": "7f3a9c01e2d4b688" }
httpHTTP/1.1 201 Created Content-Type: application/json { "session_id": "ns_01J8Z4Q", "route_version": 1, "expires_at": 1790571600 }
ETA in the route response. New fields:
json{ "duration_s": 3120, "typical_duration_s": 2700, "delay_s": 420, "traffic_epoch": 179057120, "segments_congestion": [ { "from_i": 212, "to_i": 380, "level": "HEAVY" } ] }
duration_s is our ETA with traffic; typical_duration_s is the same route at this time of week without live data; traffic_epoch names the weight snapshot that produced it (step 2.3).
The reroute notice. When the server has a better route for a session, the next ping's ACK carries notice = REROUTE, route_version = 4. The app then fetches it:
httpGET /v1/navigation/sessions/ns_01J8Z4Q/route?version=4 HTTP/1.1 Host: api.maps.example Authorization: Bearer <access token>
The response is the normal route body plus reason (CLOSURE, FASTER_ROUTE) and the IDs of the closed edges, so the app can check whether the problem is still ahead of it.
Recap
- 32-byte binary pings over one long-lived connection per phone; the ACK is our way back to the phone.
- Navigation sessions with a
route_versionthat only goes up. - Routes carry an ETA with traffic, a typical ETA, and the weight snapshot they used.
R2.4 Design Evolution: From Fixed Weights to Live Traffic
Step 2.1: "4M Pings a Second: Where Do They Go?"
The problem: 4 million pings arrive every second, 6 million at the yearly peak. We need them turned into speeds per road, and we'd like to learn from them later. What would you do?
Synthesizing vector architecture diagram...
Keyed by device until the matcher, because matching needs one device's pings in order; keyed by road segment after it, because speed needs all vehicles on one road together.
Primitive: Message Queues vs Event Streams
Step 2.2: "Pings Wander Onto the Parallel Road"
The problem: a car drives down a motorway with a frontage road 20 m to the side. Its pings, 8 to 12 m off, snap to the frontage road half the time. The frontage road "shows" 100 km/h traffic, and the motorway loses half its data. What would you do?
A worked example. A car on motorway H, with frontage road F alongside. Three pings, 5 s apart, about 139 m apart (100 km/h). We use log-probabilities (so we add instead of multiply) and drop constants that are the same for every candidate; m, m.
| Ping | Distance to H | Distance to F | Emission score H: | Emission score F |
|---|---|---|---|---|
| 1 | 12 m | 6 m | −0.72 | −0.18 |
| 2 | 5 m | 9 m | −0.125 | −0.405 |
| 3 | 8 m | 14 m | −0.32 | −0.98 |
Transitions, : staying on H differs by 2 m (−0.1); staying on F by 3 m (−0.15); switching needs the 440 m ramp, a difference of about 300 m (−15) or 350 m (−17.5).
| Ping | Best path ending on H | Best path ending on F |
|---|---|---|
| 1 | −0.72 | −0.18 |
| 2 | from H: −0.72 − 0.1 − 0.125 = −0.945 | from F: −0.18 − 0.15 − 0.405 = −0.735 |
| 3 | from H: −0.945 − 0.1 − 0.32 = −1.365 | from F: −0.735 − 0.15 − 0.98 = −1.865 |
- Nearest road says F, H, H: a jump across a barrier.
- Viterbi after ping 3 says H, H, H: the best final score is H, and its path came from H at every step.
- Deciding ping 1 after only one more ping would have said F: at ping 2 the best path still ends on F. That's why dense areas wait for two.
Synthesizing vector architecture diagram...
The thick path is the winner. Switching roads between pings costs so much that the sequence stays on one road, even though ping 1 alone looked closer to F.
Step 2.3: "Traffic Changes Weights, but CH Was Built for Fixed Weights"
The problem: Flink now produces fresh speeds for about 5 million road segments every few seconds. Our routing servers hold a contraction hierarchy built last week. Rebuilding it for North America takes far longer than 30 seconds, and just overwriting edge weights in the old hierarchy can give wrong routes. What would you do?
Synthesizing vector architecture diagram...
A query uses real roads only inside the origin's and destination's cells (A and C) and jumps across cell B using its precomputed boundary-to-boundary cost. When traffic changes a road inside B, only B's table and its parent's are recomputed.
Keeping the weights in order. A server that restarts, or that finds an epoch missing (expired or never written), must not apply later deltas on top of a gap: a segment could keep a stale jam forever. So:
- On start: load the latest S3 snapshot (epoch M), then apply deltas M+1, M+2 ... up to
weights:latest, in order, and only then take traffic. - In normal running: apply epoch N+1 only after N. If N+1 is missing, reload the snapshot and catch up again, rather than skip.
- Replays are harmless: after a Flink restart the same window can be written again under the same epoch number; a server that already passed that epoch ignores it, and because deltas carry absolute speeds, applying one twice gives the same result.
Closures are not weights. A closed bridge can't be a value in a delta: the next delta for that segment (or the blend toward the historical profile when no cars report) would reopen it, snapshots don't contain it, and the historical fallback would drop it. So each routing server keeps a separate closure overlay, loaded from the closure store (a DynamoDB table road-closures; global in Round 3). Whenever the server computes a segment's effective weight, after a delta, a snapshot reload or a switch to historical weights, the rule is the same: infinity if a closure on that segment is active by the server's own clock, otherwise the delta, snapshot or historical weight. On start, and on any snapshot reload, the server re-reads all active closures before it takes traffic. A new closure is pushed to the overlay at once (and servers re-read the table every 10 seconds as a backstop), then customizes just the cells that contain it.
The 30-second budget (worst case, dependent steps add)
| Stage | Worst case | Why |
|---|---|---|
| Ingest batching into Kinesis | 0.5 s | The ingest fleet flushes batches every 500 ms |
| Raw stream to matcher | 1 s | Consumers poll each shard about once a second |
| Map-matching decision | 5 s (10 s in dense areas) | Wait for 1 more ping (2 in dense areas) |
| Observation stream to Flink | 1 s | Polling again |
| Window firing | 10 s | A 60-second window that fires every 10 seconds |
| Delta written to Valkey | 0.5 s | |
| Routing server poll | 1 s | Polls once a second |
| Customization of changed cells | ~4 s | Estimate: a full North America customization is roughly 4 s on 12 cores, scaled from Europe's ~1 s by the edge count; partial updates on 4 cores should land near that. To load-test. |
| Total | 23 s (28 s in dense areas) | Under 30 s. A typical case is closer to 15 s. |
"Reflected" means the first window that contains the slow cars. A 60-second window blends old and new, so a sudden stop pulls the average down a little in the first firing and fully within a minute. We accept that: reacting to one car braking would be worse.
Step 2.4: "The ETA Is Wrong at 5 pm"
The problem: a driver leaves at 4:45 pm for a 46-minute drive. Live speeds describe the roads now, but the driver reaches the last part of the trip at about 5:20 pm, when rush hour is at its worst. The old ETA used distance divided by speed limit and said 33 minutes. What would you do?
| Part of the route | Reached after | Live | Historical at that time | Blend | |
|---|---|---|---|---|---|
| 10 km motorway | 0 min | 1.000 | 8 min | 9 min | 8.0 min |
| 20 km arterial | 8 min | 30 min | 25 min | 0.587 × 30 + 0.413 × 25 = 27.9 min | |
| 5 km city streets | 8 + 27.9 = 35.9 min | 12 min | 10 min | 0.091 × 12 + 0.909 × 10 = 10.2 min | |
| Total | 46.1 min |
Distance ÷ speed limit (105, 60 and 40 km/h) gives 5.7 + 20.0 + 7.5 = 33.2 min, 13 minutes too optimistic.
Step 2.5: "Everyone Got Rerouted Onto the Same Side Street"
The problem: a crash on the motorway adds 30 minutes. A residential street parallel to it saves 4 minutes for anyone who takes it. It carries about 400 cars an hour before it jams. Within a minute we send it 3,500 cars an hour's worth of drivers, and it jams worse than the motorway. Then live speeds say the side street is slow, we send everyone back, and the two roads take turns jamming. What would you do?
Synthesizing vector architecture diagram...
Almost flat up to capacity, then very steep: a street at twice its capacity takes 3.4 times as long, at three times 13 times as long.
Keeping planned volume correct in both directions. The counts must go down as well as up, or streets look busier every hour until everything is penalized. We don't keep them as counters that routing servers increment (a retried increment counts twice, and a missed decrement never comes back). The session index (step 2.6) derives them: each session contributes +1 to each segment on its current route version with an expected passage time in the next hour. A new route version replaces the old one's contribution; an ended or expired session removes its own; a passed segment drops out when its time goes by. The index applies each session's updates in version order and ignores older versions, and it recomputes all counts from scratch every 5 minutes to catch any drift.
Step 2.6: "A Bridge Closed; 15,000 Drivers Are Headed to It"
The problem: a ship hits a bridge and it closes in both directions. Operators mark it closed. About 15,000 navigating drivers have it on their route, some 2 minutes away, some 40. What would you do?
Synthesizing vector architecture diagram...
The nearest drivers are handled in the first second and hear about it on their next ping. Nothing is sent to all 2.1M sessions.
Lost notices. Valkey's pub/sub is fire-and-forget: an ingest server that was reconnecting misses messages. So each notice is also added to a set per 10-second bucket, notices:{bucket}, kept 10 minutes; every ingest server re-reads the last bucket every 10 seconds. A phone that hasn't pinged for 30 seconds while it has a pending closure notice also gets a visible push notification ("Road closed ahead, new route available"), which is appropriate for a user-facing alert.
Primitive: Circuit Breaker, Bulkhead and Fault Tolerance Patterns (the scheduler's rate limit is a bulkhead that protects the routing fleet from its own reroute storm)
Round 2 Step Summary
| Step | Problem | Component | What it costs us |
|---|---|---|---|
| 2.1 | 4M pings/s | NLB → ingest fleet → Kinesis → matchers → Kinesis → Flink | A streaming pipeline |
| 2.2 | Noisy GPS | HMM map matching with Viterbi, 1–2 pings of delay | CPU per ping; 5–10 s |
| 2.3 | CH can't take new weights | MLD: cells with precomputed boundary costs, customized every 10 s from ordered deltas | Slower queries; CPU for customization |
| 2.4 | Wrong ETAs | Historical profiles blended with live speeds by arrival time; error measured | Profiles; a model to keep honest |
| 2.5 | Herding onto side streets | Planned volume, BPR penalties, split assignment, hysteresis | Slightly slower routes for some |
| 2.6 | Closures and active drivers | Session store + in-memory index; nearest-first waves; notices in the ping ACK | An index, a scheduler, a notice path |
R2.5 Architecture v2
Synthesizing vector architecture diagram...
Three flows meet at the routing fleet: the release graph from S3 (not drawn), live weights from Valkey every 10 seconds, and requests from phones. Tiles never touch it.
The pieces added since Round 1:
- Ingest fleet (45 ×
c7g.4xlargebehind an NLB with TCP pass-through): holds about 450K TLS connections each, validates frames, batches to Kinesis, and writes ACKs with notices. - Map matchers (21 ×
c7g.4xlarge): Kinesis Client Library consumers, each holding the North America graph with shapes and a spatial index (about 8 GB), keeping each device's recent pings and Viterbi state in memory. - Managed Flink (about 120 processing units): per segment, a 60-second window firing every 10 seconds; one vote per vehicle per segment per window, which both stops one idling car from outvoting ten moving ones and makes duplicate observations (from a matcher restart) harmless. It computes the harmonic mean of the vehicles' speeds (the right average for travel time over a fixed distance), requires at least 3 vehicles, and otherwise blends toward the historical profile. It validates, then writes deltas and snapshots.
- Routing fleet (21 ×
c7g.4xlarge): the MLD graph with turn tables, historical profile IDs, and two weight sets in memory (the one serving queries and the one being customized). - Session API, DynamoDB, its change stream (Kinesis Data Streams for DynamoDB) and the session index, plus the reroute scheduler on ECS Fargate. The closure store and each server's closure overlay sit beside the weights.
- Traffic tiles:
GET /v1/traffic/{z}/{x}/{y}.mvtwithCache-Control: max-age=30. The base tiles already carry each road's segment ID, so a traffic tile only carries "segment → congestion level" pairs, about 5 KB. A small Fargate service builds them from the latest weights.
Data model: the session item
| Attribute | Type | Example | Notes |
|---|---|---|---|
session_id (partition key) | S | ns_01J8Z4Q | |
device_pseudonym | S | 7f3a9c01e2d4b688 | Links notices to the ping connection |
route_version | N | 3 | Only ever increases; every route write is conditional on it |
route_edges | B | compressed edge IDs | About 170 edges for a 25-minute trip |
edge_eta_offsets | B | compressed seconds from start | Expected time at each edge, for the index |
polyline | S | encoded polyline | For re-sending the route |
last_closure_id | S | cl_77120 | The last closure this session was rerouted for |
expires_at | N | 1790571600 | Epoch seconds. The app and the index treat a session as ended once this time passes, by their own clocks. It's also the DynamoDB TTL attribute, but only for cleanup: TTL deletes items eventually (typically within a few days), never at a deadline. |
A reroute write:
textUpdateItem ActiveNavigation Key: session_id = "ns_01J8Z4Q" Update: SET route_version = :v4, route_edges = :edges, edge_eta_offsets = :etas, polyline = :pl, last_closure_id = :cl Condition: route_version = :v3 AND expires_at > :now_from_scheduler_clock
The condition compares with a time the scheduler supplies from its own clock; DynamoDB itself has no notion of "now" in conditions.
Trace 1: a ping becoming a speed
Synthesizing vector architecture diagram...
A slowdown at 12:00:00 is in routes by about 12:00:15 here, inside the 23-second worst case.
Trace 2: a reroute on a closure. The sequence in step 2.6: the closure reaches every routing server's closure overlay within a second (it doesn't wait for the 10-second window), the index returns 15,000 sessions in well under a second, the scheduler works through them nearest-first at 1,000 a second, and each phone learns on its next ping. The last session is rerouted after about 15 s and hears within about 21 s.
R2.6 Numbers and Cost
Telemetry
| Quantity | Math | Value |
|---|---|---|
| Pings, busy hour | 20M phones ÷ 5 s | 4M/s |
| Pings, yearly peak | 30M ÷ 5 s | 6M/s |
| Wire bandwidth in | 4M × ~100 B = 400 MB/s | 3.2 Gbps (4.8 Gbps at 6M/s) |
| Raw stream bytes at peak | 6M × 32 B | 192 MB/s → 192 shards; 240 provisioned |
| Observation stream at peak | 6M × 24 B | 144 MB/s → 144 shards; 180 provisioned |
| Matcher CPU | 6M × ~30 µs (estimate) | 180 vCPUs busy |
| Matcher fleet | 7 per AZ × 16 vCPUs = 336 vCPUs: 54% busy at peak; 224 vCPUs after losing an AZ: 80% | 21 × c7g.4xlarge |
| Ingest fleet | assume 200K pings/s per server: 6M ÷ 200K = 30 needed, and still 30 after losing an AZ | 15 per AZ, 45 servers |
| Flink | 6M observations/s ÷ ~50K per processing unit (estimate) | ~120 units |
The Kinesis quota check: 240 + 180 + 8 (session changes) = 428 shards, far under the default quota of 20,000 per account in us-east-1; we still check it in Service Quotas.
Navigation
| Quantity | Math | Value |
|---|---|---|
| New routes, average | 50M ÷ 86,400 s | 578.7/s |
| New routes, busiest hour | 10% of 50M = 5M in 3,600 s | ≈ 1,389/s |
| Sessions at the busiest hour | 5M trips/h × 25 min average trip ÷ 60 | ≈ 2.1M |
| Reroutes | assume 1 per session per 10 min (off-route plus faster routes): 2.08M ÷ 600 s | ≈ 3,472/s |
| Route computations at peak | 1,389 + 3,472 | ≈ 4,861/s → plan 10,000/s (closure waves, retries, an AZ lost) |
| Route computations per day | 50M new + 50M × 25 min ÷ 10 min = 125M reroutes | 175M/day |
Tiles
| Quantity | Math | Value |
|---|---|---|
| Base tiles, average | 1B ÷ 86,400 s | 11,574/s |
| Base tiles, peak | given | 50,000/s |
| Origin requests at peak | 50,000 × (1 − 0.992) | 400/s (average 92.6/s) |
| Peak egress | 50,000 × 25 KB = 1.25 GB/s | 10 Gbps |
| Month | 1B × 30.4 = 30.4B requests × 25 KB | 760 TB |
| Traffic tiles | assume 200M/day (20% of views), 5 KB, 30-second cache | 6.08B requests, 30.4 TB a month |
The 25 KB is an average per request, not per tile. A planet-wide build averages under 1 KB per tile, but most tiles are ocean and empty countryside; people look at cities, where tiles are tens of KB.
Routing server memory (North America)
| Part | Math | Size |
|---|---|---|
| Graph size | GraphHopper documents its planet graph at about 415M edges (2022), 86M of them in North America. Road graphs have about 1.2–1.3 edges per junction, so about 70M nodes (estimate); two directions per edge gives about 172M directed edges (an upper bound: one-way roads have one) | 70M nodes, 172M edges |
| Node array | 70M × 16 B | 1.12 GB |
| Edge array | 172M × 16 B | 2.75 GB |
| Shapes | 86M edges × ~6 points × 8 B | 4.1 GB |
| Names and instructions | ~10 B per edge | 0.84 GB |
| Turn tables | estimate | 0.5 GB |
| MLD cell structure and unpacking data | estimate | 2.0 GB |
| Snapping index | estimate | 1.5 GB |
| Profile IDs and table | 172M × 2 B + 2.75 MB | 0.35 GB |
| Two weight sets (serving + customizing) | 2 × (172M × 4 B + ~40% for cell tables) | 1.9 GB |
| Total | ≈ 15 GB |
A c7g.4xlarge has 32 GiB: the graph fits with room for the process, and new releases go out blue/green on fresh instances, so we never hold two graphs on one server.
Routing fleet. Per route computation we plan on about 10 ms of CPU (snapping, an MLD query of a few ms, alternatives, unpacking, ETA, JSON: an estimate to load-test). At 10,000/s that's 100 vCPUs busy; at a 60% target, 167 vCPUs. Each server gives 12 vCPUs to queries (4 go to customization), so we need 14 servers after losing an AZ: 7 per AZ, 21 servers.
Route latency budget (P50, one request, dependent steps add; estimates)
| Step | Time |
|---|---|
| Parse, authenticate | 0.5 ms |
| Snap origin and destination | 0.2 ms |
| MLD query | ~3–4 ms |
| Two alternatives (sharing work with the main query) | ~3 ms |
| Unpack, shapes, steps, BPR penalties | ~2 ms |
| ETA along the route | ~1 ms |
| JSON | ~1 ms |
| Total | ≈ 11 ms, leaving room under the 25 ms P99 for queueing and long routes |
Monthly cost (us-east-1 on-demand list prices, 730 hours, 30.4 days)
| Item | Math | Monthly |
|---|---|---|
| CloudFront base tiles, data out | 760 TB by tier: 10 × $85 + 40 × $80 + 100 × $60 + 350 × $40 + 260 × $30 per TB | ≈ $31,850 |
| CloudFront base tiles, requests | 30.4B ÷ 10,000 × $0.0100 | ≈ $30,400 |
| CloudFront traffic tiles | 6.08B ÷ 10,000 × $0.0100 + 30.4 TB × $30 (the next tier) | ≈ $6,990 |
| Origin Shield | (243M + 1.2B misses) ÷ 10,000 × $0.0075 | ≈ $1,090 |
| S3 tiles (storage and GETs) | ≈ $100 | |
Ingest fleet, 45 × c7g.4xlarge | 45 × $0.58/h × 730 h | ≈ $19,050 |
| NLB | $0.0225/h + LCUs: ~640 MB/s both ways ≈ 2,304 GB/h → 2,304 LCUs × $0.006/h × 730 h | ≈ $10,110 |
| Kinesis raw, 240 shards | 240 × $0.015/h × 730 h + ~13B PUT units × $0.014 per million | ≈ $2,810 |
Map matchers, 21 × c7g.4xlarge | 21 × $0.58/h × 730 h | ≈ $8,890 |
| Kinesis observations, 180 shards | 180 × $0.015/h × 730 h + PUT units | ≈ $2,110 |
| Managed Flink, 121 units | 121 × $0.11/h × 730 h + 50 GB storage per unit × $0.10 | ≈ $10,320 |
| S3 observations, 30 days | ~40 MB/s compressed ≈ 105 TB kept: 50 TB × $23 + 55 TB × $22 | ≈ $2,360 |
Routing fleet, 21 × c7g.4xlarge | 21 × $0.58/h × 730 h | ≈ $8,890 |
Session index, 3 × r7g.xlarge | 3 × $0.2142/h × 730 h | ≈ $470 |
| Session change stream, 8 shards | Kinesis Data Streams for DynamoDB: 175M writes/day × 2 KB ≈ 10.6B change units × $0.10 per million + 8 shards × $0.015/h × 730 h | ≈ $1,150 |
| DynamoDB sessions, on-demand | 175M writes/day × 2 write units × 30.4 = 10.6B × $0.625 per million, plus reads | ≈ $6,950 |
| ElastiCache for Valkey, 2 nodes | ≈ $260 | |
| Fargate services (session API, scheduler, traffic tiles) | ≈ $1,500 | |
| ALB | ≈ $450 | |
| Weekly profile and graph builds | ≈ $2,000 | |
| CloudWatch | ≈ $4,000 | |
| Total | ≈ $151,800/month |
Tiles are about $70,000 of it, and half of that is the per-request charge. The whole telemetry pipeline, from NLB to Flink, is about $55,600. Routing itself is under $9,000. Per route computation, the whole system costs about $28.5 per million (151,800 ÷ 5.32B computations a month).
Where the money is is a senior signal here: making routing faster saves little; the levers are the CDN (fewer, better-cached requests, and at this volume a committed-use discount such as CloudFront's savings bundle) and the telemetry pipeline (the NLB's per-byte charge, the ingest fleet).
R2.7 Trade-Offs
Handling changing weights (published Western Europe figures, rough and hardware-dependent; our graph is about 4 times larger)
| CH | CCH | CRP / MLD (our choice) | ALT | Graph database | |
|---|---|---|---|---|---|
| Query | ~0.11 ms | ~0.14–0.30 ms | ~1.7 ms with turn costs | Tens of ms or more | Seconds |
| New weights | Rebuild: minutes to hours; wrong answers if you only patch weights | Customize: ~0.6–1.3 s on 16 threads | Customize: ~0.4–1 s on 12 cores; only changed cells | Nothing to redo, if weights never drop below the ones landmarks were computed with | A write per edge |
| Weight-independent preprocessing | Part of the rebuild | ~6 min (ordering) | ~11 min (partitioning) | Minutes | None |
| Turns | Edge-based graph, ~5× memory | Edge-based or turn-aware variants | Compact turn tables | Either | Properties |
| Extra memory per weight set | A whole hierarchy | All shortcut weights | Small: ~0.07 GiB in the Europe figures | Landmark tables | Rewrites |
| Publicly known use | OSRM (CH option), GraphHopper | Research and open-source libraries | Bing Maps (CRP, per a 2016 survey); OSRM (MLD option) | GraphHopper (landmark mode) | None for continental routing |
How fine-grained speeds should be
| Per road (all segments of a street) | Per segment and direction (our choice) | Per lane | |
|---|---|---|---|
| Accuracy | Misses a jam at one end | Captures jams where they are | Best, for exits and turn lanes |
| Data needed per value | Plenty | Enough on busy roads; profiles elsewhere | Rarely enough from phones |
| Delta size every 10 s | Small | Hundreds of thousands of values | Much larger |
Push vs polling for reroutes
| Client polls for a new route every N s | Push notification service | Notice in the ping ACK (our choice) | |
|---|---|---|---|
| Delay | Up to N s, plus a request | Seconds, not guaranteed | Up to 5 s (the next ping) |
| Extra load | 2.1M ÷ N requests/s, almost all "no change" | One push per reroute | None: the ping already goes both ways |
| Delivery guarantee | Yes | Background pushes may be throttled | Yes while the app is pinging; a visible push as the fallback |
R2.8 Failure Modes
| Failure | What you'd see | How the design responds |
|---|---|---|
| GPS lost in a tunnel | Pings stop or jump into the harbor | The phone switches to dead reckoning once accuracy is worse than about 50 m: it moves the car along the route line by the last good speed (), helped by the phone's motion sensors, and snaps back to GPS at the exit. On a 2 km tunnel at 60 km/h that's 120 s blind; a 10% speed error puts the icon about 200 m off at the exit. The matcher treats the gap as a longer transition, so the tunnel still gets a speed from exit-minus-entry time. |
| Herd rerouting oscillation | Two parallel roads jam in turns | Planned volume with BPR penalties, capped splitting, and the 2-minute-and-10% hysteresis (step 2.5). The runbook's lever: raise for minor roads in that area. |
| Stale weights (Flink stopped, deltas stop arriving) | weights:latest stops moving | Each server alarms when its newest epoch is older than 60 s. Flink re-sends every segment with live data at least every 5 minutes, so each segment's live speed is used for 10 minutes after its last update; after that the server falls back to the historical profile for that segment, and it keeps a pre-customized "historical for this time slot" weight set, refreshed every 15 minutes, to switch to in one step if the feed is gone for 5 minutes. The closure overlay is applied on top of historical weights too, so closures survive the fallback. Routes stay correct, just less current. |
| A bug writes negative, zero or absurd weights | Weird routes, or a customization that fails | Flink validates before publishing: every weight must be finite and positive, and at least the segment's length at 1.3 times its speed limit (no car does 400 km/h). Values that fail are dropped and counted. If more than 1% of a delta fails, the whole delta is rejected, an alarm fires, and servers keep the last good weights. Routing servers check again before customizing; MLD, like Dijkstra, requires non-negative weights. |
| Routing server memory pressure | Swapping, then slow queries | The graph is 15 GB of 32 GiB; the only growth is the second weight set during customization. A server that can't allocate it skips that epoch and alarms, instead of swapping. New releases never load into a serving instance. |
| A matcher dies | Its shards pause | Kinesis Client Library leases move its shards to the others within about a minute; they re-read from the last checkpoint (up to 10 s back), rebuild each device's recent pings, and re-emit some observations. Flink's one-vote-per-vehicle rule makes the repeats harmless. |
| An ingest server dies | ~450K phones reconnect | Phones reconnect with exponential backoff and full jitter, so the NLB spreads them over the others in about a minute instead of all in one second; the fleet was sized to lose an AZ (15 servers), so one server is noise. |
| Session index replica restarts | One of three replicas missing | It rebuilds by reading the session change stream (24-hour retention) from 6 hours back (longer than any session), in order, applying only newer route versions per session, and joins only once it has caught up to the stream's tip. The other two serve meanwhile. |
R2.9 Production Gotchas
| Gotcha | Symptom | Cause | Fix |
|---|---|---|---|
| Shortest path in SQL | Seconds per route; the database falls over under load | A recursive query per route | Graph in memory (Round 1) |
| Raster tiles | Huge bills and a full re-render for every style change | Images instead of vector data | Vector tiles, styled on the phone |
| Unsynchronized weight updates | Rare wrong or looping routes after an update | Patching weights in a plain CH, or applying deltas out of order or over a gap | CCH or MLD; epochs applied strictly in order; reload the snapshot on any gap |
| Nearest-road snapping of pings | Speeds on the wrong road; frontage roads look like motorways | No sequence model | HMM map matching, with enough decision delay |
| Counters that only go up | Every street slowly looks congested | Increment on assignment, no reliable decrement | Derive planned volume from current session versions; recompute periodically |
| Averaging speeds per ping | A parked car or a queue at one light dominates | Many pings from one car | One vote per vehicle per window; harmonic mean |
R2.10 Pillar Check
| Pillar | What Round 2 adds |
|---|---|
| Reliability | Streams decouple ingestion from processing; every consumer is safe to replay (absolute speeds, one vote per vehicle, ordered epochs, conditional route writes); stale or invalid weights fall back to historical ones; reroutes are rate-limited; Kinesis and instance quotas checked before launch REL 4 · REL 5 · REL 11 · REL 1 |
| Performance Efficiency | MLD chosen by comparison for changing weights; the routing fleet, matchers and ingest fleet sized from peak with an AZ lost; a 30-second freshness budget proved stage by stage PERF 1 · PERF 2 · PERF 3 |
| Security | Location data classified as sensitive: pseudonymous device IDs rotated daily, no raw ping archive, observations kept 30 days; TLS on every connection; the ingest fleet can only write to its stream SEC 7 · SEC 3 · SEC 9 |
| Cost Optimization | About $151,800 a month; we know where it goes (tiles 46%, telemetry 37%), rejected per-request services for pings, and priced the CDN by tier COST 5 · COST 8 |
| Operational Excellence | Alarms on weight age, iterator age, match rate, customization time and ETA error, each with a first action (R3.9 lists them) OPS 8 |
| Sustainability | Light this round: raw pings aren't stored; observations expire after 30 days; Graviton instances throughout SUS 4 · SUS 5 |
R2.11 Round 2 Rubric and Follow-Ups
What a senior (L6) answer adds over L5
- Explains exactly why plain CH breaks with changing weights (missing shortcuts, not just stale ones) and compares CCH and MLD with real figures.
- Builds a telemetry pipeline sized in shards and servers, keyed correctly at each stage, and rejects per-request services with the arithmetic.
- Does map matching as a sequence problem and knows the decision delay it costs.
- Proves a freshness budget end to end, stage by stage.
- Blends live and historical speeds by arrival time, and measures ETA error.
- Prevents herding with planned volume, BPR and splitting, and keeps the counts correct in both directions.
- Reroutes nearest-first in rate-limited waves, with conditional writes and retries that read state first.
Follow-up questions
-
"Why not do map matching inside Flink?" Answer: matching needs the road graph, shapes and a spatial index in memory, about 8 GB for North America, next to each device's recent pings. A Flink processing unit has 4 GB. We could split the graph by area, but then a car crossing a boundary would need its sequence handed over. A matcher fleet on larger instances, keyed by device, is simpler. Aggregation, which only needs per-segment state, fits Flink well.
-
"A city partner wants the live speed of every segment every 10 seconds." Answer: we already produce that: the deltas. We publish them to the partner as files (or a feed) with the epoch numbers, so the partner can detect gaps and reload a snapshot, exactly as our servers do. We aggregate further if the contract or privacy rules require it; one vehicle's speed on a quiet street at 3 am can identify someone, so we suppress segments with fewer than a minimum number of vehicles.
-
"What if we must honor a truck's height limit on some routes?" Answer: that's another weight set: a truck metric where low bridges and restricted roads have infinite weight. MLD's shape-independent preprocessing is shared, and each weight set costs only its own customization and about 1 GB of memory at our size. We'd keep a truck metric customized on the same 10-second deltas, or less often if trucks are few.
Interview gotchas from this round
| Gotcha | Why it's wrong |
|---|---|
| "Update CH weights in place" | CH's skipped shortcuts assume the old weights; answers can be wrong. |
| "Store every ping" | 4M writes a second of the most sensitive data you have, rarely read. |
| "API Gateway for telemetry" | Per-request pricing at 4M/s is millions a month. |
| "Snap each ping to the nearest road" | Jumps between parallel roads; use a sequence model. |
| "Always the fastest route" | Herds drivers into jams you created. |
| "Five nines" | 26 seconds a month; one bad release spends it. |
Round 3 · Architect · "The Whole Planet, Offline, and Multi-Modal"
~45 min · Principal (L7) · 3 regions (us-east-1, eu-west-1, ap-southeast-1) plus the CDN · 415M-edge planet graph · 12M pings/s (18M peak) · 875M route computations/day, 50K/s planned · 4B tiles/day, 200K/s peak · closures everywhere within seconds · 99.99% per region
R3.0 Where We Left Off
This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.
Round 2 in 60 seconds. "A consumer maps app for North America: 20M phones send a 32-byte ping every 5 seconds, 4M a second, over long-lived TLS connections to an ingest fleet behind an NLB. Kinesis keyed by device feeds map matchers that run an HMM with Viterbi, deciding each ping after one or two more arrive; a second stream keyed by road segment feeds Flink, which takes one vote per vehicle per segment, a harmonic mean over 60 seconds, validates it, and every 10 seconds writes a delta of absolute speeds with an epoch number. Because plain CH gives wrong answers when weights change, routing servers run MLD: cells with precomputed boundary costs, customizing only changed cells, applying epochs strictly in order and reloading a snapshot on any gap, all within a 23-second worst case. ETAs blend live and historical speeds by when we reach each segment. Planned volume with BPR penalties and split assignment prevents herding. Closures reroute the affected sessions nearest-first at 1,000 a second, delivered in the ping ACK. About 21 routing servers for 10,000 computations a second, and about $151,800 a month, half of it tiles. Open costs: one region, one continent, driving only, one map source, and no offline use."
Architecture v2, compact
Synthesizing vector architecture diagram...
Round 2 in one picture: a live-weight pipeline feeding a customizable routing graph, for one continent.
Steps so far
| Step | Problem | Component |
|---|---|---|
| 1.1 | Map rendering | Vector tiles per release on a CDN |
| 1.2–1.4 | Route search | Graph in RAM; Dijkstra → A* → CH |
| 1.5–1.6 | Turns, start points | Edge-based graph; snap to edges |
| 2.1–2.2 | Telemetry | Streams; HMM map matching |
| 2.3 | Changing weights | MLD with ordered 10-second deltas |
| 2.4 | ETAs | Historical profiles blended with live speeds |
| 2.5–2.6 | Herding, closures | BPR and splitting; session index and reroute waves |
Open costs: one region, one continent, driving only, one map data source, nothing offline.
R3.1 The Scope Raise
Interviewer: "We're going global. The app should work on every continent, with the data served from nearby. Users want to download a city before a trip and navigate with no signal. They want walking, cycling and public transit, not just driving."
Interviewer: "Our map data now comes from our own survey vehicles, from partners like transport agencies, and from users who report and fix things, and those sources disagree. And when a road closes, that has to show up for everyone within seconds, wherever they are."
We ask back, and say what each answer changes.
| We ask | Interviewer answers | What it changes in the design |
|---|---|---|
| Which regions? | us-east-1 for the Americas, eu-west-1 for Europe, Africa and the Middle East (EMEA), ap-southeast-1 for Asia and Oceania. Traffic splits about 40/35/25. | Each continent's graph lives in its region (step 3.1); cross-continent trips need a plan. |
| How big is the global load? | About 5 times Round 2's routes, 3 times its telemetry and 4 times its tiles, plus offline downloads. | 875M route computations a day, 12M pings/s, 4B tiles/day (R3.6). |
| How big may an offline area be? | A city and its surroundings; a few hundred MB at most. | Packs made of fixed cells, with a size cap (step 3.2). |
| Which transit data? | Agencies publish timetables as GTFS feeds, some with real-time updates. Thousands of feeds, of varying quality. | A timetable router and a feed validation pipeline (step 3.3). |
| Who may edit the map? | Our staff, partners with contracts, and any user, with review. | Conflation with confidence scores and human review (step 3.4). |
| "Within seconds" for closures: routes, map display, or both? | Routes within a few seconds; the map display within a minute. | Closures as a live overlay, separate from map releases (step 3.5). |
| Should we build all of this, or buy some? | Tell us. | Build vs buy with numbers (step 3.6). |
Scope change
| Round 2 | Round 3 | |
|---|---|---|
| Area | North America | The planet: 3 regions |
| Graph | 86M edges, driving | 415M edges; driving, walking, cycling; plus transit timetables |
| Map data | One source | Surveys, partners, users; conflated and versioned |
| Offline | None | Downloadable city packs with local routing |
| Closures | Applied in one region | Everywhere within seconds |
| Routes | 10,000 computations/s planned | 50,000/s planned across regions |
| Tiles | 1B/day | 4B/day, plus 1.2 PB/month of offline downloads |
R3.2 What Breaks in the Round 2 Design
| Round 2 choice | What breaks at the new scope |
|---|---|
| One graph per server | The planet with four weight sets is about 82 GB (R3.6). It would fit a 128 GiB machine, but every server would customize the whole planet every 10 seconds, users on other continents would wait 150 ms or more of round trip for every route, and one bad delta or release would hit everyone. |
| Driving only | Walking and cycling use different roads and paths; transit runs on a timetable, not a graph with fixed weights. |
| One map data source | Survey, partner and user data disagree; last-write-wins would let one wrong report delete a road. |
| Tile regeneration per release | A closure can't wait for a weekly release: a published planet build of zoom 0–14 takes a few hours on a large machine. |
| Cross-continent trips | Europe and Asia are joined by road; a route from Berlin to Beijing crosses two regions' graphs. |
| Everything online | No signal, no map and no route. |
R3.3 New Requirements and API Additions
A mode parameter. POST /v1/routes now takes mode: DRIVE, WALK, BIKE or TRANSIT, and for transit a depart_at or arrive_by time.
The offline pack manifest. An offline area is a set of pack cells. A pack cell is the area of one zoom-8 tile: about 156 km wide at the equator, about 97 km at London's latitude. Fixed cells mean packs are built once per release and shared by everyone who downloads that area.
json{ "cell": "8/127/85", "release": "r2026-09-20", "expires_at": "2026-11-15T00:00:00Z", "parts": [ { "kind": "tiles", "bytes": 88400000, "sha256": "4be1...", "url": "/v1/packs/r2026-09-20/8/127/85/tiles.pmtiles" }, { "kind": "graph", "bytes": 41200000, "sha256": "90ac...", "url": "/v1/packs/r2026-09-20/8/127/85/graph.bin" }, { "kind": "search", "bytes": 19800000, "sha256": "d17f...", "url": "/v1/packs/r2026-09-20/8/127/85/search.bin" } ], "diff_from": [ { "release": "r2026-09-13", "bytes": 23900000, "url": "/v1/packs/r2026-09-20/8/127/85/diff-r2026-09-13.bin" } ] }
| Field | Why |
|---|---|
release | The pack is one consistent snapshot; the app never mixes tiles of one release with a graph of another |
expires_at | After this, the app warns the user that the offline data is old, and prefers online data when it has signal |
sha256 | Every part is checked after download; a corrupt pack is never used |
diff_from | A small update from a recent release, instead of the full pack again |
Transit feed ingest (partners and our crawler):
httpPUT /internal/v1/transit/feeds/sg-lta HTTP/1.1 Host: ingest.maps.example Content-Type: application/zip <GTFS zip: stops.txt, routes.txt, trips.txt, stop_times.txt, calendar.txt ...>
httpHTTP/1.1 202 Accepted Content-Type: application/json { "feed_id": "sg-lta", "version": 118, "status": "VALIDATING" }
GTFS (General Transit Feed Specification) is the standard format agencies publish timetables in: a zip of CSV files for stops, routes, trips and each trip's stop times. GTFS Realtime is its companion for live data (trip delays, vehicle positions, service alerts), encoded as Protocol Buffers.
Map edits:
httpPOST /v1/map-edits HTTP/1.1 Host: api.maps.example Authorization: Bearer <access token> Content-Type: application/json { "feature_ref": "edge:88123901", "change": { "attribute": "oneway", "from": "no", "to": "yes" }, "evidence": { "photo_ids": ["ph_71c2"], "observed_at": "2026-09-27T09:12:00Z" }, "client_release": "r2026-09-20" }
httpHTTP/1.1 202 Accepted Content-Type: application/json { "edit_id": "me_01J90A", "status": "PENDING_REVIEW", "risk": "HIGH" }
202 Accepted: an edit is a proposal, never an immediate change to the map.
Recap
- Routes take a mode; transit takes a time.
- Offline areas are made of fixed zoom-8 cells, each a versioned, checksummed, expiring pack with diffs.
- Timetables arrive as GTFS; edits arrive as proposals with evidence.
R3.4 Design Evolution: From One Continent to the Planet
Step 3.1: "The Planet Graph Doesn't Fit One Server's Budget"
The problem: the planet graph is 415M edges. With walking, cycling and two driving weight sets it's about 82 GB per server, every server would customize the whole planet every 10 seconds, and a user in Sydney would cross an ocean for every route. What would you do?
Synthesizing vector architecture diagram...
The EMEA server does the search; APAC answers two small questions. Everything inside one continent stays inside one region.
Primitive: Database Sharding and Partition Keys (the same lesson: partition by the key your queries stay inside, here geography)
Step 3.2: "Offline Maps"
The problem: a traveler downloads London before a flight and needs to navigate with the phone in airplane mode. A full-planet tile set is about 99 GB. What would you do?
How big is a pack? London as an example, estimates:
| Part | Math | Size |
|---|---|---|
| Tiles | ~60 × 50 km: 1.52 km per zoom-14 tile at 51.5° → about 40 × 33 = 1,320 tiles, × 4/3 for zooms 9–13 ≈ 1,760 tiles × ~50 KB (dense city) | ≈ 88 MB |
| Routing subgraph | a few hundred thousand edges with shapes and profile IDs | ≈ 40 MB |
| Search | addresses and places | ≈ 20 MB |
| Pack | ≈ 150 MB | |
| Monthly diff | ~20% of tiles changed + a new graph | ≈ 25 MB |
Synthesizing vector architecture diagram...
The app only ever routes on a pack in Ready or Stale. An update never touches the pack in use until the new one is verified.
Step 3.3: "Transit Routing"
The problem: a user asks how to get across Singapore by bus and train, leaving at 8:10. A first attempt adds bus routes to the road graph as edges with an average travel time. What would you do?
textRAPTOR(origin_stops_with_walk_times, depart_at, max_rounds): arrival[stop] = depart_at + walk time for origin stops, infinity elsewhere marked = origin stops for k = 1 .. max_rounds: routes_to_scan = routes serving any marked stop (from its earliest marked stop) marked = {} for each route r in routes_to_scan: trip = none for each stop p along r, from the earliest marked stop: if trip is set and trip's time at p < arrival[p]: arrival[p] = trip's time at p; mark p -- ride to p if we can catch an earlier trip of r at p (arrival_prev_round[p] <= its departure): trip = the earliest such trip -- (re)board for each marked stop p: relax footpaths p -> q (arrival[p] + walk <= arrival[q]) if nothing was marked: stop answer = best of arrival[stop] + walk time from stop to the destination
Step 3.4: "Map Data From Many Sources Disagrees"
The problem: a partner's feed says a street is two-way; our survey says one-way; a user edit yesterday flipped it back to two-way. The last write wins, so drivers are routed the wrong way down it. What would you do?
Synthesizing vector architecture diagram...
No source writes to the map directly. Everything becomes a scored proposal, and the map changes only through a validated release.
Step 3.5: "A Closure Must Show Up Everywhere in Seconds"
The problem: a landslide closes a mountain road in Italy. Routes must avoid it within seconds, in EMEA and anywhere else that might route over it, and the map should show it. Map releases are weekly. What would you do?
Synthesizing vector architecture diagram...
Routes in the owning region change within about a second, other regions within a few seconds, the map display within about a minute.
Step 3.6: "Build or Buy?"
The problem: a board member asks why we run all this instead of paying a mapping service per request, for example Amazon Location Service. What would you do?
Round 3 Step Summary
| Step | Problem | Component | What it costs us |
|---|---|---|---|
| 3.1 | The planet on one server | One continent per region; planet top level everywhere; stitched cross-region routes; modes as weight sets | Rare slow routes; a region serves only its continent |
| 3.2 | Offline | Zoom-8 cell packs per release: tiles, subgraph, search; diffs; local A* | A pack pipeline; 1.2 PB/month of downloads |
| 3.3 | Transit | RAPTOR on GTFS, walking legs, real-time updates | A second routing engine |
| 3.4 | Conflicting sources | Conflation: match, score, review by risk, weekly release | Data team and reviewers |
| 3.5 | Closures in seconds | Live overlay in a global table; owning region applies first | Two paths to reconcile |
| 3.6 | Build or buy | Numbers per scale; buy the small, build the strategic | A platform team |
R3.5 Global Architecture
Synthesizing vector architecture diagram...
Each region is a full copy of Round 2 for its continent. Only three things are global: the CDN, the closures table, and the weekly release that every region loads.
The pieces:
- Per region: Round 2's telemetry pipeline, MLD routing fleet, sessions and reroute waves, plus a transit engine (RAPTOR over that region's feeds) and a closure service.
- Standby graphs: each region keeps 3 warm servers (one per AZ) holding another region's graph: EMEA's in us-east-1; the Americas' and APAC's in eu-west-1 (R3.8).
- Map data: conflation and release building run in one region; releases (graphs, tiles, packs) are replicated to all three regions' S3 buckets and served through CloudFront.
- Telemetry stays in its region. Pings are processed where they arrive; only aggregated speeds would ever leave.
Trace 1: a cross-continent route (Berlin to Beijing)
Synthesizing vector architecture diagram...
Two round trips across regions, for a route that takes days to drive.
Trace 2: an offline route. The phone in airplane mode holds the four cells covering greater London (release r2026-09-20, verified). The user asks for a route from Camden to Greenwich. The app snaps both points in the local search index, runs bidirectional A* on the merged cell graph with ETAs from the pack's historical profiles for the current time slot, and shows the route. It marks the ETA "offline, no live traffic". When signal returns, it fetches current closures for its cells and, if the route crosses one, recomputes locally or asks the server.
Trace 3: a closure broadcast. The sequence in step 3.5: the closure is written once in eu-west-1, applied to EMEA routing in about a second, affected sessions are rerouted nearest-first, other regions get it in about a second more, the overlay tiles show it within about a minute, and offline phones on their next connection.
R3.6 Numbers and Cost
Graph size and memory per region (graph sizes from GraphHopper's 2022 figures of 415M edges for the planet, 150M in Europe and 86M in North America; the rest of the split is our assumption. Memory per edge scaled from Round 2's 13.2 GB of shared structures for 86M edges, plus four weight sets: two for driving, one each for walking and cycling.)
| Region | Edges | Shared structures (≈ 0.153 GB per million edges) | 4 weight sets | Total |
|---|---|---|---|---|
| Americas | 86M + ~35M South America = 121M | 18.6 GB | 4 × 1.36 GB | ≈ 24 GB |
| EMEA | 150M + ~34M Africa and Middle East = 184M | 28.2 GB | 4 × 2.06 GB | ≈ 37 GB |
| Asia and Oceania | ~110M | 16.9 GB | 4 × 1.23 GB | ≈ 22 GB |
| Planet, for comparison | 415M | 63.7 GB | 4 × 4.65 GB | ≈ 82 GB |
Each region fits a c7g.8xlarge (64 GiB, 32 vCPUs) with room to spare.
Load per region
| Quantity | Americas (40%) | EMEA (35%) | APAC (25%) |
|---|---|---|---|
| Pings/s, busy hour (12M total) | 4.8M | 4.2M | 3.0M |
| Pings/s, yearly peak (18M total) | 7.2M | 6.3M | 4.5M |
| Raw stream shards at peak (× 32 B, +25%) | 230 → 288 | 202 → 252 | 144 → 180 |
| Route computations/day (5 × Round 2's 175M = 875M) | 350M | 306M | 219M |
| Route computations/s planned (50K total) | 20,000 | 17,500 | 12,500 |
| Tile requests/day (4B total) | 1.6B | 1.4B | 1.0B |
The Kinesis quota: the default is 20,000 shards per account in us-east-1, eu-west-1 and us-west-2, and 1,000 or 6,000 in other regions (we check ap-southeast-1's value in Service Quotas). ap-southeast-1 needs about 315 shards for the two streams (180 raw + about 135 observations), so it fits, but we confirm it and raise the standby regions' quotas (instances, shards) before launch.
Routing fleets (12 ms of CPU per request in EMEA for its bigger graph, 11 ms elsewhere, estimates; 60% target; 26 of 32 vCPUs for queries; enough servers after losing an AZ)
| Region | vCPUs needed | Servers after losing an AZ | Per AZ | Servers |
|---|---|---|---|---|
| Americas | 20,000 × 0.011 ÷ 0.6 = 367 | 367 ÷ 26 = 14.1 → 15 | 8 | 24 |
| EMEA | 17,500 × 0.012 ÷ 0.6 = 350 | 13.5 → 14 | 7 | 21 |
| APAC | 12,500 × 0.011 ÷ 0.6 = 229 | 8.8 → 9 | 5 | 15 |
Tiles, offline packs and the CDN. CloudFront bills data transfer by the region of the edge that serves the viewer, and computes volume tiers separately for each such region, so we price each region with its own tiers. Asia-Pacific prices are higher at every tier.
| Region (edge pricing used) | Tiles | Offline packs | Traffic tiles | Total data out |
|---|---|---|---|---|
| Americas (US/Canada/Mexico rates) | 1.6B × 25 KB × 30.4 = 1,216 TB | 40% of 1.2 PB = 480 TB | 48.6 TB | ≈ 1,745 TB |
| EMEA (Europe rates) | 1,064 TB | 420 TB | 42.6 TB | ≈ 1,527 TB |
| APAC (Southeast Asia rates) | 760 TB | 300 TB | 30.4 TB | ≈ 1,090 TB |
Offline downloads: assume 5M full cell downloads a month × ~140 MB = 700 TB, and 20M diff updates × ~25 MB = 500 TB, 1.2 PB in total.
| Region | Data out by tier | Requests | CDN total |
|---|---|---|---|
| Americas | 10 TB × $85 + 40 × $80 + 100 × $60 + 350 × $40 + 524 × $30 + 721 × $25 per TB ≈ $57,800 | (1.6B + 0.32B traffic) × 30.4 = 58.4B × $0.0100 per 10,000 ≈ $58,400 | ≈ $116,200 |
| EMEA | same tiers, 1,527 TB ≈ $52,300 | 51.1B × $0.0120 per 10,000 ≈ $61,300 | ≈ $113,600 |
| APAC | 10 × $120 + 40 × $100 + 100 × $95 + 350 × $90 + 524 × $80 + 66 × $70 per TB ≈ $92,800 | 36.5B × $0.0120 per 10,000 ≈ $43,800 | ≈ $136,500 |
| CDN | ≈ $366,300 |
APAC serves the smallest share of traffic and costs the most. The table simplifies: it prices the Americas at US rates and EMEA at European rates, although South America, Africa and the Middle East cost more, and it prices all of APAC at Southeast Asian rates, although Japan, Australia and India have their own. A real bill needs the per-country split. At this volume, a committed-use discount or private pricing is the first lever.
Tile storage per release. A published full-planet build of zoom 0–14 is about 264M tiles and 99 GB; with packs (about the same again) and graphs, a release is a few hundred GB, replicated to three regions and kept for the last 8 releases: a few TB, under $1,000 a month.
Monthly cost (on-demand list prices; EC2 in eu-west-1 about 7% and in ap-southeast-1 about 15% above us-east-1, from the c7g.4xlarge prices of $0.6202 and $0.6664 against $0.58 an hour; we apply the same uplift to each region's whole pipeline, a simplification)
| Item | Math | Monthly |
|---|---|---|
| CDN (tiles, traffic tiles, offline packs) | above | ≈ $366,300 |
| Telemetry pipelines | Round 2's ≈ $55,650 per 4M pings/s → $13,912 per 1M/s: 4.8 × 13,912 + 4.2 × 13,912 × 1.07 + 3.0 × 13,912 × 1.15 | ≈ $177,290 |
| Routing fleets | 24 × $1.16 × 730 + 21 × $1.16 × 1.07 × 730 + 15 × $1.16 × 1.15 × 730 | ≈ $53,960 |
Standby routing (9 × c7g.8xlarge) | 3 in us-east-1 × $1.16 × 730 + 6 in eu-west-1 × $1.16 × 1.07 × 730 | ≈ $7,980 |
| Sessions, index, scheduler, API services | Round 2's ≈ $10,780 × 5, with regional uplift | ≈ $55,790 |
| Transit engines | 6 × c7g.2xlarge per region | ≈ $4,000 |
| Origin Shield | ≈ $4,000 | |
| Map data: conflation and review tooling (excluding imagery collection) | estimate | ≈ $15,000 |
| Builds: tiles, graphs, packs, profiles | ≈ $8,500 | |
| Closures global table, S3 releases, replication | ≈ $900 | |
| CloudWatch | ≈ $12,000 | |
| Total | ≈ $705,700/month |
Synthesizing vector architecture diagram...
Serving pictures of the map costs more than all the computing combined; the famous part, routing, is under a tenth.
Per route computation, the routing fleets and standbys cost about $2.3 per million (61,940 ÷ 26.6B computations a month), and the whole platform about $26.5 per million. The telemetry pipelines cost about $5.6 per billion pings (177,290 ÷ 31.5 trillion pings a month).
R3.7 Trade-Offs
How to split the graph
| One planet graph per server | One continent per region (our choice) | Smaller partitions (country per server) | |
|---|---|---|---|
| Memory per server | ≈ 82 GB | 22–37 GB | A few GB |
| Customization every 10 s | The whole planet | One continent | One country |
| Cross-partition routes | None | Rare (< 0.1%), stitched, under 1 s | Common (every border crossing), a round trip each |
| Blast radius of a bad delta | Everyone | One continent | One country |
| Latency to users | One region for all | Near most users | Near most users |
Offline freshness vs download size
| Full pack every release | Diff per release (our choice) | Tiles only, no routing | |
|---|---|---|---|
| Monthly download for a city | ~150 MB × 4 weekly releases | ~25 MB | ~90 MB |
| Offline routing | Yes | Yes | No |
| Freshness | Weekly | Weekly | Weekly |
| Server work | Build packs | Build packs + diffs per recent release | Tiles only |
Build vs buy
| Buy (Amazon Location Service or another vendor) | Build (our choice at Round 3's scale) | |
|---|---|---|
| Cost at 1–2B requests a month | Tens of thousands a month; no platform team | A few thousand in AWS, plus a team |
| Cost at 100B+ requests a month | Millions a month | About $700K a month in AWS, plus a larger team |
| Live traffic | The provider's | Our own phones' |
| Custom weights, offline packs | Limited to what the API exposes | Anything we build |
| Data rights | The provider's terms | Our data and our licenses |
Closing the loop. Round 1 found the shape of the problem: static tiles on a CDN, a static graph in memory, and preprocessing that turns a country-wide search into a few hundred steps. Round 2 made the weights live: a telemetry pipeline, map matching, and a preprocessing technique that separates the graph's shape from its weights, with a proven 30-second path from a braking car to a new route. Round 3 made it planetary: continents in their own regions, modes as weight sets, timetables with their own router, offline packs, conflated data, and closures as a live overlay. Through all three rounds the same idea holds: precompute what is static, stream what changes, and combine them at query time.
R3.8 Failure Modes
| Failure | What you'd see | How the design responds |
|---|---|---|
| A regional outage (eu-west-1 unavailable) | EMEA route requests fail health checks | Other regions don't hold EMEA's graph at full capacity, so we planned it: us-east-1 keeps 3 warm servers with EMEA's graph. Route 53 health checks move EMEA's route traffic to us-east-1; the standby serves navigating sessions' reroutes first and sheds new route previews, while its Auto Scaling group grows to 21 servers from an image with the graph built in (a few minutes to launch and load, an estimate; RTO about 10 minutes to full capacity). EMEA telemetry is lost for the duration, so routes use historical weights. Tiles keep working: CloudFront serves from cache, and misses fail over to a replicated bucket. The standby region's quotas (instances, Kinesis shards) are checked in Service Quotas and raised in advance where needed. |
| A bad map release | Route regressions in one area after the switch | Roll back by release number: style.json points apps at the previous tiles, routing fleets redeploy the previous graph blue/green, packs stay on their verified release. Closures in the overlay are independent of releases, so they survive the rollback. |
| An offline pack with a wrong closure | An offline user is routed into, or around, a road that isn't closed | Packs never contain closures, by design; they only reach phones as an online overlay with valid_until. A wrong closure is ended in the global table; phones drop it on their next sync or when valid_until passes by the phone's own clock. |
| A transit feed with errors | Trips with times going backwards, a whole day with no service, or 40% of trips gone | Every feed version is validated before use: schema and references, stop times that never go backwards, calendar coverage, and a comparison with the previous version (more than 30% of trips gone is quarantined for review). A failed version never replaces the last good one. A real-time feed that goes silent falls back to the timetable. |
| A replication lag spike on the global table | Closures reach other regions late | The owning region already applied it; other regions only need it for cross-region stitching and standby graphs. An alarm on replication latency; ops can apply the closure directly in the other regions if the lag persists. |
| A cross-region stitch fails (APAC slow or down) | Berlin-to-Beijing requests time out | A 2-second budget, then an error saying long-distance routing is degraded; within-continent routes are unaffected. |
R3.9 Runbook and Incident Response
Signals, per region OPS 8 · REL 6
| Signal | Alarm | Severity | First action |
|---|---|---|---|
| Route P99 server time | > 25 ms for 5 min | P2 | Check CPU and customization time; scale the fleet |
| Weight epoch age on routing servers | > 60 s | P1 | Check Flink and Valkey; fleets fall back to historical weights at 5 min |
| Raw stream iterator age (telemetry lag) | > 15 s for 5 min | P2 | Scale matchers; check for a hot shard |
| Map-matching rate (share of pings matched to a road) | < 90% for 10 min | P2 | A bad release (missing roads) or a matcher bug |
| Customization duration | > 6 s | P2 | CPU pressure; delta sizes |
| Tile hit ratio | < 98.5% | P3 | A new release rolling out, or a Cache-Control regression |
| Reroute push latency (closure to notice delivered, P95) | > 30 s | P2 | Scheduler and ingest notice path |
| Closure replication lag | > 10 s | P3 | Apply directly in other regions if it persists |
| ETA median error | > 15% for an hour in a region | P3 | Check profiles and the live blend |
Emergency closure procedure OPS 10
- Create the closure in the owning region through the closure tool (which writes the global-table item with a condition that it doesn't exist yet, and applies it to the fleet).
- Verify on a routing server that the edges carry infinite weight, and that a test route across them now avoids them.
- Watch the reroute wave: sessions found, rerouted, notices delivered.
- Check other regions received the item (replication lag).
- Set
valid_untilto the expected reopening; extend it rather than let it lapse silently.
Routing fallback procedure
- If deltas are invalid or customization keeps failing, pin the fleet to the historical weight set (a feature flag): routes stay correct, only less current.
- If a new graph release is suspect, roll back to the previous release number, blue/green.
- If a region is down, fail over its route traffic to the standby region and scale the standby out.
Go deeper: CLI playbook
Plain commands, one at a time. Replace names and IDs with real ones.
text# 1. Telemetry lag on a region's raw stream aws cloudwatch get-metric-statistics --region eu-west-1 --namespace AWS/Kinesis --metric-name GetRecords.IteratorAgeMilliseconds --dimensions Name=StreamName,Value=telemetry-raw-euw1 --start-time 2026-09-28T10:00:00Z --end-time 2026-09-28T11:00:00Z --period 60 --statistics Maximum # 2. The raw stream's shard count and status aws kinesis describe-stream-summary --region eu-west-1 --stream-name telemetry-raw-euw1 # 3. Tile cache hit rate (CloudFront metrics live in us-east-1; this one needs additional metrics turned on) aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/CloudFront --metric-name CacheHitRate --dimensions Name=DistributionId,Value=E2EXAMPLE1 Name=Region,Value=Global --start-time 2026-09-28T10:00:00Z --end-time 2026-09-28T11:00:00Z --period 300 --statistics Average # 4. Create a closure only if it doesn't exist yet aws dynamodb put-item --region eu-west-1 --table-name road-closures --item file://cl_88210.json --condition-expression "attribute_not_exists(closure_id)" # 5. Roll apps back to the previous tile release aws s3 cp s3://maps-releases-use1/styles/r2026-09-13/style.json s3://maps-public-use1/v1/style.json --cache-control "max-age=60" aws cloudfront create-invalidation --distribution-id E2EXAMPLE1 --paths "/v1/style.json" # 6. Scale a standby routing group during a regional failover aws autoscaling set-desired-capacity --region us-east-1 --auto-scaling-group-name routing-standby-emea --desired-capacity 21
R3.10 Pillar Check
| Pillar | What Round 3 adds |
|---|---|
| Reliability | Continents isolated in their own regions; warm standby graphs in a second region with a planned failover; releases versioned and replicated, rolled back by number; quotas checked, and raised where needed, in standby regions REL 10 · REL 13 · REL 9 · REL 1 |
| Performance Efficiency | Routes answered in the user's region; the planet's top level held everywhere for rare long routes; modes as extra weight sets instead of extra graphs; the CDN near every user PERF 1 · PERF 4 |
| Security | Telemetry stays in its region; user edits are proposals with evidence, reviewed by risk; location and edit data classified and retained for set periods; encrypted at rest (S3, DynamoDB, Kinesis with KMS); least-privilege roles per pipeline SEC 7 · SEC 3 · SEC 8 |
| Cost Optimization | About $705,700 a month; CDN priced per edge region and tier (APAC highest); build vs buy decided by scale and weighed against team cost COST 8 · COST 11 · COST 7 |
| Operational Excellence | Release validation and staged rollout; runbooks for closures, fallback and failover; signals per region with first actions OPS 6 · OPS 8 · OPS 10 |
| Sustainability | Regions near users; offline diffs instead of full re-downloads; walking and cycling as weight sets on shared structures; observations expire SUS 1 · SUS 3 · SUS 4 |
R3.11 Round 3 Rubric and Follow-Ups
What an architect (L7) answer adds over L6
- Partitions the graph by geography because queries stay inside geography, and explains how the multi-level structure stitches the rare cross-partition route.
- Treats modes as weight sets on one structure, and transit as a different problem with a timetable algorithm.
- Designs offline as versioned, checksummed, expiring packs with diffs, and keeps live data out of them.
- Replaces last-write-wins with provenance, confidence and review by risk, ending in validated releases.
- Separates urgent overlays (closures) from slow releases, with one writer per closure and clocks that aren't the database's.
- Plans the regional failure honestly: which region can serve which continent, at what capacity, after how long.
- Decides build vs buy with numbers per scale, including team cost and data rights.
Follow-up questions
-
"Why not put the whole planet on 128 GiB servers everywhere and skip the stitching?" Answer: it would fit, at about 82 GB. But every server in every region would customize the whole planet every 10 seconds, for traffic on continents its users never drive on; one bad delta anywhere would reach every server; and we'd still want the servers near users. The stitch costs two round trips on under 0.1% of requests. If cross-continent demand grew, a middle ground is to hold neighboring continents' graphs as well, with historical weights only.
-
"A driver goes online halfway through an offline route, and the server's route disagrees." Answer: the server's route is built on the current release with live weights; the phone's on an older pack with historical weights. When online, the app asks the server and, if the new route saves at least our reroute threshold (2 minutes and 10%), offers it; otherwise it keeps the current one. Routes carry their
map_release, so a mismatch is explainable in support logs. -
"A country requires that location data from its citizens stays in the country." Answer: that's a legal question first; we keep the answer general until counsel confirms what's required. Architecturally, telemetry already stays in the region where it's processed, so the change is to add an ingest and processing pipeline in a region inside that country (if AWS has one) and route that country's pings there by the phone's region setting, publishing only aggregated speeds outward.
Interview gotchas from this round
| Gotcha | Why it's wrong |
|---|---|
| "Shard the graph by node hash" | Every search step becomes a network hop. Partition by geography. |
| "Treat buses as road edges" | Transit costs depend on the time you arrive at the stop. |
| "Last write wins for map data" | One bad edit deletes a road; keep provenance and review by risk. |
| "Bake closures into offline packs" | Wrong closures persist for weeks on phones that can't be reached. |
| "Other regions can take over any continent" | Only if they hold its graph and have the quota; plan standbys explicitly. |
| "Vendors are always cheaper" | At 100B+ requests a month, per-request pricing is millions. |
Loop Closer: Interview Strategy for All Three Rounds
How to Run Each 60-Minute Round
| Time | Round 1 | Round 2 | Round 3 |
|---|---|---|---|
| 0–5 min | Scoping: area, graph size, modes, turns, traffic or not | Restate Round 1 in 60 seconds | Restate Round 2 in 60 seconds |
| 5–15 min | Requirements, z/x/y tiles, the route API | Scope raise → what breaks | Scope raise → what breaks |
| 15–40 min | Steps 1.0–1.6: tiles on a CDN, graph in RAM, Dijkstra → A* → CH, turns, snapping | Steps 2.1–2.6: telemetry, map matching, CCH vs MLD, ETAs, herding, closures | Steps 3.1–3.6: partitions, offline, transit, conflation, closures, build vs buy |
| 40–50 min | Numbers (graph memory, tiles, CDN cost) and trade-offs | Numbers (shards, fleets, the 30-second budget), cost | Numbers per region, CDN by edge region, total cost |
| 50–60 min | Failures + pillar check | Failures, gotchas + pillar check | Failures, runbook, pillar check |
For how to spend a single 45-minute round, see the 45-minute interview blueprint.
The Two Sentences That Matter Most
- Opening a round: "Before I design: how big is the road network, which modes, do weights change, and how fast must routes and traffic be?"
- When the scope is raised: "Here's what breaks. I'll fix anything that could give a wrong or illegal route first, then freshness, then cost."
And the one sentence specific to this system: "Precompute what is static, stream what changes, and combine them at query time: tiles on a CDN, a graph whose shape is preprocessed once, and weights that are customized every few seconds from an ordered stream."
Well-Architected Review Sheet
| Pillar | Question you'll hear | One-sentence answer | Round | Backed by |
|---|---|---|---|---|
| Reliability | "What if a routing server dies?" (REL 11) | Stateless servers in 3 AZs; a replacement loads the graph and warms up before taking traffic. | 1 | R1.9 |
| "What if the traffic feed stops or goes bad?" (REL 5) | Deltas are validated and applied in order; stale segments fall back to historical profiles, and the fleet to a historical weight set. | 2 | Step 2.3, R2.8 | |
| "What if a region fails?" (REL 13) | A warm standby of that continent's graph in another region takes over, scaled out in minutes, on historical weights. | 3 | R3.8 | |
| Performance | "How do you route in milliseconds?" (PERF 1) | Graph in RAM, preprocessed (CH, then MLD) so a query touches thousands of nodes, not millions. | 1–2 | Steps 1.4, 2.3 |
| "How fresh is traffic?" (PERF 3) | 23 seconds worst case, 28 in dense cities, proved stage by stage. | 2 | Step 2.3 | |
| Cost | "What does it cost?" (COST 5) | About $5,400, $151,800 and $705,700 a month; the CDN is the biggest line in every round. | 1–3 | R1.7, R2.6, R3.6 |
| "Why not buy it?" (COST 11) | At a delivery company's scale buying is reasonable; at 100B+ requests a month it's millions more. | 3 | Step 3.6 | |
| Operations | "How do you ship a new map safely?" (OPS 6) | Validated releases, staged rollout through style.json, blue/green graphs, rollback by release number. | 1–3 | R1.6, R3.8 |
| "What do you page on?" (OPS 8) | Route P99, weight age, telemetry lag, match rate, reroute latency, tile hit ratio. | 2–3 | R3.9 | |
| Security | "How do you protect location data?" (SEC 7) | Classified as sensitive: rotating pseudonymous IDs, no raw archive, 30-day observations, kept in its region. | 2–3 | Step 2.1, R3.5 |
| Sustainability | "Where is the waste?" (SUS 4) | Not storing raw pings; diffs instead of full pack downloads; one structure for three modes. | 2–3 | Steps 2.1, 3.1, 3.2 |
Rubric Across Levels
| Dimension | L5 (Round 1) | L6 (Round 2) | L7 (Round 3) |
|---|---|---|---|
| Maps | Vector z/x/y tiles on a CDN with immutable release URLs. | A short-lived traffic layer that references base-tile segments; CDN cost by tier. | Offline packs with diffs; CDN priced per edge region. |
| Routing | Graph in RAM; Dijkstra → A* → CH, with a worked example; turns in the graph. | Why CH breaks with live weights; CCH vs MLD; ordered deltas with snapshots. | Geographic partitions stitched through the top level; modes as weight sets; RAPTOR for transit. |
| Traffic | Not yet: typical speeds, and says so. | Streams sized in shards; HMM map matching; a proved 30-second budget; ETAs by arrival time. | Per-region pipelines; closures as a global live overlay. |
| Correctness | Legal routes; nearest edge, not node. | Validated weights; counts correct in both directions; conditional reroute writes. | Provenance and review for map data; one writer per closure; server clocks, not database clocks. |
| Cost | Knows the CDN is the bill, not the servers. | Rejects per-request services for pings with arithmetic; knows where each dollar goes. | Build vs buy per scale, including team cost and data rights. |
| Evolving under new scope | Builds from "render and SQL" one problem at a time. | Opens with "what breaks" and fixes wrong routes before freshness. | Keeps each continent independent while making the planet feel like one map. |