Uber: Real-Time Dispatch, Ringpop and Schemaless

This page is one interview loop in three rounds, built on a real company's public engineering history. All three rounds design the same system: the part of Uber that matches riders with drivers in real time, and the part that stores every trip. Each round is an era. In each era, rapid growth forced Uber to rebuild one of those two parts while the business kept running.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: Why Uber?

Rebuilding the Plane While Flying It

Imagine a taxi company whose number of rides doubles every four months. Every piece of software you wrote last quarter is now running at twice the load it was designed for. You can't stop the business to rebuild: people are standing on street corners waiting for a car right now. That was Uber from about 2012 to 2016.

Two parts of the system felt the pressure most:

• Dispatch. The live marketplace. Drivers' phones report where they are every few seconds; riders ask for a car; dispatch picks a driver and sends an offer. Its data is fleeting: a driver's position from ten seconds ago is already useless.
• The trip store. The record of every trip: who, where, when, how much. Its data is permanent and valuable: billing, driver pay, support and fraud checks all read it. Losing a trip means losing money or trust.

Synthesizing vector architecture diagram...

Read it as three eras. Each rebuild answered a limit of the era before it: one database and one process (Round 1), then an eventually consistent, peer-to-peer design that was hard to reason about (Round 2).

A few words we'll use all page:

What Makes It Hard

• Growth outruns designs. Trips grew about 20% a month in early 2014 (Uber, 2015): that's 1.2^12 ≈ 8.9× in a year. A design with "a year of headroom" has about four months.
• Dispatch is live and heavy. Every online driver writes a position every 4 seconds, and every rider with the app open reads nearby cars. Reads and writes both scale with the fleet.
• Trip data can never be lost, and it is never finished. A trip gets its fare at the end, then billing attempts, then maybe a fare adjustment or a rating days later. Each arrives on its own schedule.
• Cities are wildly uneven. Uber's 2015 dispatch talk described its first, city-sharded design as getting harder to manage as cities were added, because some cities are big, some small, and only some have sharp load spikes.

The Question the Whole Loop Answers

How do we scale a live marketplace's dispatch and data layers step by step, without ever stopping the rides?

The answer grows every round:

• Round 1: split the one dispatch process by city, get the trip data off the shared database, and start breaking up the monolith.
• Round 2: shard dispatch by map cell over a self-organizing gossip ring (Ringpop), and move trips to an append-only store of immutable cells on sharded MySQL (Schemaless), with triggers that feed billing and analytics from the store's own log.
• Round 3: agree on one hexagonal grid (H3), give the stores transactions (Docstore, and Cloud Spanner for fulfillment), and plan for losing a whole site.

---

Round 1 · Mid-level · "Era 1: A Monolith and One Database"

~35 min · SDE II (L5) · about 2010–2014 · 20,000 drivers online at the peak and 300,000 trips a day (planning assumptions) · trips growing about 20% a month (published) · keep matching and keep every trip

R1.1 Establish Design Scope

The interviewer sets the scene: "It's early 2014. We started as a black-car service in San Francisco and now run in dozens of cities. There's a Python app with most of the business logic, a dispatch service that keeps the drivers in memory, and one PostgreSQL database that holds almost everything, trips included. Our trips grow about 20% a month. Keep it running, and tell us what to change first."

Out of scope for this round: sharding the trip store itself (Round 2), pooled rides, surge pricing, multi-region.

R1.2 Functional Requirements, Derived Step by Step

Not yet: partitioned dispatch beyond one process per city, a sharded trip store, asynchronous downstream processing.

R1.3 Non-Functional Requirements: the Questions

We name each quality first; the numbers come in R1.7.

• Matching latency. How long from "request" to "a driver has an offer"? Riders are staring at a spinner.
• Freshness of positions. A position older than a couple of ping intervals is misleading.
• Write durability for trips. An acknowledged trip must survive a database crash; a completed trip that isn't stored can't be billed.
• Write availability. Uber (2016): "We needed write availability." When a write to PostgreSQL failed, the trip was parked in Redis to retry later.
• Growth per city. Each city grows at its own pace; one big city must not take down the others.
• Safe change. Adding a column, an index or a feature must not stop the business.

R1.4 The API

This API is illustrative: Uber's internal APIs of this era aren't public, so the paths and fields below are ours.

A driver's location ping (over a persistent connection from the driver app; shown as JSON):

jsonCopy
{
  "type": "location",
  "driver_id": "d_2044",
  "lat": 37.77493,
  "lng": -122.41942,
  "heading_deg": 81,
  "speed_mps": 6.2,
  "accuracy_m": 8,
  "status": "AVAILABLE",
  "sent_at_ms": 1398350000000,
  "seq": 18221
}

• seq is a per-driver counter the phone increments with every ping, so dispatch can drop a ping that arrives after a newer one.
• sent_at_ms is the phone's clock. We use it to judge age, not to order pings (phones' clocks drift).

Request a ride

httpCopy
POST /v1/trips HTTP/1.1
Authorization: Bearer <rider session token>
Content-Type: application/json
Idempotency-Key: 5d3c1f7e-2b8a-4c55-9e0a-7a1d2f9c3b61

{
  "product": "UBERX",
  "pickup": { "lat": 37.77493, "lng": -122.41942 },
  "dropoff": { "lat": 37.78917, "lng": -122.40145 }
}

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{ "trip_id": "t_7f3a91", "status": "REQUESTED", "poll_after_ms": 2000 }

• Idempotency-Key lets the rider app retry after a timeout without creating a second trip. The server stores the key with the trip and returns the same trip for a repeat.

Status codes

The trip's states

Synthesizing vector architecture diagram...

A trip moves forward only along these arrows. The loop between REQUESTED and OFFERED is dispatch trying the next driver. Everything after COMPLETED happens after the ride, on its own schedule.

Recap

• Two very different traffic types: frequent, tiny, disposable pings; and rare, valuable trip changes.
• The rider's request is idempotent; pings carry a sequence number.

R1.5 Design Evolution: From One Process and One Database to Split Responsibilities

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

Synthesizing vector architecture diagram...

Two programs and one database. The dispatch process keeps every driver's position in memory and scans all of them for each request; the monolith writes every trip change into the one PostgreSQL.

Uber published this shape: "The early architecture of Uber consisted of a monolithic backend application written in Python that used Postgres for data persistence" (2016), and its 2015 dispatch talk describes dispatch as written almost entirely in Node.js.

Step 1.1: One Dispatch Process for Every City

Loop: Design a Ride-Sharing Dispatch Service (Round 1: pings, offers, conditional accepts and timeouts for one city, in depth) · Loop: Design a Proximity Service (grid cells and radius search)

Step 1.2: Trip Writes Are Filling the One Database

Primitive: Database Sharding and Partition Keys · Loop primitive: Write-Ahead Log, fsync & Group Commit (Part 9 on the WAL in B-tree engines, and Part 10 on replication as a durability level)

Step 1.3: Every Deploy Is a Deploy of Everything

Primitive: Circuit Breaker, Bulkhead and Fault Tolerance

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

Each city's drivers live in that city's dispatch process. All trip reads and writes pass through one library, so the store behind it can change. The Redis buffer keeps a trip when PostgreSQL rejects the write, at the price of that trip being invisible to billing until it's replayed.

Trace: a ride request in this era (our design around the published shape)

Synthesizing vector architecture diagram...

Every trip change is an in-place UPDATE of the same PostgreSQL row, which is why the trips table's write load grows faster than its row count. The post-trip work runs as one chain: if any step fails, the whole chain is retried.

Our AWS translation (not Uber's setup). Dispatch processes and the Python services on EC2, behind a load balancer; the one database as an RDS for PostgreSQL instance with a Multi-AZ standby in another AZ; the Redis buffer as a small ElastiCache cluster.

Sources for this round

• Schmidt, Project Mezzanine: The Great Migration, Uber blog, dated July 2015: ~20% monthly trip growth; storage volume and IOPS by the end of 2014; one PostgreSQL; trips the largest, fastest-growing and most IOPS; close to 100 services by early 2014; downtime for new columns and indexes; UUIDs; the tripstore switch; mirrored writes and validation.
• Thomsen, Designing Schemaless, Uber Engineering's Scalable Datastore Using MySQL, Uber blog, January 2016: "running out of database space" in early 2014; the five requirements; the Redis buffer; the synchronous post-trip processing.
• Klitzke, Why Uber Engineering Switched from Postgres to MySQL, Uber blog, July 2016: the Python monolith on Postgres; write amplification; verbose replication to the East Coast; the 9.2 replica corruption; hours-long upgrades.
• Haddad, Service-Oriented Architecture: Scaling the Uber Engineering Codebase As We Grow, Uber blog, September 2015: one offering in one city; deploying everything at once; 500+ services; "a distributed monolithic API".
• Ranney, Scaling Uber's Real-time Market Platform, QCon London, March 2015 (summarized by High Scalability): dispatch in Node.js; the original dispatch sharded by city; the old in-memory index in "a handful of processes"; pings every 4 seconds.
• Lozinski, How Ringpop from Uber Engineering Helps Distribute Your Application, Uber blog, February 2016: the earlier system that "had to search through every single car".

Everything else in this round (the grid inside each city, the API shapes, the service cut, the trace details and every number in R1.7 not marked published) is our design or our assumption.

R1.7 Numbers

Figures marked published come from the sources above; the rest are assumptions (January 2014 planning figures) or derived from them.

Dispatch load at the evening peak

Trips

The runway (45 GB of trip data in January 2014: 300,000 trips × 30 days × 5 KB per trip including index entries and dead row versions, assumption; 1,200 GB free)

The disk fills about 9.3 months in: early November 2014. By then the peak row updates have grown 1.2^9 ≈ 5.2×, to about 537/s, and page changes to about 4,830/s. Uber's trip store moved to Schemaless about a month before Halloween 2014 (published).

Synthesizing vector architecture diagram...

Compound growth hides the wall until late: more than half of the free space goes in the last three months before it fills.

R1.8 Trade-Offs

A monolith vs services, at this stage

When to shard

Synchronous standby or not? A synchronous PostgreSQL standby means a commit waits for the local WAL flush plus a round trip to the standby and its WAL write. With a standby on the other coast (an assumed 70 ms round trip), every trip update waits at least 70 ms longer. A standby in the same data center is cheap and protects against a lost disk, but not a lost site.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Separates fleeting location data (memory) from durable trip data (a database).
• Sizes the dispatch load and shows that CPU, not memory, is the limit, then fixes it with city sharding and a grid.
• Knows that replicas don't add write capacity.
• Computes a runway under compound growth and starts the migration in time.
• Puts all trip access behind one interface before changing the store.
• Splits the monolith along flows that fail differently.

Follow-up questions

1. "Why not keep trips in PostgreSQL and partition the table?" Answer: partitioning one table on one server splits the files, not the machine: disk, IOPS and the single writer stay the same. Uber's requirement was to "linearly add capacity by adding more servers", which needs shards on many servers.

2. "A trip starts in one city and ends in another. Which dispatch process owns it?" Answer: the pickup city's, for the whole trip. Ownership that moves mid-trip creates a window with two owners or none. The dropoff city only matters for pricing rules.

3. "The Redis buffer: what happens if Redis loses the trip too?" Answer: the trip is gone, which is why Uber wanted a store with write availability built in. Round 2's buffered writes put the backup copy in a second MySQL cluster, on disk.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Era 2: Dispatch on a Gossip Ring, Trips on Schemaless"

~40 min · Senior SDE (L6) · about 2014–2018 · "many millions of trips per day" on six continents (published, 2016) · ~5.5M trips a day and 1M drivers online at the peak (planning figures) · 250,000 pings/s · no single database limit, capacity added without downtime

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "Uber began as a Python monolith on one PostgreSQL, with a Node.js dispatch service that kept every online driver in memory. Drivers ping every 4 seconds. Scanning every driver for every request burned CPU in a single-threaded process, so dispatch was sharded by city, and inside a city drivers belong in a grid, not a list. Trips grew about 20% a month. Trip data was the biggest, fastest-growing and most I/O-heavy table in the shared database, and PostgreSQL's update model writes a new row version and new index entries for most updates, so the disk and IOPS would run out by late 2014. Replicas don't help writes. So all trip access went behind one tripstore library, trip IDs became UUIDs, failed writes were parked in Redis, and the monolith started splitting into services. Open costs: cities are uneven and each is still one process; the trip store still has to be sharded; and post-trip work runs as one synchronous chain."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: dispatch split by city, trips behind one library, still one database underneath.

Round 1 step summary

Open costs: big cities outgrow one process; the trip store is still one database; downstream work (billing, analytics) is synchronous and fragile.

R2.1 The Scope Raise

Interviewer: "It's late 2014 and growth hasn't slowed. We're in hundreds of cities on six continents. A city-to-process table that people edit by hand doesn't scale, and a big city still fits in one process only until next quarter. Dispatch must share work across many machines and survive losing some of them, without a central coordinator we'd have to keep alive. The trip database runs out of room in months. And billing, analytics and our other data centers need every change to every trip, without losing one."

Scope change

The "Not yet" list from R1.2 comes back: partitioned dispatch and a sharded trip store are in scope now.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

The Schemaless API (Uber published these calls; the HTTP shapes below are ours). Uber described two kinds of access: random access to cells, and log-style access by shard.

httpCopy
PUT /v1/instances/mezzanine/cells HTTP/1.1
Content-Type: application/json

{
  "row_key": "8f1c2d9e-4b7a-4e21-9c1d-5a6b7c8d9e0f",
  "column": "STATUS",
  "ref_key": 2,
  "body": { "attempt": 2, "is_completed": true, "processor_ref": "ch_91ab" }
}

httpCopy
HTTP/1.1 409 Conflict
Content-Type: application/json

{ "error": "CELL_EXISTS", "row_key": "8f1c2d9e-4b7a-4e21-9c1d-5a6b7c8d9e0f", "column": "STATUS", "ref_key": 2 }

• A 409 for an existing cell is how a writer learns it lost a race or is repeating itself. For a retry of the same write, the client treats it as success.
• A write accepted while the shard's master is down returns success with a flag saying it isn't readable yet (Uber: "Schemaless can tell the client if the master is down"). We show it as 202 Accepted.

Trigger subscriptions (the concept is published; this config shape is ours). Uber: "Client programs link into the Schemaless triggers framework by configuring which Schemaless instance and which columns to poll data from."

yamlCopy
trigger_client: billing
instance: mezzanine
columns: [BASE]
workers: 64                 # at most one worker per shard; 4,096 shards
offset_store: zookeeper     # the added ID per shard; Uber: ZooKeeper or the instance itself
poison_policy:
  max_attempts: 5           # then move the cell to a retry queue and continue
  stop_after_marked: 100    # too many marked cells means a systematic bug: stop and page

A secondary index (the concept is published: Uber's example is an index of a driver's trips, sharded on the driver's UUID, with the trip's city and creation time copied in; this definition is ours):

yamlCopy
index: driver_trips
column: BASE
shard_field: driver_uuid      # must be immutable and well spread: UUIDs are best
fields:
  - driver_uuid
  - city_uuid                 # denormalized from the cell
  - trip_created_at           # denormalized from the cell
alias: driver_trips_v3        # a new version is backfilled, then the alias switches

Recap

• Writes are inserts of immutable cells; there is no update call.
• Reading "the current state" means reading the latest ref key.
• Downstream systems read the store's own ordered log, one shard at a time.

R2.4 Design Evolution: A Ring for Dispatch, Cells for Trips

Step 2.1: Which Machine Owns This Part of the Map?

Primitive: Consistent Hashing · Primitive: Geospatial: Geohash, Quadtree and S2 · Drill: The Ring Rebalance That Crushed Node 07 (answered here: why adding 4 nodes doesn't shed 25% each without replica points, and why bounded loads are the wrong tool for stateful owners) · Loop: Design a Ride-Sharing Dispatch Service (Round 2: 500,000 location writes a second and batch matching)

Step 2.2: A Machine Dies, or Only Looks Dead

Primitive: Gossip Protocol and Failure Detection · Drill: The GC Pause That Declared the Cluster Dead (answered here: why a fixed threshold flaps, and gossip vs a central coordinator)

Step 2.3: Trips Outgrow One Database

Primitive: Database Sharding and Partition Keys · Loop: Design a Distributed Key-Value Store (partitioning and replication from the inside)

How the move happened (published). Mezzanine ran in stages: trip IDs to UUIDs; a column layout for trips; a backfill from PostgreSQL; mirrored writes to both stores; rewriting every query; and "validation, validation, validation, and validation!", replaying queries against Schemaless and comparing, sampled to limit load on the busy PostgreSQL. The switch, about a month before Halloween 2014, "was a non-event". One rule makes a backfill and mirrored writes converge (our reading): both must write each trip version under the same ref key. Then whichever arrives second is rejected as a duplicate. A backfill that picked "latest + 1" instead would write an old version as the newest and hide the mirrored update, which is the append-only version of "backfills keep original write timestamps".

Step 2.4: A Master Dies Before It Replicates

Loop primitive: Write-Ahead Log, fsync & Group Commit (Part 1: what "acknowledged" has to mean; Part 10: replication as a durability level) · Loop primitive: Idempotency & Effectively-Once Processing (Part 4: the late original; Part 5: idempotent by design)

Step 2.5: Billing and Analytics Need Every Change

Primitive: Change Data Capture and the Outbox Pattern · Loop primitive: Change Streams & the Transactional Outbox (Part 1: why writing twice fails; Part 5: reading the database's own log; Part 6: poison records and repair) · Loop primitive: Idempotency & Effectively-Once Processing (Part 6: queues and logs; Part 7: side effects you can't take back) · Drill: The Dual-Write That Broke Search Consistency (answered here: the dual write in the wrong answers, and polling vs log tailing above)

Step 2.6: Find a Driver's Trips

Primitive: Database Sharding and Partition Keys (global secondary indexes)

Step 2.7: Two Owners for One Trip

Loop primitive: Leases, Fencing Tokens & Distributed Locks (Part 4: the pause, the zombie and the fencing token; Part 5: conditional writes are the check) · Primitive: Distributed Locks and Leases · Drill: The GC Pause That Corrupted Shared Storage (answered here: why a TTL lock doesn't survive a pause, and version checks vs an external lock) · Loop: Design a Ride-Sharing Dispatch Service (Round 1, step 1.3: the conditional accept written out in full)

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Live dispatch state sits on Ringpop-sharded services backed by Cassandra; the permanent trip record goes to Schemaless when the trip ends, and everything downstream reads Schemaless's log through triggers.

Our AWS translation (not Uber's setup). Ringpop processes on EC2 in one Auto Scaling group per AZ, behind a Network Load Balancer for driver connections (priced in R2.6). Each Schemaless storage cluster as an RDS for MySQL instance with two read replicas in the other AZs: RDS read replicas replicate asynchronously, like Uber's minions, so buffered writes are still needed. The worker tier and trigger workers on EC2; offsets in a small replicated store.

Trace: a membership change during a search

Synthesizing vector architecture diagram...

Losing an owner loses nothing permanent: its cells refill from pings within one ping interval after the ring moves them. The cost is about 12 to 14 seconds of partial results for 0.3% of cells.

Sources for this round

• Thomsen, Designing Schemaless, January 2016: requirements; alternatives and "operational trust"; the data model; the trip columns; triggers; indexes (shard field, denormalization, "well below 20ms").
• Thomsen, The Architecture of Schemaless, January 2016: worker and storage nodes; 4,096 shards; one master and two minions across data centers; asynchronous replication; reads from the master by default; circuit breakers; buffered writes; idempotent writes; the entity table; MessagePack and zlib.
• Thomsen, Using Triggers On Schemaless, January 2016: the API; Schemaless as a partitioned log; added IDs identical on replicas; the triggers framework, leader, offsets, at-least-once, poison cells, scale.
• Schmidt, Project Mezzanine: The Great Migration, July 2015: 5 months to build; splitting servers in two; the migration stages; the switch in 2014.
• Nielsen and Johnsen, Code Migration in Production: Rewriting the Sharding Layer of Uber's Schemaless Datastore, February 2018: 40+ instances and many thousands of storage nodes in 2016; reads about 90% of traffic; the Go rewrite's results.
• Lozinski, How Ringpop from Uber Engineering Helps Distribute Your Application, February 2016: Ringpop's parts; Geospatial as its first use; many millions of trips a day on six continents.
• Uber, ringpop-go and ringpop-node source code (checked September 2026): 100 replica points, SWIM timeouts, suspect and faulty durations, flap-damping defaults, suspect members still reachable.
• Ranney, Scaling Uber's Real-time Market Platform, QCon London, March 2015: supply, demand and DISCO; S2 level 12 as the shard key; 1M writes/s design goal; hundreds of processes; Ringpop as AP; thousands of nodes; TChannel.
• Neerabail, Medisetty, Thangavelu and others, Uber's Fulfillment Platform: Ground-up Re-architecture, July 2021: the 2014 fulfillment stack (rt-demand, rt-supply, serial queues, Cassandra through a storage gateway, sagas) and its split-brain overwrites.
• Reuters, Uber reaches 2 billion rides six months after hitting its first billion, July 2016, reporting the CEO's announcement.
• MySQL 8.4 Reference Manual, Semisynchronous Replication; S2 Geometry, S2 Cell Statistics.

Uber hasn't published its trigger poll interval, its number of storage clusters, its peak factors, or how it handled index gaps, added-ID reuse after promotion, or fencing in the fulfillment stack; those parts of this round are ours.

R2.6 Numbers and Cost

All figures are assumptions unless marked published or derived.

Trips and the trip store

The trip store's write rate is modest. What forced sharding was volume, I/O on one machine, schema changes and write availability.

Storage

Doubling capacity means going from 32 to 64 clusters: each old cluster hands 64 of its 128 shards to a new one, "splitting each MySQL server in two".

Dispatch

Tail latency from pauses (Node.js garbage collection, assumptions). Suppose each process spends 0.5% of its time in 100 ms pauses. A request that lands in a pause waits anywhere from 0 to 100 ms, evenly spread. A search touches 8 processes (the one it lands on, plus 7 owners), so about 1 − 0.995⁸ ≈ 3.9% of searches meet a pause:

(For the search's p99: the slowest 1% of all searches are the slowest quarter of the 3.9% that met a pause, which waited at least three-quarters of 100 ms.) Uber's answer, built into TChannel (published, 2015): backup requests with cross-server cancellation. Send the read to one replica, and if it hasn't answered after a short delay, send it to a second one; whichever answers first cancels the other.

The connection tier (AWS translation, us-east-1 list prices; 30-day month = 720 hours, used on every cost on this page). One Network Load Balancer with a TCP listener carries 1M persistent driver connections; TLS ends on our fleet.

Billed on the largest dimension: 360 × $0.006 = $2.16 an hour × 720 = $1,555 a month, plus $0.0225 × 720 = $16 for the load balancer hour: ≈ $1,571 a month. (A TLS listener would count only 3,000 active connections per NLCU: 1M ÷ 3,000 = 334 NLCUs by connections, but the bill stays at 360 because bytes still dominate, and the TLS work moves to AWS.)

Cross-AZ transfer for pings (translation; $0.01/GB each way, so $0.02 for every GB that crosses)

The replication is what buys surviving an AZ loss without waiting 4 s for pings to refill. Two copies instead of three halves it; skipping replication entirely is also defensible for data that refills itself in 4 seconds.

Trigger polling (from step 2.5)

R2.7 Trade-Offs

Gossip vs a central coordinator for ownership

Append-only cells vs update-in-place rows

Build our own store vs adopt one. Uber's reason to build was "operational trust": a 3 am page on the trip store must be something the team can fix, and it knew MySQL. The thin layer took about 5 months. The long-term costs were an in-house system to staff and a "restrictive API" (Uber, 2021). Uber also noted, in 2016, that it had since "adopted both Cassandra and Riak with success in other areas". The honest rule: build when no product meets a hard requirement and you can staff it for years.

Triggers vs a Kafka stream (a translation question). Kafka (Amazon MSK on AWS) gives consumer groups, retention and replay for free, but it's a second copy of every change and a dual write unless it's fed from the database log. If we did use MSK, we'd set unclean.leader.election.enable to false (MSK's default is to allow unclean leader election, which can lose acknowledged messages, unless tiered storage is on, which requires it to be false), produce with acks=all and set min.insync.replicas to 2 with 3 replicas.

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Explains consistent hashing with replica points, and why neither vnodes nor bounded loads fix one hot key.
• Walks through SWIM's ping, ping-req, suspicion and incarnation numbers, with a timing chain.
• Designs an append-only cell store and says what it costs (latest-version reads, growth, no transactions).
• Knows asynchronous replication loses acknowledged writes, and how buffered writes and idempotent cells close that gap.
• Builds a change feed from the store's own log, with at-least-once handlers and a poison-cell path; compares polling with log tailing.
• Fences split-brain writes at the store with versions, not in memory.

Follow-up questions

1. "Why did Uber shard dispatch by S2 cell instead of by city?" Answer: city shards are as uneven as cities, and a big city is one shard. Cells are small and many, so hashing spreads them evenly across hundreds of processes, and a big city simply uses more processes. The price is fan-out: a search touches several cells' owners.

2. "A trigger function charges a card, then crashes before recording the result. What happens?" Answer: the cell is delivered again (at least once). The function reads the latest STATUS from the master, sees no result, and charges again, unless the charge carries an idempotency key the processor recognizes. That key is what makes a side effect you can't take back safe to retry.

3. "Could Schemaless use semisynchronous replication instead of buffered writes?" Answer: it would protect acknowledged writes while a replica keeps up, but it falls back to asynchronous after the timeout (10 s by default), and it gives no write availability while the master is down. Buffered writes give both, at twice the inserts.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Era 3: Hexagons, a New Datastore and Surviving a Site Loss"

~45 min · Principal (L7) · about 2018–today · "40 million+ trips per day" and 9.7M monthly drivers and couriers (published, Q4 2025) · 3M online at the global peak and 750,000 pings/s (planning figures) · one grid, transactions, survive losing a site

R3.0 Where We Left Off

Round 2 in 60 seconds. "Dispatch was sharded by S2 map cell on Ringpop: every process embeds a SWIM membership protocol and a consistent hash ring with 100 points per node, and any process can take a request and forward it to the key's owner. Failure detection uses ping, ping-req through 3 others and a 5-second suspicion window with incarnation numbers, so pauses don't evict nodes, but a real crash takes about 13 seconds to act on and every deploy is a membership change. Live trip and driver state sat in Ringpop-owned services with serial queues, stored in Cassandra, with sagas for multi-entity changes; split brain during deploys and failovers let last-write-wins overwrite state. Trips moved to Schemaless: immutable JSON cells keyed by row, column and version on 4,096 MySQL shards, with buffered writes on a random second cluster so a master failure loses nothing acknowledged, triggers that read each shard's log in added-ID order at least once, and sharded, eventually consistent indexes. Open costs: the ring's size caps how big a city can get; there are no transactions across entities; every team draws its own map grid; and a whole site can still go down."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: an AP ring for live state, an append-only store for the record, and a log-driven pipeline downstream.

Round 2 step summary

Open costs: ring size limits, split-brain overwrites, sagas instead of transactions, many map grids, and one site's failure still stranding its cities.

R3.1 The Scope Raise

Interviewer: "It's 2019. We're not just rides any more: food delivery with a restaurant in the middle, freight, reservations, airport queues, a bus driver with a dozen passengers. Every product team has drawn its own map zones. Our fulfillment stack is an AP ring with sagas, and debugging it is a nightmare. Our trip store can't do transactions. And we run on our own data centers and two failure domains; losing one must not stop the marketplace."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

A shared grid library (H3; function names from the open-source library).

One transaction for "driver accepts an offer" (a design in the style of Uber's fulfillment platform; SQL shown for clarity, not Uber's schema).

sqlCopy
BEGIN;
  SELECT owner_epoch FROM city_owners WHERE city_id = 'san_francisco' FOR UPDATE;  -- must be 41, or abort
  -- the offer must still be open and meant for this driver
  UPDATE offers   SET state = 'ACCEPTED'
   WHERE offer_id = 'o_55' AND state = 'OPEN' AND driver_id = 'd_2044';
  -- each UPDATE must touch exactly 1 row, or abort
  UPDATE trips    SET state = 'ACCEPTED', driver_id = 'd_2044', version = version + 1
   WHERE trip_id = 't_7f3a91' AND state = 'OFFERED';
  UPDATE supplies SET active_jobs = active_jobs + 1, version = version + 1
   WHERE driver_id = 'd_2044' AND active_jobs < max_jobs;
  -- post-commit work, committed atomically with the state change
  INSERT INTO late_tasks (shard_id, task_id, kind, payload)
  VALUES (17, 'k_91', 'NOTIFY_RIDER', '{"trip_id":"t_7f3a91"}');
COMMIT;

All three entities change together or not at all, and the notification is recorded in the same commit, so it can't be lost or sent for a change that didn't happen.

Site ownership controls (our design).

yamlCopy
city: san_francisco
home_region: region_a
standby_region: region_b
owner_epoch: 41            # bumped on every move; every write for this city carries it
failover:
  detection: outside_probes   # health checks run from outside both regions
  approval: on_call           # a human confirms before cities move
data_residency: us_only       # the standby must satisfy the same rules

R3.4 Design Evolution: One Grid, Transactions, and a Plan for Losing a Site

Step 3.1: Every Team Has Its Own Map Zones

Primitive: Geospatial: Geohash, Quadtree and S2 · Loop: Design a Proximity Service (Round 2: cell sizes, dense and sparse areas) · Loop: Design a Ride-Sharing Dispatch Service (Round 2, step 2.5: surge by zone)

Step 3.2: The Record Store Needs Transactions

Primitive: Distributed Consensus: Raft and Paxos · Loop primitive: LSM-Trees & Compaction (MyRocks is an LSM engine under MySQL; Part 7 compares B-trees and LSM trees) · Drill: The Network Partition That Elected Two Leaders (answered here: what a 2-node minority can commit, and 3 nodes vs 5)

Step 3.3: Live Dispatch State Needs Transactions Across Regions

Primitive: Two-Phase Commit and Saga Orchestration · Primitive: Database Isolation Levels, ACID and Concurrency Anomalies · Loop primitive: Change Streams & the Transactional Outbox (Part 2: the outbox in one transaction; Part 3: the relay and at-least-once delivery) · Loop: Transactional Outbox and Event-Driven Ledger

Step 3.4: A Whole Site Goes Down Mid-Trip

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active · Loop primitive: Leases, Fencing Tokens & Distributed Locks (Part 7: leader election and single writers) · Loop: Design a Ride-Sharing Dispatch Service (Round 3, step 3.6: phones resyncing trips after a region loss, in detail) · Drill: The Booking That Existed in Frankfurt but Not in Virginia (answered here: the double assignment from replication lag, and why not everything synchronous)

Step 3.5: Our Own Data Centers or the Cloud?

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each region runs everything for the cities it owns. Live, contended state commits synchronously across regions; the record store replicates Raft groups within a region and copies across regions; failover decisions and city ownership live outside both.

Our AWS translation (not Uber's setup). Two US Regions, us-east-1 and us-west-2, three AZs each. Dispatch and marketplace services on EC2 in Auto Scaling groups per AZ, behind Network Load Balancers. Live fulfillment state in an Aurora DSQL multi-Region cluster (the two Regions plus a witness Region). The record store as MySQL: RDS for MySQL Multi-AZ DB clusters (a writer and two readable standbys in three AZs, replicating semisynchronously), with our own sharding layer. City ownership in Route 53 Application Recovery Controller routing controls, with Route 53 health checks probing from outside. CloudWatch for alarms.

Trace: a driver accepts an offer

Synthesizing vector architecture diagram...

The accept, the trip, the driver's plan and the notification task commit as one. The notification may be sent twice (at least once), never zero times.

Trace: region A fails

Synthesizing vector architecture diagram...

The store already had every acknowledged change, so the digest mostly confirms what B knows. The epoch keeps a recovering region A from writing over B.

Sources for this round

• Brodsky, H3: Uber's Hexagonal Hierarchical Spatial Index, Uber blog, June 2018: why a grid; surge in hexagons; one neighbor distance; icosahedron, 122 base cells, 12 pentagons; 16 resolutions, one-seventh area; approximate containment.
• H3 documentation, Tables of Cell Statistics Across Resolutions (checked September 2026): average areas and edge lengths.
• Chatterjee, Chaudhary and Tariq, Evolving Schemaless into a Distributed SQL Database, Uber blog, February 2021: Docstore's motivation, layers, partitions of 3–5 nodes, Raft, strict serializability per partition, CP, MySQL transactions.
• Uber, MySQL to MyRocks Migration in Uber's Distributed Datastores, Uber blog, September 2022: shared topology, Raft within a region, replication across regions, tens of petabytes and tens of millions of requests a second, MyRocks since 2019.
• Pozniansky and others, How Uber Serves Over 40 Million Reads Per Second from Online Storage Using an Integrated Cache, Uber blog, February 2024: one leader and two followers; Flux CDC.
• Neerabail, Medisetty, Thangavelu and others, Uber's Fulfillment Platform: Ground-up Re-architecture, Uber blog, July 2021: scale; the old stack's problems; requirements; options considered; Spanner nam3; LATE.
• He, Medisetty and others, Building Uber's Fulfillment Platform for Planet-Scale using Google Cloud Spanner, Uber blog, September 2021: 20 drivers per core; evaluated databases; data model; LATE tailers; hybrid networking; cost split; snapshot then delete.
• Uber, Adopting Arm at Scale: Bootstrapping Infrastructure, Uber blog, February 2025: the February 2023 move to Oracle Cloud Infrastructure and Google Cloud.
• Uber, Fourth Quarter and Full Year 2025 Results and prepared remarks, February 2026: 3,751 million trips in the quarter; "40 million+ trips per day"; 9.7 million monthly drivers and couriers; 202 million monthly active platform consumers.
• Ranney, Scaling Uber's Real-time Market Platform, QCon London, March 2015: the state digest on drivers' phones.
• AWS documentation: Aurora DSQL, its resilience and quotas (checked September 2026).

Uber hasn't published its fleet sizes, its failover times, whether Docstore's cross-region replication is synchronous, or how state digests are versioned; those parts of this round, the site ownership controls and the whole AWS translation are ours.

R3.6 Numbers and Cost

Figures marked published come from the sources above; everything else is our assumption or derived from one.

Scale today (published, Q4 2025)

Why the fulfillment rewrite paid for itself (efficiency)

Capacity per site (our design). Each of two regions must carry 100% of the load if the other fails: 15,000 cores per region, 5,000 per AZ in three AZs, 30,000 in all, half of it idle-ready at any moment. On Arm instances one vCPU is one core, so each region needs a Running On-Demand Standard instances quota of at least 15,000 vCPUs, plus about 10% for the extra instances a rolling deploy adds: 16,500 vCPUs per Region, checked in both Regions before we need them.

H3 (from step 3.1)

Docstore commits (from step 3.2; assumed 1 ms between zones). A 3-node partition commits when the leader and one follower have the transaction: about one cross-zone round trip plus a log flush, a millisecond or two. A 5-node partition waits for the second-fastest of four followers.

Region failover (from step 3.4)

Transfer costs in the AWS translation (list prices; $0.02 per GB across AZs, $0.02 per GB between us-east-1 and us-west-2; 30-day month)

The expensive bytes are the fleeting ones. The permanent record is cheap to copy anywhere; the ping stream is what a design must keep inside a zone where it can.

Published cost shape of the Spanner move: nodes about 80% of the cost, network transfer about 20% (because the applications ran on-premises), and multi-region configurations 3–4× the price of single-region ones. Storage was "trivial" because only live data stays.

R3.7 Trade-Offs

Hexagons vs squares

Build vs buy datastores, over the whole loop

The pattern: build where the team's operational knowledge is the advantage and the workload is Uber-specific; buy where the hard part (global consensus, TrueTime) is someone else's core business.

An AP ring vs a CP transactional store for live state

Active-active vs active-passive. Active-active keeps both regions warm and proven every day, at the price of standby capacity inside every region and an ownership protocol per city. Active-passive is cheaper to run and more likely to fail on the day it's used.

What changed from Round 1. Round 1 split one process and one database so each could grow. Round 2 made growth automatic by giving up consistency: gossip, last-write-wins, sagas and eventually consistent indexes. Round 3 bought consistency back where it matters (who has which trip, one transaction per accept) and kept the append-only, log-driven ideas where they fit (the record store, CDC, post-commit tasks). The lesson of the loop: at each scale, choose the weakest guarantee the business can live with, and be ready to buy stronger ones back when debugging the weak ones costs more than the stronger store.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6

Procedure: ring instability REL 11

1. Stop changing membership. Pause deploys and autoscaling for the ring's service.
2. Find the common cause. Are the flapping nodes in one zone, one rack, one deploy batch? Are they in long GC pauses?
3. Remove the sick, not the noisy. Take nodes with confirmed problems out deliberately, a few at a time, letting the ring converge (checksums agree) between each.
4. Only then tune. If healthy nodes are being suspected, lengthen the suspicion window before shortening anything.

Procedure: shard or partition overload PERF 3

1. Confirm it's one partition: latency is high for one partition's keys, normal elsewhere.
2. Find the key: a hot trip, merchant or city? A key can't be split by adding nodes.
3. Shed or reshape: rate-limit the source; move other shards off that partition; for a structurally hot key, change its partition key.

Procedure: region evacuation REL 13 · OPS 10

1. Confirm from outside that the region, not a service, is down.
2. Check the survivor has capacity and quota for 100% of the peak, and may hold the affected cities' data.
3. Flip routing controls for the affected cities; confirm each city's epoch moved.
4. Watch resyncs and the first-offer latency in the moved cities.
5. Fence the failed region's services before it rejoins; move cities back one at a time.

Go deeper: CLI and SQL checks (AWS translation; replace names with real ones)

textCopy
# 1. Which targets behind the driver connection tier are unhealthy
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/driver-conn/abc123

# 2. Scale the dispatch group in one AZ
aws autoscaling set-desired-capacity --auto-scaling-group-name dispatch-use1-az1 --desired-capacity 120

# 3. Read, then flip, a region's routing control (run against a cluster endpoint)
aws route53-recovery-cluster get-routing-control-state --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/abc/routingcontrol/def --region us-west-2 --endpoint-url https://host-aaa.us-west-2.example.com/v1
aws route53-recovery-cluster update-routing-control-state --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/abc/routingcontrol/def --routing-control-state Off --region us-west-2 --endpoint-url https://host-aaa.us-west-2.example.com/v1

# 4. The vCPU quota for Running On-Demand Standard instances, in the survivor Region
aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A --region us-west-2

# 5. Alarms currently firing for dispatch
aws cloudwatch describe-alarms --state-value ALARM --alarm-name-prefix dispatch-

# 6. On a MySQL replica: how far behind is it?
SHOW REPLICA STATUS\G

# 7. On a MySQL master of the record store: how many buffered cells wait for replay?
SELECT COUNT(*) FROM buffered_cells;

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Chooses consistency per kind of data: synchronous for contended live state, asynchronous for the record and analytics, none for pings.
• Explains why a hexagonal grid suits a marketplace, and computes a k-ring.
• Compares an AP ring with sagas against a transactional store, using Uber's own published reasons.
• Sizes a Raft partition (3 vs 5) and says what a minority can do.
• Designs a region failover end to end: outside detection, a gated move, capacity and residency in the survivor, epochs, resync, and a timing chain checked against the error budget.
• Separates what Uber published from what it designs, and reads published numbers for what they imply.

Follow-up questions

1. "Why did Uber buy Spanner for fulfillment but build Docstore for everything else?" Answer: fulfillment needed multi-row, multi-table transactions across regions, the hardest thing a database does, and wanted "low operational overhead"; Spanner provides exactly that. Docstore serves many teams on Uber's own infrastructure, where Uber already had deep MySQL expertise and wanted control of cost and features. Different workloads, different make-or-buy answers.

2. "What's the RPO of your design if region A fails?" Answer: zero for live fulfillment state, which commits synchronously in both regions. For asynchronously replicated data, its replication delay at the moment of failure, typically seconds. Pings aren't replicated: they refill in 4 seconds.

3. "Could you make the failover fully automatic?" Answer: for unambiguous signals, yes: probes from two outside places and the surviving region all agree. The risk is a partition that makes a healthy region look dead; the epoch fence makes that safe for data, but not free for riders, who'd see a needless move. So automate the clear cases and page a human for the rest.

4. "Is Ringpop still how Uber dispatches?" Answer: Uber's 2021 posts describe the fulfillment platform moving off its Ringpop-based stack to Spanner, and the Node.js library's repository says it "is no longer under active development". The Go library, ringpop-go, is still on GitHub and is a dependency of Uber's open-source Cadence workflow engine. The ideas (a hash ring, SWIM, handle-or-forward) remain standard.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. For offers, accepts, batching and surge in a generic dispatch service, see the ride-sharing dispatch loop; for grids and radius search, the proximity loop; for another store that outgrew its first design, the Discord case study.

The Two Sentences That Matter Most

• Opening any round: "Pings are fleeting and huge, trips are permanent and valuable, so I'll keep positions in memory, sharded by map cell, and put trips in a store that grows by adding shards and never loses an acknowledged write."
• When scale arrives: "I'll pick the weakest guarantee each kind of data can live with: no replication for pings, an append-only log with at-least-once consumers for the trip record, and synchronous transactions only for who has which trip, with an epoch on every write so a recovering site can't overwrite the new owner."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels

Sources

All sources used on this page, oldest first.

• Ranney, Scaling Uber's Real-time Market Platform, QCon London, March 2015, and the High Scalability summary of that talk.
• Schmidt, Project Mezzanine: The Great Migration, Uber blog, July 2015.
• Haddad, Service-Oriented Architecture: Scaling the Uber Engineering Codebase As We Grow, Uber blog, September 2015.
• Thomsen, Designing Schemaless, The Architecture of Schemaless and Using Triggers On Schemaless, Uber blog, January 2016.
• Lozinski, How Ringpop from Uber Engineering Helps Distribute Your Application, Uber blog, February 2016.
• Reuters, Uber reaches 2 billion rides six months after hitting its first billion, July 2016.
• Klitzke, Why Uber Engineering Switched from Postgres to MySQL, Uber blog, July 2016.
• Nielsen and Johnsen, Rewriting the Sharding Layer of Uber's Schemaless Datastore, Uber blog, February 2018.
• Brodsky, H3: Uber's Hexagonal Hierarchical Spatial Index, Uber blog, June 2018.
• Chatterjee, Chaudhary and Tariq, Evolving Schemaless into a Distributed SQL Database, Uber blog, February 2021.
• Neerabail, Medisetty, Thangavelu and others, Uber's Fulfillment Platform: Ground-up Re-architecture, Uber blog, July 2021.
• He, Medisetty and others, Building Uber's Fulfillment Platform for Planet-Scale using Google Cloud Spanner, Uber blog, September 2021.
• Uber, MySQL to MyRocks Migration in Uber's Distributed Datastores, Uber blog, September 2022.
• Pozniansky and others, How Uber Serves Over 40 Million Reads Per Second from Online Storage Using an Integrated Cache, Uber blog, February 2024.
• Uber, Adopting Arm at Scale: Bootstrapping Infrastructure, Uber blog, February 2025.
• Uber, Fourth Quarter and Full Year 2025 Results, February 2026.
• Uber open source: ringpop-node, ringpop-go, H3 and Cadence (checked September 2026).
• H3, Tables of Cell Statistics Across Resolutions; S2 Geometry, S2 Cell Statistics (checked September 2026).
• MySQL 8.4 Reference Manual, Semisynchronous Replication.
• AWS, Aurora DSQL documentation, Elastic Load Balancing pricing and EC2 data transfer pricing (us-east-1, checked September 2026), for the translation's figures.