Design a Real-Time Gaming Leaderboard

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Leaderboard?

You Already Know One: the Arcade Scoreboard

Walk up to an old arcade cabinet and the attract screen shows ten names and ten scores. That is a leaderboard: players ranked by score, highest first. Every modern game has one, and it asks the store behind it three things:

What Makes It Hard

The top 10 is easy. Any database can sort and return ten rows. The hard question is the one every player actually asks: "what is my rank?" A rank means counting everyone above me. For the player in 4,812,331st place, that is 4,812,330 people to count, and their scores keep changing while we count. Ask that question a few thousand times a second and a normal database index can't keep up, because an index knows the order of rows but not how many rows sit before a given one.

The Question the Whole Loop Answers

How do we answer "what is my rank?" exactly and instantly, while scores keep changing?

The answer gets sharper every round:

• Round 1: use a data structure that keeps counts inside its order, a sorted set, and keep a durable copy so memory can be lost.
• Round 2: at esports scale, "exact" gets subtle: ties, season boundaries, cheaters, and 50,000 lobbies asking for the same top 100.
• Round 3: one sorted set per board stops being the right shape. We learn that exact ranks are only needed where money or pride is at stake, and approximate ranks are fine everywhere else.

---

Round 1 · Mid-level · "One Game's All-Time Leaderboard"

~35 min · SDE II (L5) · 1 region, 3 AZs · 1M players · ~500 submissions/s and ~2K reads/s peak · 99.9% · reads < 50 ms

R1.1 Establish Design Scope

The interviewer says: "Our mobile game needs a leaderboard. Design it." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Tie-breaking (two players with the same score).
• Seasons, multiple modes, and "players around me".
• Spectators watching the board live.

R1.2 Functional Requirements, Derived Step by Step

Not yet: ties, seasons, players around me, spectators.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Read latency. The rank shows on the results screen; the target is under 50 ms at our API.
• Write rate. How many submissions a second at the evening peak? That decides whether one machine is enough.
• Exactness. Is "about 4.8 millionth" good enough? For Round 1, no: the rank must be exact.
• Availability. 99.9%. The leaderboard is not the game, but an empty or wrong leaderboard gets noticed fast.
• Durability. If the in-memory board is lost, scores must not be. A player's best score is something they earned.

R1.4 The API

1. Submit a result. Only our game servers call this. The body is the match result the game server produced, and a signature over it (step 1.4 explains why).

httpCopy
POST /v1/games/g1/scores HTTP/1.1
Host: leaderboard.internal.example-games.com
Content-Type: application/json
X-Result-Signature: hmac-sha256=6f1c9a0e52b7d4...

{
  "match_id": "m_7f3k29",
  "player_id": "p48213377",
  "score": 50120,
  "ended_at": "2026-09-26T19:20:00.000Z",
  "server_id": "gs-use1-az2-0183"
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "player_id": "p48213377", "best_score": 50120, "new_best": true, "rank": 4812 }

If the score doesn't beat the player's best, the answer is the same shape with "new_best": false and the old best. Sending the same result twice is harmless: the second time it can't beat itself.

2. Top N.

httpCopy
GET /v1/games/g1/leaderboard?top=10 HTTP/1.1
Authorization: Bearer <player token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "entries": [
    { "rank": 1, "player_id": "p00000912", "name": "Valkyrie", "score": 98450 },
    { "rank": 2, "player_id": "p00731144", "name": "IronClad", "score": 97120 }
  ],
  "as_of": "2026-09-26T19:20:01.120Z"
}

3. One player's rank.

httpCopy
GET /v1/games/g1/players/p48213377/rank HTTP/1.1
Authorization: Bearer <player token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "player_id": "p48213377", "rank": 4812, "score": 50120, "total_players": 1000000 }

Recap

• Three operations: submit (from game servers only), top N, my rank.
• Best score counts; ranks must be exact and immediate.
• Losing memory must not lose scores.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From a Sorted Table to a Sorted Set

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

A relational table scores(player_id, best_score) with an index on best_score.

• Top 10: SELECT player_id, best_score FROM scores ORDER BY best_score DESC LIMIT 10. The index is already in score order, so the database reads ten entries from its end. Fast.
• My rank: SELECT COUNT(*) + 1 FROM scores WHERE best_score > :mine. Correct, and simple.

What it costs us: that COUNT(*) has to walk every index entry above my score. A B-tree index (the usual database index: a sorted tree of keys) knows where my score sits, but its nodes don't record how many entries are below them, so it can't skip ahead. For the top player the count is tiny; for a median player it is ~500,000 entries per request.

Step 1.1: "My Rank" Takes Seconds at 1M Players

Primitive: Distributed Cache Patterns & Eviction

Step 1.2: The Cache Restarted and the Leaderboard Is Empty

The durable table:

Step 1.3: Only the Best Score Counts

Step 1.4: Clients Send Fake Scores

Primitive: Distributed Rate Limiting · Loop: Design a Distributed Rate Limiter

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

Writes go durable-first: the conditional update in DynamoDB, then ZADD GT in the cache. Reads touch only the cache (plus names). If the cache is lost, the rebuild job refills it from DynamoDB and sets the ready marker last.

Key layout

The {g1} in braces is a hash tag: in cluster mode, only the part in braces decides which shard a key lives on, so all of one game's keys stay together. We don't need cluster mode yet, but the naming costs nothing now.

Trace 1: a score submission

Synthesizing vector architecture diagram...

Trace 2: a rank read

textCopy
GET /players/p48213377/rank
  → ZREVRANK lb:{g1}:alltime p48213377   → 4811    (O(log N), well under 1 ms in the engine)
  → ZSCORE   lb:{g1}:alltime p48213377   → 50120
  → 200 { rank: 4812, score: 50120 }

R1.7 Numbers

Traffic (assumptions: 40% of players play on a given day; 20 matches each; peak hour is 5× the daily average; each match leads to about 4 leaderboard reads: the results screen plus lobby visits)

Memory. The per-member size of a large sorted set is an estimate: roughly a hash-table entry, a skip-list node with its links, the member string, and allocator rounding. We use ~96 bytes per member for planning (Round 2 checks it bottom-up and gets about 100–115 bytes).

$$1{,}000{,}000 \times 96 \text{ B} = 96 \text{ MB} \;\times\; 1.35 \text{ (fragmentation and overhead)} \approx 130 \text{ MB}$$

The smallest memory-optimized node, cache.r7g.large, has 13.07 GiB. We use about 1% of it. We still take two of them (primary + replica in another AZ), because the replica is what makes a failover take seconds instead of a rebuild.

Operations per second on the cache: about 500 writes + 2,000 reads (each read is one or two O(log N) commands) ≈ 4,500 commands/s at peak. A single node handles on the order of 100K such commands a second (an assumption we confirm with a load test), so we have 20× headroom.

Rebuild time. 1M items of ~100 bytes is ~100 MB. A parallel Scan reads that in seconds, and adding 1M members in pipelined batches takes a few seconds more (at ≥100K adds a second). A full rebuild is under a minute.

Availability budget. 99.9% over a 30.4-day month is $43{,}776 \text{ min} \times 0.001 \approx 43.8$ minutes.

Monthly cost (us-east-1 on-demand list prices, 730 hours a month; rounded)

Say the headline in the interview: the whole leaderboard for a million players fits in 130 MB and costs about $575 a month. The sorted set turned a database problem into a memory problem, and at this scale memory is cheap.

R1.8 Trade-Offs

Two notes on the DynamoDB column. The index's sort key must be a Number: stored as strings, scores sort as text, so "9000" lands above "10000". And there's still no rank: an index answers "who is next", not "how many are before me".

In memory vs durable. We keep both, on purpose: the durable store answers "what is true", the sorted set answers "what is my rank" fast. We never ask the cache to be the only copy of anything.

R1.9 Failure Modes

Drill: CDC outbox dual-write drift

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Starts from SQL, and explains why COUNT(*) for rank gets slower as the table grows (an index has order but no counts).
• Names the sorted set and knows what's inside: a hash map for scores and a skip list whose spans make rank O(log N).
• Keeps a durable copy and treats the in-memory board as rebuildable.
• Makes "best score only" a conditional write in each store, and knows the two stores aren't atomic together.
• Refuses client-submitted scores.
• Derives traffic, memory and cost, and sees that the problem is small at this scale.

Follow-up questions

1. "Why not rebuild the board from DynamoDB on every restart and skip the replica?" Answer: we could at 1M players (under a minute). But every restart would show "rank updating" for that minute, and every failover would become a rebuild. The replica costs $128 a month and turns most failures into a few seconds. In Round 2, with 50M members per board (500M across ten), rebuilds take much longer, and the replica stops being optional.

2. "A player asks for rank 500,000 to 500,010. Is a deep page expensive?" Answer: no. A rank range costs O(log N + count): the skip list jumps to position 500,000 using the spans, then walks 10 members. What is expensive is a huge range (say 50,000 members in one call): the engine is single-threaded, so one big reply delays every other command. We cap top at 100 per call.

3. "Two players have the same score. Who ranks higher?" Answer: today, whatever the engine does: equal scores are ordered by member (the player ID) as a byte string, and in reversed ranges the order flips too. That's deterministic but meaningless to players. It's fine for Round 1; Round 2 makes "who scored first" the rule.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Esports Scale: Modes, Seasons, Ties and Cheaters"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 50M players · 100K submissions/s and 50K lobby reads/s peak · 99.99% · reads P99 < 5 ms, writes P99 < 15 ms

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built an all-time leaderboard for one mobile game with a million players: about 500 score submissions and 2,000 reads a second at peak, one region, three AZs, 99.9%. A database can sort, but it can't cheaply answer 'what is my rank?', because an index has order but no counts. So the board is a sorted set in ElastiCache: a hash map for scores plus a skip list whose spans give rank in O(log N). DynamoDB holds each player's best score and is the source of truth; the board is rebuildable, and a ready marker says when a rebuild is complete. Both stores update only if the new score is greater: a conditional UpdateItem and ZADD GT, written durable-first; they're monotonic and idempotent, so retries converge. Only our game servers submit, with HMAC-signed results. It's about 130 MB of memory and $575 a month. Three things are still open: ties are ordered by player ID, there's one board with no seasons, and a cheat that plays through a real match isn't caught."

Architecture v1, compact

textCopy
 game server ──signed result──► API ──1. UpdateItem IF best < new──► DynamoDB Scores (truth)
                                    └─2. ZADD GT──► ElastiCache lb:{g1}:alltime ──► replica (other AZ)
 app ──► API ──► ZREVRANK / ZRANGE REV (top 10) ──► names from Profiles
 rebuild job: Scan DynamoDB ──► ZADD in batches ──► set :ready

Round 1 step summary

Open costs: ties, seasons, cheats inside real matches, and one small board.

R2.1 The Scope Raise

Interviewer: "The game became an esports title. We have 50 million players, 10 million a day, across 10 game modes. Seasons are weekly and roll over at midnight UTC on Monday. When two players tie, whoever got the score first must rank higher; prize money depends on it. Players want to see the five players above and below them. During tournaments, submissions spike to 100,000 a second, and spectators want the top 100 live. Organized cheating has started. Reads must be under 5 ms at P99, writes under 15 ms, and we want 99.99%."

A scope raise is not the end of scoping. Before we fix anything, we ask back, and we say what each answer changes.

Scope change

The "Not yet" list from R1.2 is now mandatory: ties, seasons, players around me, spectators.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

Boards are now named by mode and season. m03 is a mode; 2026-w39 is ISO week 39 of 2026 (Monday 2026-09-21 00:00 UTC to Monday 2026-09-28 00:00 UTC).

Submit gains a season-proof end time and a short-lived signature; the season is not in the request, because the server derives it from ended_at:

httpCopy
POST /v2/games/g1/modes/m03/scores HTTP/1.1
Content-Type: application/json
X-Result-Signature: kid=gs-2026-09;hmac-sha256=9b04e1...

{
  "match_id": "m_9x2Qa1",
  "player_id": "p48213377",
  "score": 50120,
  "ended_at": "2026-09-21T00:20:00.000Z",
  "match_duration_s": 1140,
  "server_id": "gs-use1-az2-0183",
  "signed_at": "2026-09-21T00:20:00.410Z"
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "season": "2026-w39", "new_best": true, "best_score": 50120, "rank": 42, "status": "RANKED" }

status is RANKED or PENDING_REVIEW (step 2.5). kid names the signing key, so keys can rotate.

Players around me:

httpCopy
GET /v2/games/g1/modes/m03/seasons/current/players/p48213377/around?radius=5 HTTP/1.1
Authorization: Bearer <player token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "season": "2026-w39",
  "me": { "rank": 42, "score": 50120 },
  "entries": [
    { "rank": 37, "player_id": "p00990012", "name": "Kestrel", "score": 50410 },
    { "rank": 42, "player_id": "p48213377", "name": "Nova", "score": 50120 },
    { "rank": 47, "player_id": "p31000456", "name": "Moth", "score": 49980 }
  ]
}

(entries holds all 11 players; three are shown.) radius is capped at 10.

Top 100, cacheable. The same response for every viewer, with no personal data in it:

httpCopy
GET /v2/games/g1/modes/m03/seasons/current/top?limit=100 HTTP/1.1

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: public, max-age=2, stale-while-revalidate=2

{ "season": "2026-w39", "version": 1790197201500, "entries": [ { "rank": 1, "player_id": "p00000912", "name": "Valkyrie", "score": 98450 } ] }

Spectator stream over WebSocket: wss://live.example-games.com/v2/boards/g1/m03/current. The first frame is a full top 100; later frames are diffs, each with a version, at most one per second.

jsonCopy
{ "type": "diff", "board": "g1:m03:2026-w39", "version": 1790197203000,
  "changes": [ { "rank": 1, "player_id": "p77120003", "name": "Orca", "score": 99500 } ] }

R2.4 Design Evolution: Ties, Windows, Seasons, Crowds and Cheaters

Step 2.1: Two Players Tie; Whoever Scored First Must Rank Higher

Step 2.2: "Show Me the Players Around Me"

Step 2.3: Season 39 Ends at Midnight, and Scores Are in Flight

Synthesizing vector architecture diagram...

The write path never reads the pointer: the signed end time alone decides the board, so every API task agrees. The pointer only tells lobbies which season to display.

Step 2.4: 50,000 Lobbies Poll the Top 100 Every Second

Primitive: WebSocket, SSE & Long Polling · Drill: Caching hot product page · Drill: WebSocket reconnect thundering herd

Step 2.5: A Cheater Jumps to Rank 1

Primitive: Bot Defense, Sybil Resistance & Registration Abuse · Primitive: Message Queues vs Event Streams

Step 2.6: Memory Churn and One Thread Under 100K Submissions a Second

Primitive: Database Sharding & Partition Keys · Primitive: Circuit Breaker, Bulkhead & Fault-Tolerance Patterns

Go deeper: closing the dual-write gap with the table's change stream. Round 1 wrote DynamoDB, then the cache. At 100K submissions a second, a task crashing between the two writes is no longer rare. And a retry can't fix it: the retry's conditional update fails ("not greater", because the first attempt already stored it), so the API would skip the ZADD. Two fixes, used together:

• On a failed condition, the API asks DynamoDB to return the stored item (ReturnValuesOnConditionCheckFailure = ALL_OLD). If the stored best came from this match_id, it's our own earlier attempt, and the API re-applies ZADD GT.
• DynamoDB Streams on the Scores table records every change to an item exactly once, in order per item, for 24 hours. A Lambda function reads it and re-applies ZADD GT for every new best. Lambda may process a batch more than once, which is harmless: ZADD GT with the same value is a no-op. The direct write is the fast path; the stream is the guarantee.

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Three paths. Submissions: the API checks the signature and plausibility, writes DynamoDB and the audit stream in parallel, then updates the board; a change-stream Lambda repairs any board write a crash skipped. Reads: rank and window from replicas; the top 100 from CloudFront, which asks us a few times a second. Spectators: two publishers diff the top 100 and gateways push the newest version.

Trace 1: a tied score

textCopy
A: ended 00:20:00.000 → t = 12,000 → 50,120 × 10^7 + 9,987,999 = 501,209,987,999
   UpdateItem Scores(p_A, m03#2026-w39) SET best = 501209987999 IF best < 501209987999 → OK
   ZADD lb:{g1:m03}:2026-w39 GT 501209987999 p_A
B: ended 00:20:00.350 → t = 12,003 → 501,209,987,996
   same steps → OK; B's value is 3 lower
ZRANGE … REV 0 1 → [p_A, p_B]; decoded scores both 50,120; A is rank 1

Trace 2: the Monday rollover

textCopy
23:59:58.2  match ends; signed ended_at = 2026-09-27T23:59:58.200Z
00:00:03    result arrives → week(ended_at) = 2026-w39 → ZADD lb:{g1:m03}:2026-w39   (grace until 00:15)
00:00:05    another match ends at 00:00:04 → 2026-w40 → first ZADD creates lb:{g1:m03}:2026-w40
00:00:30    current_season pointer → 2026-w40 (lobbies switch as each task polls; writes unaffected)
00:15:00    2026-w39 frozen; late results → review queue
00:15       board writers now drop any write for w39 (freeze check); DynamoDB still records late results
00:15–02:00 archive job waits for w39's quarantine to empty, pages w39 from a replica → S3; count == ZCARD;
            top 1,000 → DynamoDB; UNLINK

Trace 3: a cheater quarantined

textCopy
result: p_new, 99,500 in 240 s; account age 6 days; points/min above the mode's ceiling
cached 1,000th-place value: 71,300 → this would enter the top 1,000 → PENDING
  UpdateItem SET pending_best = … ; ZADD lbq:{g1:m03}:2026-w39 …   (public board untouched)
  response: { status: "PENDING_REVIEW" }
review worker (from Kinesis): replays telemetry → reject → flag account; re-check its other results

Tables and keys

R2.6 Numbers and Cost

Traffic (the current track's figures, re-derived)

Board memory. The track's estimate, and a check:

$$50\text{M} \times 96 \text{ B} = 4.8 \text{ GB per board} \times 10 = 48 \text{ GB} \times 1.35 \approx 64.8 \text{ GB per node}$$

Every replica holds a full copy, so 64.8 GB is per node, not a total. The ~96 B is an estimate. A bottom-up count for the default large-set layout: a hash-table entry (24 B, rounded up by the allocator to ~32), a skip-list node (score, member pointer, back pointer and on average ~1.33 levels of 16 B, rounded up: ~53), the member string p48213377 with its header (~16), and hash-bucket slots (~8–16). That's roughly 100–115 B, depending on engine version. At 113 B the ten boards would be ~56.5 GB. We measure MEMORY USAGE on a sampled board before buying nodes. The 50M per board is also an upper bound: it assumes every registered player plays every mode every week.

Sizing, carried plan vs our plan. ElastiCache reserves 25% of a node's memory for non-data use by default; the engine's memory limit is the other 75%.

Both fit. We pay about $1,530 more a month for six write threads, failovers and rebuilds of at most two boards each, and parallel rebuilds.

Why old boards must go within hours: if week 39 stayed a full week, each shard would briefly hold two full weeks: 19.2 GB ≈ 17.9 GiB, 91% of the limit. Deleting within two hours keeps the overlap to a few percent.

Network

Latency budgets (P99, in-region; the DynamoDB and Kinesis figures are assumptions we confirm under load)

The DynamoDB write and the Kinesis put don't depend on each other, so they run in parallel and the step costs the slower of the two, not the sum. The read path's steps do depend on each other, so they add.

50K ZCOUNTs a second per replica on a 50M-member set is within what we expect of one engine thread, but it is a load-test question; each extra replica for the tournament shard would add ≈ $255 a month. And the Kinesis P99 of 10 ms is our assumption, not a published figure: if Kinesis P99 exceeds ~12 ms, the API writes the audit record asynchronously after the response, at least once from a local buffer with retries (the lake deduplicates by match_id), and we alarm on buffer drops, since a crash can lose a buffered record that the game server won't resend.

Audit storage

$$500\text{M matches/day} \times 200 \text{ B (compressed Parquet)} = 100 \text{ GB/day} \qquad 100 \text{ GB} \times 365.25 \times 5 \approx 182.6 \text{ TB}$$

Availability budget. 99.99% over a 30.4-day month is $43{,}776 \times 0.0001 \approx 4.4$ minutes.

Monthly cost (us-east-1 on-demand list prices; rounded)

Say the headline: serving the same top 100 to 50,000 lobbies a second is about two-thirds of the bill. The leaderboard's memory is $4.6K. The levers are on the lobby side (R2.7).

R2.7 Trade-Offs

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Treats the score as an encoding problem: proves the tie-break fits in $2^{53}$, states the resolution, and knows how equal scores are ordered by default (and that reverse ranges reverse it).
• Makes the season a property of the match (signed end time), not of a clock or a switch.
• Separates "the same answer for everyone" (edge cache) from "a personal answer" (rank), and knows what the CDN does and doesn't remove.
• Designs anti-cheat as layers with honest limits, and puts the strictest path where the money is.
• Checks the carried memory estimate, sizes with reserved memory and headroom, and spreads boards for write threads, not just for memory.
• Uses idempotent, monotonic writes so repairs and replays are always safe.

Follow-up questions

1. "Most submissions aren't new bests. Can we avoid paying for a DynamoDB write on each?" Answer: yes: read the stored best first (an eventually consistent read costs a tenth of a write) and only send the conditional update if the result looks higher. The read is a filter, not the guard: two racing requests can both pass it, and the conditional update still decides. At 80% non-improving results, that's roughly $9.5K → $1K of reads plus $1.9K of writes a month. Round 3 does this.

2. "Why not store the top-100 response in the cache and skip CloudFront?" Answer: that saves our cache a few commands, but our servers would still send 6 Gbps of responses. The expensive part is the bytes and requests at 50,000 a second, and only a layer in front of us removes them from our servers.

3. "A tournament needs a separate board that starts at 18:00 and ends at 21:00. What changes?" Answer: it's a season with custom boundaries: its own key, its own start time for the tie-break offset (3 hours in 100 ms units is 108,000, far under $10^7$), and the same signed-end-time rule for eligibility.

Interview gotchas from this round

---

Round 3 · Architect · "Half a Billion Players, Friends and Regions"

~45 min · Principal (L7) · 4 regions · 500M players, 100M DAU · ~500K submissions/s peak worldwide · 99.99% per region · exact, auditable ranks for prize tiers

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run an esports leaderboard in one region: 50 million players, 10 million a day, 10 modes with weekly seasons, 500 million submissions a day, 100,000 a second at tournament peak, 99.99%, reads under 5 ms and writes under 15 ms at P99. Each board is a sorted set per mode and season. Ties go to whoever finished first: the score is an integer, raw score times 10^7 plus the inverted time since season start in 100 ms units, proven below 2^53. The season comes from the game server's signed end time, with a 15-minute grace window; old boards are archived to S3 and unlinked within hours. DynamoDB holds each player's best and is the truth; a change stream repairs any board write a crash skipped. Ten boards, about 48 GB, sit on six shards with two replicas each; ranks are counted on replicas. Lobbies get the top 100 from CloudFront with a 2-second TTL; spectators get versioned diffs over WebSocket. Suspicious scores that would enter the top 1,000 are quarantined, and every match goes to an audit lake. About $71K a month, two-thirds of it serving the lobby top 100. Three things are still open: it's one region, a board is one sorted set on one shard, and there are no friends boards or provable payouts."

Architecture v2, compact

textCopy
 game server ─signed─► API ─┬─ UpdateItem IF best < new ─► DynamoDB Scores ─streams─► repair Lambda ─┐
                            ├─ PutRecord (parallel) ─► Kinesis ─► Firehose ─► S3 audit; review workers │
                            └─ ZADD GT ─► ElastiCache: 6 shards × 3 nodes, lb:{g1:mode}:<week> ◄───────┘
 app ─► API ─► ZCOUNT / ZREVRANK + ZRANGE REV on replicas ─► profile cache (SET NX fills)
 lobby ─► CloudFront (2 s) ─► API        spectators ─► WebSocket gateways ◄─ 2 versioned publishers

Steps so far

Open costs: one region, one sorted set per board, no friends, no provable payouts.

R3.1 The Scope Raise

Interviewer: "We're global. 500 million players, about 100 million a day, in four regions, and at a world final submissions peak around 500,000 a second. Players want three boards: global, their region, and their friends. Prize payouts for the top 1,000 must be exact and provable if someone disputes them. And the leaderboard must survive losing a region."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Boards have a scope, and ranks have a precision.

httpCopy
GET /v3/games/g1/modes/m03/seasons/current/players/p48213377/rank?scope=global HTTP/1.1
Authorization: Bearer <player token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "scope": "global",
  "season": "2026-w39",
  "score": 31250,
  "precision": "approximate",
  "percentile": 12.3,
  "rank_estimate": 61500000,
  "error_bound": 50000,
  "as_of": "2026-09-23T21:04:11Z"
}

For a player near the top, the same call returns "precision": "exact" and "rank": 4812. scope is global, region or friends; region also returns "region": "ap-northeast-1", and friends returns the ranked list of friends.

Certified standings for a closed season (public, read-only):

httpCopy
GET /v3/games/g1/modes/m03/seasons/2026-w39/standings/certified HTTP/1.1

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "season": "2026-w39",
  "cutoff": "2026-09-28T00:15:00Z",
  "tie_rule": "score desc, ended_at asc (ms), match_id asc",
  "entries_url": "https://standings.example-games.com/g1/m03/2026-w39/standings.json",
  "sha256": "4b1f…9e0a",
  "signature": "MEUCIQ…",
  "key_id": "alias/standings-signing",
  "algorithm": "ECDSA_SHA_256"
}

Anyone with the public key can check that the standings file hashes to sha256 and that the signature matches.

R3.4 Design Evolution: Tiers, Precision, Friends, Regions and Proof

Step 3.1: One Board No Longer Belongs on One Shard

Primitive: Database Sharding & Partition Keys · Drill: Sharding tenant hotspot

Step 3.2: Most Players Don't Need an Exact Rank

textCopy
board g1:m03:2026-w39 (global)
  exact tier   (sorted set, top 1,050,000)    ranks 1 … 1,050,000        exact
  histogram    bucket 9,999  [98,110 … max]    count 50,112  ┐
               bucket 9,998  [96,870 … 98,109] count 49,871  │  rank ≈ Σ counts above
               …                                             │        + position in my bucket
               bucket 0      [0 … 210]         count 50,006  ┘

Step 3.3: "Rank Me Among My Friends"

Step 3.4: Players in Four Regions, One Global Board

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active · Drill: Replication multi-region consistency

Step 3.5: Prize Payouts Must Be Provably Correct

Primitive: Event Sourcing & CQRS · Drill: Event sourcing projection rebuild lag

Step 3.6: A Region Is Gone

Synthesizing vector architecture diagram...

The partner already holds the data and the standby board. Only the last unreplicated second is missing, and the reconcile restores it by taking the maximum over both regions' match logs.

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each region is a complete leaderboard for its own players, plus standby boards for its partner and its own copy of the global views. The only things that cross regions are the best-score table's replication, the audit lake's copy to the partner, and heartbeats. No read ever leaves the region.

Trace 1: a Tokyo player's new best reaches the global board

textCopy
t = 0 ms    game server (ap-northeast-1) → POST score; signature, checks
            GetItem PlayerBest (filter): stored 31,100 < new 31,250 → continue
            UpdateItem IF best < new → OK   ∥   PutRecord match-events
            rank from the local histogram snapshot → "top 12.3%" in the response
t ≈ 1 s     replicated to us-east-1, eu-west-1, ap-southeast-1
t ≈ 1–2 s   each region's stream: aggregator moves one count bucket 8,760 → 8,771;
            the board consumer skips the exact tier (31,250 < floor 88,400)
t ≈ 2 s     API nodes load the next histogram snapshot; the player's global percentile updates everywhere

Trace 2: a payout computation

textCopy
Mon 00:15   grace window closes for 2026-w39
Mon 00:30+  each region seals its log (Firehose flushed past the cutoff; manifest written)
            start-execution name=season-close-g1-m03-2026-w39   (a repeat start can't create a second run)
            Athena: per player, max(score) over valid matches, and the earliest ended_at (then smallest match_id) at that score
            ORDER BY score DESC, ended_at ASC, match_id ASC
            exclude disqualified / unapproved; top 1,000
            diff vs live exact tier → report (e.g. 3 players reordered by ms ties)
            SHA-256(standings.json) → KMS Sign (ECDSA_SHA_256) → S3 Object Lock (compliance, 5 years)
            publish certified standings; finance pays from this file only

R3.6 Numbers and Cost

Traffic (assumptions: 20% of players daily, as in Round 2; 50 matches each; 20% of results are new bests; the peak is a world final overlapping evening peaks)

Step 3.1's design (exact for everyone), per region. Global boards: 10 modes × 500M × 96 B = 480 GB. Own regional boards, us-east-1: 30% of that, 144 GB; its partner's standby boards (eu-west-1, 25%, step 3.6): 120 GB. Total ≈ 744 GB of data. Keeping each shard at half of a cache.r7g.4xlarge's ~85 GB limit (42.6 GB), that's $744 \div 42.6 \approx 17.5$, so 18 shards × 3 nodes = 54 nodes. eu-west-1 is the same (480 + 120 + 144 GB); the two Asian regions hold 480 + 120 + 96 = 696 GB, 17 shards, 51 nodes each. That's 210 nodes: 210 \times \1.396 \times 730 \approx **\214K a month**. Rebuilding one 500M-member board through one thread at an assumed 200,000 adds a second takes $500\text{M} \div 200{,}000 = 2{,}500$ s ≈ 42 minutes, against a monthly downtime budget of 4.4 minutes. Split across 10 tiers in parallel, about 4 minutes.

Step 3.2's design, per region.

Two shards of cache.r7g.xlarge (limit ≈ 19.7 GiB each) with two replicas each: under 10% full, and two write threads. 6 \times \0.3496 \times 730 \approx $1{,}530 per region, **≈ \6.1K** for four. About 35× cheaper than exact-for-everyone, and a rebuild is 1M members per tier, a few seconds.

Histogram precision. ~50,000 players per bucket globally (once the season's distribution resembles last season's) → rank error at most the player's bucket population, ~50,000; a 0.1% percentile step is 500,000 players, 10× the error. For a regional board of 100M, buckets hold ~10,000 players.

Friends boards (assume 2 views per daily player per day, 100 friends on average, 20% served from the 30 s cache)

$100\text{M} \times 2 \times 30.4 \approx 6.08\text{B views/month} \times 50 \text{ RRU} \times 0.8 \approx 243\text{B RRU} \times \$0.125/\text{M} \approx \$30\text{K}$

Latency. Exact rank: one ZREVRANK on a replica, ~1 ms. Approximate rank: a lookup in the API's memory, well under 1 ms. Friends: friend list ~5 ms (an assumption for the social service), then up to 10 parallel BatchGetItems, the slowest ~10 ms, plus sorting: ~15–20 ms, under the 50 ms target.

Audit log. $5\text{B} \times 200 \text{ B} = 1$ TB a day, ≈ 1.83 PB over five years, plus the partner copy.

Monthly cost (list prices, us-east-1 rates where regions differ; rounded; storage a year in)

A little over a quarter of a cent per daily player per month. The biggest lines are the ones every player touches: writing each new best four times, and serving the lobby. Step 3.2 is the line that didn't appear: exact-for-everyone would have added ~$208K of memory for a precision no one below the top can see.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we answer "what is my rank?" exactly and instantly while scores keep changing? The answer is now:

• Instantly: a sorted set gives O(log N) rank for the players who get an exact answer; a histogram in the API's memory answers everyone else without touching the cache.
• Exactly, where it matters: exact near the top, with ties encoded into a proven 53-bit integer; exact and provable for payouts, recomputed from the log and signed.
• While scores keep changing: every write is a maximum, so retries, replays, repairs and region reconciles are all safe; the only numbers we add up (histogram counts) are counted exactly once by saving state with stream positions.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per region OPS 8 · REL 6

Zero-downtime season rollover (nothing is switched on the write path; each step can be repeated) OPS 7

1. Before Monday 00:00: confirm next season's histogram boundaries are loaded (last season's quantiles) and every region's API has the season schedule.
2. 00:00: nothing to do for writes; each result's signed ended_at picks its week.
3. 00:00–00:01: point current_season at the new week (CLI 5), for lobby reads.
4. 00:15: grace ends; the old week is frozen in every region.
5. 00:30+: start the season-close workflow per board (CLI 6); it waits for all four regions to seal.
6. After it succeeds: archive and UNLINK the old week's exact tiers; drop its histograms; verify the certified file's signature (CLI 9) and lock (CLI 10).

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace names, IDs and times with real ones.

textCopy
# 1. Board cache memory (per node)
aws cloudwatch get-metric-statistics --region ap-northeast-1 --namespace AWS/ElastiCache --metric-name DatabaseMemoryUsagePercentage --dimensions Name=CacheClusterId,Value=lb-boards-apne1-0001-001 --statistics Maximum --period 60 --start-time 2026-09-27T20:00:00Z --end-time 2026-09-27T21:00:00Z

# 2. Board consumer lag (Lambda reading DynamoDB Streams)
aws cloudwatch get-metric-statistics --region ap-northeast-1 --namespace AWS/Lambda --metric-name IteratorAge --dimensions Name=FunctionName,Value=board-consumer --statistics Maximum --period 60 --start-time 2026-09-27T20:00:00Z --end-time 2026-09-27T21:00:00Z

# 3. Replication lag from us-east-1 into eu-west-1
aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/DynamoDB --metric-name ReplicationLatency --dimensions Name=TableName,Value=PlayerBest Name=ReceivingRegion,Value=eu-west-1 --statistics Average --period 60 --start-time 2026-09-27T20:00:00Z --end-time 2026-09-27T21:00:00Z

# 4. Which season do lobbies show?
aws ssm get-parameter --region us-east-1 --name /leaderboard/g1/current_season

# 5. Point lobbies at the new season (reads only)
aws ssm put-parameter --region us-east-1 --name /leaderboard/g1/current_season --value 2026-w40 --type String --overwrite

# 6. Start the season close (the name is unique, so a repeat can't start a second run)
aws stepfunctions start-execution --state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:season-close --name season-close-g1-m03-2026-w39 --input '{"game":"g1","mode":"m03","season":"2026-w39"}'

# 6b. Resume a failed season close from where it failed
aws stepfunctions redrive-execution --execution-arn arn:aws:states:us-east-1:111122223333:execution:season-close:season-close-g1-m03-2026-w39

# 7. Board cluster status (shards, failover state)
aws elasticache describe-replication-groups --region ap-northeast-1 --replication-group-id lb-boards-apne1

# 8. Is a region's API health check failing?
aws route53 get-health-check-status --health-check-id 11111111-2222-3333-4444-555555555555

# 9. Verify the standings signature (the message is the 32-byte SHA-256 digest of the file)
aws kms verify --region us-east-1 --key-id alias/standings-signing --message fileb://standings-2026-w39.sha256 --message-type DIGEST --signing-algorithm ECDSA_SHA_256 --signature fileb://standings-2026-w39.sig

# 10. Confirm the standings file is locked
aws s3api get-object-retention --bucket lb-certified-standings --key g1/m03/2026-w39/standings.json

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Asks where exactness matters, and designs exact, approximate and provable ranks as three different products.
• Compares score-range and hash sharding by the cost of a rank query, not just by write balance.
• Refuses additive counters fed by at-least-once consumers; uses ZCARD, or saves state with stream positions, and publishes snapshots.
• Places data by who writes it (one home region per player) and knows what global tables do with conditions and conflicts, including during failover.
• Makes the log the truth for money, and relies on maxima being idempotent to make every replay safe.
• Prices the alternatives and shows which line the design removed.

Follow-up questions

1. "A player moves from Europe to Asia. What happens?" Answer: their home region changes (written by the old home region, then owned by the new one). Their PlayerBest items stay where they are in the global table, but future writes come from the new region. Their regional-board membership moves: the stream consumer sees the home-region attribute change and re-places them by their current stored item: add to the new region's exact tier (if they qualify) and histogram, then remove from the old region's, so a late or repeated record can't leave them in both.

2. "Why not keep the regional boards exact for everyone? 150M players per region is only ~14 GB." Answer: we could, per board. But it's 10 modes × own and partner regions, and those boards bring back the problems of step 3.1: large failovers, slow rebuilds and rank queries on the cache for every player. We'd do it only if the product shows exact regional ranks deep down; the measured question is whether anyone looks at them.

3. "The esports league wants live exact ranks for the top 10,000 in a regional qualifier. Does the design allow it?" Answer: yes: the exact tier is 1M deep, so 10,000 is well inside it. For the qualifier's payout, the same season-close workflow runs over the qualifier's board and window.

Interview gotchas from this round

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: does a player's best score count or the sum, who submits scores, and how exact and how fresh must a rank be?"
• When the scope is raised: "Here's what breaks, and I'll fix it in this order: anything that can put a wrong score or a wrong payout in front of players, then anything that can take the board down, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels