Design a Real-Time Gaming Leaderboard

1. Problem Statement & Scope Clarification

System Mission

Design an ultra-low-latency, globally distributed real-time gaming leaderboard platform (similar to Xbox Live, PlayStation Network, and competitive esports platforms) capable of processing hundreds of thousands of score updates per second, maintaining exact global rankings across tens of millions of active players, and delivering Top-$N$ and surrounding relative rank windows in under 5 milliseconds.

Functional Requirements

1. Score Submission (SubmitScore): Ingest and atomically update a player's score for a specific game and season, recalculating their exact global rank in real time.
2. Top-$N$ Global Leaderboard (GetTopRankings): Retrieve the current top $N$ players (e.g., Top 100) with ranks, usernames, avatars, verified scores, and regional badges.
3. Surrounding Player Window (GetSurroundingRanks): Retrieve a specific player's exact rank along with $M$ players directly above and below them (relative ranking window).
4. Deterministic Tie-Breaking: Automatically break score ties deterministically by awarding the higher rank to the player who achieved the score earlier in time.
5. Periodic & Seasonal Reset: Support daily, weekly, and seasonal leaderboard archival and zero-downtime rollover.
6. Live Top-K Push Broadcast: Stream live Top-100 rank changes over WebSockets to spectators during global live tournament matches.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.999\%$ uptime SLA for leaderboard query and score ingest endpoints.
• Ultra-Low Latency:
  • Read Latency (Top-$N$ and Relative Window): P95 $< 2\text{ ms}$, P99 $< 5\text{ ms}$.
  • Write Latency (Score Ingest & Rank Calculation): P95 $< 8\text{ ms}$, P99 $< 15\text{ ms}$.
• Throughput: Support $> 100,000\text{ score submissions/sec}$ and $> 50,000\text{ reads/sec}$ during global live gaming events.
• Accuracy: $100\%$ linearizable rank precision for top-tier competitive play (zero stale, inverted, or skipped ranks).
• Durability: Zero match score loss guarantee; all game events durably archived in an audit lake.

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Player Base & Ingest QPS

• Total Registered Players: $50,000,000$ ($50\text{M players}$).
• Daily Active Players (DAU): $10,000,000$ ($10\text{M DAU}$).
• Daily Match Activity: Each DAU completes an average of $50$ matches/day: $\text{Daily Score Updates} = 10,000,000 \times 50 = \mathbf{500,000,000\text{ updates/day}}$
• Average Ingest QPS: $\text{Ingest QPS}_{\text{avg}} = \frac{500,000,000}{86,400\text{ sec}} \approx \mathbf{5,787\text{ updates/sec}}$
• Peak Ingest QPS ($18\times$ spike during global esports tournaments): $\text{Ingest QPS}_{\text{peak}} = \mathbf{100,000\text{ updates/sec}}$
• Peak Read QPS (Top-$N$ views from game lobby screens): $\text{Read QPS}_{\text{peak}} = \mathbf{50,000\text{ reads/sec}}$

Memory Footprint Derivation (Redis Sorted Set)

A Redis ZSET is implemented as a dual-structure: an in-memory SkipList (for $O(\log N)$ range scans and rank lookups) combined with a Hash Table (for $O(1)$ player-to-score point lookups):

• Member ID (usr_998124): $16\text{ Bytes}$
• Score (float64 IEEE-754 double precision): $8\text{ Bytes}$
• SkipList Node pointers (level spans, forward/backward pointers): $\approx 32\text{ Bytes}$
• Hash Table dictionary entry (dictEntry struct): $\approx 24\text{ Bytes}$
• Jemalloc memory alignment & allocator padding: $\approx 16\text{ Bytes}$
• Total In-Memory Size per Player Record: $\approx \mathbf{96\text{ Bytes}}$.

$$\text{Active Leaderboard Memory} = 50,000,000\text{ players} \times 96\text{ Bytes} \approx \mathbf{4.8\text{ GB RAM}}$$

Accounting for 10 active game modes and seasons:

$$\text{Multi-Mode RAM} = 10\text{ game modes} \times 4.8\text{ GB} = \mathbf{48\text{ GB RAM}}$$

Adding jemalloc fragmentation and replication-buffer overhead ($\times 1.35$):

$$\text{Required RAM per Node} = 48\text{ GB} \times 1.35 \approx \mathbf{64.8\text{ GB RAM}}$$

Every replica holds a full copy of the dataset, so this is the size of each node, not the total. Provision an Amazon ElastiCache Redis replication group of $1$ primary $+ 2$ read replicas on cache.r6g.4xlarge ($105.8\text{ GiB}$ per node, one node per AZ), leaving $\approx 38\%$ headroom for fragmentation spikes (Section 8.1). Three cache.r6g.xlarge nodes ($26\text{ GiB}$ each) would not fit even one copy. If a single game mode alone outgrows one node, switch to cluster mode and hash-slot the lb:<game>:<season> keys across shards, since each ZSET must still live entirely on one shard.

Network Bandwidth

• Ingress Bandwidth (Score Updates): $\text{Bandwidth}_{\text{in}} = 100,000\text{ updates/sec} \times 500\text{ Bytes (signed match token)} = 50\text{ MB/sec} \approx \mathbf{400\text{ Mbps}}$
• Egress Bandwidth (Top-100 Reads): Each Top-100 response returns 100 hydrated player cards ($\approx 15\text{ KB}$ JSON payload): $\text{Bandwidth}_{\text{out}} = 50,000\text{ reads/sec} \times 15\text{ KB} = 750\text{ MB/sec} \approx \mathbf{6.0\text{ Gbps}}$ Placing Amazon CloudFront in front of the Top-100 endpoint with a 2-second edge cache TTL absorbs $99.9\%$ of lobby polling traffic, reducing backend egress to $< 10\text{ Mbps}$.

Historical Storage Footprint (5-Year Horizon)

• Daily Match Audit Storage: $\text{Daily Audit Volume} = 500\times 10^6\text{ matches/day} \times 200\text{ Bytes (compressed Parquet)} = \mathbf{100\text{ GB/day}}$
• 5-Year Historical Match Lake: $\text{Storage}_{5\text{yr}} = 100\text{ GB/day} \times 365.25 \times 5 \approx \mathbf{182.6\text{ TB}}$ Streamed via Amazon Kinesis Data Firehose into an Amazon S3 Analytics Lake partitioned by year/month/day/game_id.

---

3. High-Level Architecture & AWS Component Mapping

Synthesizing vector architecture diagram...

Follow the three kinds of traffic. Score writes: players reach the API fleet through the NLB, and each new score is a ZADD on the Redis primary, where a sorted set keeps ranks updated in O(log N). Rank reads: "what is my rank?" (ZREVRANK) and "show the top N" (ZREVRANGE) go to read replicas, so heavy reads never slow down writes. Spectators: the top 100 is cached at CloudFront for 2 s, and changes are pushed live over WebSocket from the primary's Pub/Sub. Every score is also written to Kinesis; in the "Durable Match Audit & Historical Archive" panel, it is archived to S3 and checked by an anti-cheat worker. Redis gives speed, the stream gives durability and auditing, and caching absorbs the spectator crowd.

Data Flow Walkthrough

1. Score Ingestion & Deterministic Tie-Breaking:
   • Game server completes a match and emits POST /v1/scores containing player ID, raw score, and HMAC-SHA256 match signature.
   • The Leaderboard API validates the token, applies the season-relative tie-breaking formula (Section 4.1) to encode submission time into the fractional part of the floating-point score, and executes an atomic ZADD ... GT against the ElastiCache Redis Primary.
   • Concurrently, the API writes the raw match event to Amazon Kinesis Data Streams for durable auditing and anti-cheat verification.
2. Top-$N$ Retrieval & Edge Offloading:
   • Game clients request GET /v1/leaderboards/{game_id}/top?limit=100.
   • The request hits Amazon CloudFront Edge Cache (TTL: 2 seconds). On edge miss, the request forwards to an ECS API task, which executes ZREVRANGE lb:<game>:<season> 0 99 WITHSCORES against an ElastiCache Read Replica ($< 1\text{ ms}$).
   • The API batch-hydrates player display names and avatar URLs from Amazon DynamoDB (or a Redis Hash cache), strips the fractional tie-breaker bits, and returns the response.
3. Surrounding Window Calculation:
   • Player queries GET /v1/leaderboards/{game_id}/surrounding?radius=5.
   • The API executes ZREVRANK to locate the player's exact 0-indexed rank $R$, followed immediately by ZREVRANGE (R - 5) (R + 5) WITHSCORES.
4. Live Tournament Push Broadcast:
   • If an update changes the Top-100 standings, the API publishes a lightweight delta event to a Redis Pub/Sub channel.
   • The WebSocket Push Gateway fans out the new Top-100 snapshot to subscribed spectator clients in real time ($< 50\text{ ms}$).

Concrete Step-by-Step Request Walkthrough: Tracing Score Submission & Rank Calculation

---

4. API Interface Design & Wire Protocols

1. Deterministic Tie-Breaking Score Math & Precision Boundary Analysis

By default, Redis ZSET orders members with identical scores lexicographically by user_id. In competitive esports, identical scores must break ties by awarding the higher rank to the player who achieved the score earlier in time.

IEEE-754 Precision Analysis & The Precision Collapse Trap

A standard Redis score is an IEEE-754 double-precision 64-bit float (float64), possessing 53 bits of significand (mantissa), providing approximately $15.95$ decimal digits of precision ($2^{53} \approx 9.007 \times 10^{15}$). If a system naively adds a 13-digit Unix millisecond timestamp ($T_{\text{now\_ms}} \approx 1.76 \times 10^{12}$) to a score of $500,000$ (6 decimal digits), total significant digits required exceed 19 decimal digits, triggering catastrophic floating-point mantissa truncation: the low-order timestamp bits are rounded off, and two scores achieved minutes apart collapse to identical float values!

Production Relative-Offset Formula

To preserve exact millisecond precision without exceeding 15 decimal digits, production systems encode a tournament-relative second offset ($T_{\text{rel}} = T_{\text{event}} - T_{\text{season\_start}}$):

• Maximum season duration: 30 days $= 2,592,000\text{ seconds}$ (7 decimal digits).
• Maximum raw score: $100,000,000$ (8 decimal digits).
• Total required precision: $8 + 7 = 15\text{ decimal digits} \le 15.95\text{ digits}$ (zero truncation).

$$S_{\text{combined}} = S_{\text{raw}} + \left( 1.0 - \frac{T_{\text{rel\_sec}}}{10^7} \right)$$

Because $T_{\text{rel\_sec}}$ increases monotonically as the tournament progresses, an earlier timestamp yields a larger fractional value, guaranteeing deterministic first-achiever precedence:

textCopy
Player A (scored at T_rel = 1000s): 50000 + (1.0 - 0.0001000) = 50000.9999000
Player B (scored at T_rel = 2000s): 50000 + (1.0 - 0.0002000) = 50000.9998000
Result: Score A > Score B -> Player A is ranked higher than Player B

64-Bit Integer Bit-Shifting Alternative (Internal Leaderboards)

When implementing custom leaderboard engines or using integer secondary indices, the score and timestamp are packed into a single 64-bit unsigned integer:

$$S_{64} = \left(S_{\text{raw}} \ll 32\right) \mid \left(2^{32} - 1 - T_{\text{rel\_sec}}\right)$$

The upper 32 bits store the raw score (up to $4.29 \times 10^9$), and the lower 32 bits store the inverted timestamp (up to 136 years of relative seconds), yielding bitwise-deterministic ordering with zero floating-point rounding hazards.

Upon read, the API displays the clean integer score:

$$\text{Display Score} = \lfloor S_{\text{combined}} \rfloor$$

2. RESTful Score Submission Endpoint

httpCopy
POST /v1/scores
Host: api.leaderboard.aws.internal
Authorization: Bearer <jwt_session_token>
X-Match-Token: 3a7b9c1d4e8f0a2b4c6e8d0f2a4b6c8e
Content-Type: application/json

{
  "game_id": "game_01",
  "season_id": "season_12",
  "user_id": "usr_998124",
  "raw_score": 50120,
  "match_id": "match_881923"
}

Response: 200 OK

jsonCopy
{
  "status": "SUCCESS",
  "data": {
    "user_id": "usr_998124",
    "global_rank": 42,
    "score": 50120,
    "is_new_high_score": true,
    "previous_rank": 78
  }
}

3. RESTful Top-100 Rankings Endpoint

httpCopy
GET /v1/leaderboards/game_01/top?season_id=season_12&limit=100
Host: api.leaderboard.aws.internal
Accept: application/json

Response: 200 OK
Cache-Control: public, max-age=2, stale-while-revalidate=5
{
  "status": "SUCCESS",
  "data": {
    "game_id": "game_01",
    "season_id": "season_12",
    "total_players": 48290124,
    "entries": [
      {
        "rank": 1,
        "user_id": "usr_apex_pro",
        "username": "Valkyrie_Queen",
        "avatar_url": "https://cdn.gaming.internal/avatars/valk.webp",
        "score": 98450,
        "country_code": "SE"
      },
      {
        "rank": 2,
        "user_id": "usr_titan_99",
        "username": "IronClad",
        "avatar_url": "https://cdn.gaming.internal/avatars/iron.webp",
        "score": 97120,
        "country_code": "KR"
      }
    ]
  }
}

4. gRPC Interface Specification (leaderboard.proto)

protobufCopy
syntax = "proto3";

package hispeeddesign.leaderboard.v1;

service LeaderboardService {
  rpc SubmitScore (SubmitScoreRequest) returns (SubmitScoreResponse);
  rpc GetTopRankings (GetTopRankingsRequest) returns (GetTopRankingsResponse);
  rpc GetSurroundingRanks (GetSurroundingRanksRequest) returns (GetSurroundingRanksResponse);
}

message SubmitScoreRequest {
  string game_id = 1;
  string season_id = 2;
  string user_id = 3;
  double raw_score = 4;
  string match_verification_token = 5;
}

message SubmitScoreResponse {
  int64 global_rank = 1;
  double verified_score = 2;
  bool is_new_high_score = 3;
}

message GetTopRankingsRequest {
  string game_id = 1;
  string season_id = 2;
  int32 top_n = 3;
}

message LeaderboardEntry {
  int64 rank = 1;
  string user_id = 2;
  string username = 3;
  string avatar_url = 4;
  double score = 5;
}

message GetTopRankingsResponse {
  repeated LeaderboardEntry entries = 1;
  int64 total_participants = 2;
  int64 last_updated_epoch_ms = 3;
}

message GetSurroundingRanksRequest {
  string game_id = 1;
  string season_id = 2;
  string user_id = 3;
  int32 radius = 4;
}

message GetSurroundingRanksResponse {
  int64 user_rank = 1;
  repeated LeaderboardEntry surrounding_entries = 2;
}

---