Design Discord's Trillions of Messages Storage — Cassandra to ScyllaDB Architecture

1. Problem Statement & Scope Clarification

System Mission

Design Discord's distributed real-time messaging storage engine capable of persisting trillions of chat messages across millions of guilds (servers) and channels. The platform must maintain consistent sub-5ms $P99$ read latencies, handle extreme write bursts ($5\text{M}+$ messages/sec globally during major gaming events or bot activity), prevent hot-partition starvation from mega-channels ($500\text{k}+$ concurrent users in a single channel), and eliminate JVM Garbage Collection stop-the-world pauses through a C++ Seastar shared-nothing thread-per-core architecture.

Synthesizing vector architecture diagram...

Functional Requirements

1. Chat Message Persistence (SendMessage): Ingest messages with strict chronological ordering guaranteed per channel using 64-bit Snowflake IDs.
2. Channel Message History (FetchMessages): Retrieve paginated historical windows ($50-100$ messages before/after a given message ID) with $P99 < 5\text{ ms}$.
3. Atomic Message Mutations (EditMessage & DeleteMessage): Update text content and execute soft deletions without triggering tombstone scan storms.
4. Hot-Partition Isolation: Isolate mega-channels and bot flooding (e.g., Midjourney, gaming announcement channels) so they cannot degrade neighboring channels hosted on the same physical storage nodes.
5. Cold Message Archival: Seamlessly transition historical messages older than 2 years to Amazon S3 while preserving queryability.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.999\%$ read and write availability across multi-AZ AWS deployments.
• Ultra-Low Latency:
  • Write Ingestion: $P50 < 2\text{ ms}$, $P99 < 8\text{ ms}$.
  • History Read ($50$ messages): $P50 < 1.5\text{ ms}$, $P99 < 5.0\text{ ms}$ (zero GC-induced pauses $> 50\text{ ms}$).
• Throughput Capacity: Sustain $5,000,000\text{ writes/sec}$ peak ingestion and $15,000,000\text{ reads/sec}$ peak history lookups.
• Data Durability: Zero data loss ($RPO = 0$ for acknowledged writes, $RF = 3$ with LOCAL_QUORUM).
• CAP / PACELC Classification: AP system under CAP; PA/EL under PACELC (Partition $\to$ Availability; Else $\to$ Latency over Consistency). Chat history privileges real-time read/write availability over linearizable global order.

Out-of-Scope

• Voice/Video WebRTC peer-to-peer media routing (handled by Discord voice servers).
• Real-time user presence status counters (handled by separate in-memory Redis clusters).
• Content moderation machine learning pipelines (processed asynchronously via Kafka).

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Traffic Scale Calculations

• Active Guilds (Servers): $25,000,000$ active Discord servers.
• Active Text Channels: $200,000,000$ channels.
• Peak Write Throughput (Ingestion): $\text{Peak Write QPS} = \mathbf{5,000,000\text{ messages/sec}}$ The 5M/s figure is a short burst ceiling (raids, bot floods, tournament finales); the sustained average is $\approx 15,000$ messages/sec: $\text{Daily Message Volume} = 15,000 \times 86,400\text{ s} \approx \mathbf{1.3\text{ Billion messages/day}} \quad (\approx 0.47\text{ Trillion/year})$ Five years of that growth ($\approx 2.4\text{ T}$) on top of the 1 T baseline is what yields the 3 T horizon used below. Peak-to-average ratios of $300\times$ are exactly why the write path needs slowmode at the gateway and the Rust write buffer in front of ScyllaDB.
• Peak Read Throughput (Message History & Polling): Average read-to-write ratio of $3:1$ (channel joins, desktop reloads, scrolling up in history): $\text{Peak Read QPS} = 5,000,000 \times 3 = \mathbf{15,000,000\text{ queries/sec}}$

Storage Sizing & Cluster Growth (5-Year Horizon)

• Total Persisted Historical Messages: $1,000,000,000,000$ ($1\text{ Trillion messages}$) active baseline; growing to $3\text{ Trillion}$ over 5 years.
• Average Message Size:
  • message_id (Snowflake): 8 bytes
  • channel_id: 8 bytes
  • author_id: 8 bytes
  • content (UTF-8 text): 150 bytes average
  • embeds & metadata overhead: 60 bytes
  • Indexing & row metadata overhead: 16 bytes
  • Total per message record: $\approx \mathbf{250\text{ bytes}}$.
• Raw Storage Calculation (1 Trillion Messages): $\text{Raw Storage} = 10^{12} \times 250\text{ bytes} = 250,000,000,000,000\text{ bytes} = \mathbf{250\text{ TB}}$
• Replication Factor ($RF = 3$ Across Multi-AZ): $\text{Replicated Hot Storage} = 250\text{ TB} \times 3 = \mathbf{750\text{ TB NVMe SSD}}$
• 5-Year Growth Volume (3 Trillion Messages): $\text{5-Year Replicated Volume} = 750\text{ TB} \times 3 = \mathbf{2.25\text{ Petabytes NVMe SSD}}$

Network Bandwidth Calculations

• Write Ingress Bandwidth: $\text{Peak Write Ingress} = 5,000,000\text{ msgs/sec} \times 250\text{ bytes} = 1.25\text{ GB/sec} = \mathbf{10.0\text{ Gbps}}$
• Read Egress Bandwidth (History fetches averaging 50 messages $= 12.5\text{ KB}$): Assuming $10\%$ of reads fetch full 50-message windows ($1.5\text{M fetches/sec}$): $\text{Peak Read Egress} = 1,500,000 \times 12.5\text{ KB} = 18.75\text{ GB/sec} = \mathbf{150\text{ Gbps}}$

Cache & Working Set RAM Calculation

• Active Channel Working Set: Active working set comprises messages sent across the top $5\%$ of active channels within the past 2 hours ($10\text{M channels} \times 20\text{ messages} \times 250\text{ bytes} \approx 50\text{ GB}$).
• Rust Data Service Singleflight Buffers: $15\text{M QPS}$ reads coalesced across 120 Rust proxy instances: $\approx \mathbf{2\text{ TB RAM}}$ aggregate cache footprint.
• ScyllaDB Row & Key Cache: $100\text{ nodes} \times 128\text{ GB RAM allocated to row cache} = \mathbf{12.8\text{ TB RAM}}$ globally.

Fleet Sizing & Compute Infrastructure

• ScyllaDB Nodes (AWS EC2 i3en.12xlarge — 48 vCPU, 384 GB RAM, $4 \times 7.5\text{ TB}$ NVMe SSD $= 30\text{ TB NVMe}$): Usable storage per node at $70\%$ disk high-water mark: $30\text{ TB} \times 0.70 \approx 21\text{ TB}$. $\text{Nodes Required for Storage} = \frac{750\text{ TB}}{21\text{ TB/node}} \approx \mathbf{36\text{ nodes}}$ $\text{Nodes Required for Throughput (Each node serves 150k CQL QPS)} = \frac{15,000,000\text{ Read QPS}}{150,000\text{ QPS/node}} = \mathbf{100\text{ nodes (34 per AZ across 3 AZs)}}$ (Throughput governs sizing today: deploy 102 i3en.12xlarge instances. At the 5-year 2.25 PB mark storage catches up: $2.25\text{ PB} / 21\text{ TB} \approx 107$ nodes, so the ring grows to ~110 by year 5 regardless of QPS.)

---

3. AWS-First High-Level Architecture

Synthesizing vector architecture diagram...

Follow a message from send to storage. The client keeps a WebSocket to the Elixir gateway cluster for real-time events and sends messages through the Rust API fleet. In the "Rust Message Data Service Layer" panel, every database call passes through a data service that routes requests for the same channel to the same instance: if 1,000 users open a busy channel at once, the singleflight coalescer turns 1,000 identical reads into one database query, and writes are batched briefly in memory. Only then do requests reach ScyllaDB in the "ScyllaDB Enterprise Storage Tier" panel, a ring spread across three AZs. Message events also stream through Kinesis and Firehose into S3 for archiving. The data service layer is what protects the database from hot channels, the main cause of Discord's earlier Cassandra problems.

Data Flow Walkthrough

1. Message Write Ingestion (SendMessage):
   • The user types a message and hits enter. The client sends an HTTP POST /v1/channels/{channel_id}/messages via the NLB to the Message API Service.
   • The API generates an authoritative Discord Snowflake ID (64-bit integer containing a 41-bit timestamp, 10-bit worker ID, and 12-bit sequence counter).
   • The message payload forwards over internal gRPC to the Rust Message Data Service.
   • The Rust Data Service computes the bucket index based on the Snowflake timestamp: $\text{bucket} = (\text{snowflake} \gg 22) / (86,400,000 \times 10)$ (10-day time buckets).
   • The write is executed asynchronously via the CQL binary protocol to the ScyllaDB cluster using consistency level LOCAL_QUORUM.
   • The Rust Data Service stages the newly written message into its local in-memory LRU cache and publishes the event to the Elixir Gateway cluster for real-time WebSocket broadcast to channel subscribers.
2. Message History Read (FetchMessages):
   • A user opens a channel. The client requests the last 50 messages before a specific Snowflake ID.
   • The request hits the Rust Message Data Service.
   • Singleflight Coalescing: If 500 users in a gaming guild open the channel simultaneously, the singleflight engine detects identical queries (channel_id, bucket, before_id) and coalesces them into a single in-flight CQL query.
   • If the request hits the local in-memory LRU cache, it returns in $< 300\text{ }\mu\text{s}$.
   • On a cache miss, the Rust service queries ScyllaDB with LOCAL_QUORUM. ScyllaDB's core-pinned Seastar thread reads directly from NVMe SSD via Linux AIO/DIO without going through the Linux page cache, returning 50 sorted rows in $< 1.2\text{ ms}$.

---

End-to-End Request Tracing Walkthrough

---

4. API Interface Design & Wire Protocol

1. Send Message gRPC Specification (SendMessage)

High-throughput internal RPC between Message API and Rust Data Service.

protobufCopy
syntax = "proto3";

package discord.messages.v1;

service MessageStoreService {
  rpc SendMessage (SendMessageRequest) returns (SendMessageResponse);
  rpc FetchMessages (FetchMessagesRequest) returns (FetchMessagesResponse);
  rpc EditMessage (EditMessageRequest) returns (EditMessageResponse);
  rpc DeleteMessage (DeleteMessageRequest) returns (DeleteMessageResponse);
}

message MessageRecord {
  int64 message_id = 1;       // 64-bit Discord Snowflake
  int64 channel_id = 2;
  int64 author_id = 3;
  string content = 4;
  int64 created_at_epoch_ms = 5;
  int64 edited_at_epoch_ms = 6;
  bool is_pinned = 7;
  repeated string attachment_urls = 8;
}

message SendMessageRequest {
  int64 channel_id = 1;
  int64 author_id = 2;
  string content = 3;
  repeated string attachment_urls = 4;
  string idempotency_key = 5;
}

message SendMessageResponse {
  MessageRecord message = 1;
  int32 bucket = 2;
}

message FetchMessagesRequest {
  int64 channel_id = 1;
  int64 before_message_id = 2; // Fetch messages older than this Snowflake
  int64 after_message_id = 3;  // Fetch messages newer than this Snowflake
  int32 limit = 4;             // Default 50, Maximum 100
}

message FetchMessagesResponse {
  repeated MessageRecord messages = 1;
  bool has_more = 2;
}

---

2. Client REST Wire Contract (GET /v1/channels/{channel_id}/messages)

httpCopy
GET /v1/channels/9482019481029384/messages?before=1549890308517425161&limit=50 HTTP/1.1
Host: discord.com
Authorization: Bot MTk4MjM0OTAxMjM...
X-Discord-Locale: en-US
User-Agent: Discord-Android/210.0

HTTP/1.1 200 OK
Content-Type: application/json
X-RateLimit-Limit: 50
X-RateLimit-Remaining: 49
X-RateLimit-Reset-After: 0.200

[
  {
    "id": "1549890119773745153",
    "channel_id": "9482019481029384",
    "author": {
      "id": "2837192847102938",
      "username": "GamerOne",
      "avatar": "a_98fbc10293"
    },
    "content": "GG everyone, outstanding raid!",
    "timestamp": "2026-09-16T21:10:00.120Z",
    "edited_timestamp": null,
    "tts": false,
    "mention_everyone": false,
    "pinned": false,
    "attachments": []
  }
]

---

3. Status Codes & Error Contracts

---

5. Data Models & Storage Architecture

Why Cassandra Failed at Scale and Why ScyllaDB Succeeded

1. JVM Garbage Collection (GC) Stop-the-World Pauses: Cassandra is written in Java. When millions of message objects are allocated and discarded during write bursts, the JVM garbage collector (even with ZGC or G1GC) triggers periodic GC pauses ranging from $500\text{ ms}$ to $5,000\text{ ms}$. This caused catastrophic P99 tail latency spikes and false-positive node gossip timeouts.
2. Seastar C++ Shared-Nothing Architecture: ScyllaDB is written in C++ using the Seastar framework. It employs a thread-per-core architecture: each CPU core owns a completely isolated slice of memory, network queues, and NVMe disk channels. Zero cross-core thread locking or lock contention exists. Direct I/O (O_DIRECT) bypasses the OS page cache entirely, delivering predictable sub-5ms P99 latencies under full saturation.
3. Partition Bucketing Strategy: Storing all messages for a channel under PRIMARY KEY (channel_id, message_id) causes unbounded partition growth. Highly active channels accumulate hundreds of millions of messages, creating multi-gigabyte partitions that cause Cassandra compaction stalls, read timeouts, and node crashes. Discord solved this via Time Bucketing.

Synthesizing vector architecture diagram...

Read the relationships top-down: a GUILD (server) contains CHANNELs, and each channel stores MESSAGEs written by USERs. The key design is in MESSAGE: its partition key is (channel_id, bucket), where bucket is a fixed time window, and message_id sorts messages inside the partition. Without the bucket, a busy channel's partition would grow forever, and huge partitions are slow to read, compact and repair; with it, each partition covers one time window and stays bounded. Loading a channel reads the newest bucket first and moves to older buckets only if the user scrolls back. Because message_id is a time-ordered Snowflake ID, messages come back already in time order.

ScyllaDB Production Schema with Time Bucketing

sqlCopy
CREATE KEYSPACE discord_chat 
WITH replication = {
    'class': 'NetworkTopologyStrategy', 
    'us-east-1': 3
};

CREATE TABLE discord_chat.messages (
    channel_id bigint,
    bucket int,                       -- 10-day time bucket derived from Snowflake ID
    message_id bigint,               -- Snowflake ID: sorts chronologically descending
    author_id bigint,
    content text,
    created_at_epoch_ms bigint,
    edited_at_epoch_ms bigint,
    is_deleted boolean,
    has_attachments boolean,
    attachment_metadata text,
    PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC)
AND compaction = {
    'class': 'TimeWindowCompactionStrategy',
    'compaction_window_unit': 'DAYS',
    'compaction_window_size': 10,
    'timestamp_resolution': 'MILLISECONDS'
}
AND default_time_to_live = 0; -- Retain forever in hot tier

Bucketing Formula & Partition Sizing

The bucket index is computed deterministically from the Snowflake ID:

$$\text{Timestamp MS} = \text{message\_id} \gg 22 \quad (\text{milliseconds since the Discord epoch } 1,420,070,400,000 = 2015\text{-}01\text{-}01)$$

$\text{Bucket Number} = \frac{\text{Timestamp MS}}{10 \times 86,400,000\text{ ms}} = \frac{\text{Timestamp MS}}{864,000,000\text{ ms}}$ The bucket is computed on the epoch-relative value directly (Discord's make_bucket(snowflake) = (snowflake >> 22) / BUCKET_SIZE); adding the Unix epoch back in would still be deterministic but would not match the value the Rust Data Service derives in §3 and the matrix in §6.4. A message sent on 2026-09-16 lands in bucket 427.

• Partition Size Invariant: In a high-traffic channel generating $10,000$ messages per day, a 10-day bucket caps the partition at $100,000$ rows ($\approx 25\text{ MB}$). This stays well below the recommended $100\text{ MB}$ ScyllaDB partition limit, eliminating wide-partition compaction stalls.

---