Design Discord's Trillions of Messages Storage — Cassandra to ScyllaDB Architecture

1. Problem Statement & Scope Clarification

System Mission

Design Discord's real-time messaging storage engine capable of persisting trillions of chat messages across millions of guilds (servers) and channels. The platform must maintain consistent $< 5\text{ ms}$ $P99$ read latencies, handle extreme write spikes (e.g., millions of messages/sec during major gaming events or Midjourney bot usage), and eliminate unpredictable JVM Garbage Collection pauses.

Functional Requirements

1. Chat Message Ingestion (SendMessage): Persist chat messages with strict millisecond-level monotonicity per channel.
2. Channel Message History (FetchMessages): Retrieve batches of $50-100$ messages before/after a given Snowflake ID with $P99 < 5\text{ ms}$.
3. Message Edit & Delete Mutations: Support atomic updates and soft-deletes without creating read-degrading tombstone storms.
4. Hot Partition Isolation: Prevent mega-channels (e.g., $500,000$ active users in a single bot channel) from starving neighboring channels sharing the same database node.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.999\%$ read/write availability across multi-AZ clusters.
• Ultra-Low Read Latency: $P50 < 1.5\text{ ms}$, $P99 < 5.0\text{ ms}$ (eliminating 2-second tail latency spikes).
• Linear Horizontal Scalability: Add storage nodes seamlessly without stopping traffic.

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Traffic & Storage Scale

• Total Persisted Messages: $> 1,000,000,000,000$ ($1\text{ Trillion messages}$).
• Peak Ingestion Rate: $5,000,000\text{ Messages/sec}$.
• Average Message Size: $250\text{ bytes}$ (Snowflake ID, Author ID, Channel ID, Text, Embed metadata).
• Total Raw Storage: $\text{Total Raw Storage} = 10^{12} \times 250\text{ bytes} = 250\text{ TB}$
• Replication Factor (RF = 3): $\text{Total Cluster Storage} = 250\text{ TB} \times 3 = \mathbf{750\text{ TB NVMe SSD}}$

3. High-Level Architecture & Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

4. API Interface Design & Wire Protocol

protobufCopy
syntax = "proto3";

package discord.messages.v1;

service MessageStoreService {
  rpc SendMessage (SendMessageRequest) returns (SendMessageResponse);
  rpc FetchMessages (FetchMessagesRequest) returns (FetchMessagesResponse);
}

message MessageRecord {
  int64 message_id = 1; // Discord Snowflake
  int64 channel_id = 2;
  int64 author_id = 3;
  string content = 4;
  int64 created_at_epoch_ms = 5;
  bool is_pinned = 6;
}

message FetchMessagesRequest {
  int64 channel_id = 1;
  int64 before_message_id = 2;
  int32 limit = 3; // Default 50, Max 100
}

message FetchMessagesResponse {
  repeated MessageRecord messages = 1;
}