Discord: Storing Trillions of Messages (MongoDB → Cassandra → ScyllaDB)

This page is one interview loop in three rounds, built on a real company's public engineering history. All three rounds design the same system: the store that keeps every chat message Discord has ever carried. Each round is an era of that store, and each era ended because the one before it failed in a specific, published way.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: Why Is Chat History Hard?

A Diary That Never Stops Growing

Discord decided early "to store all chat history forever", so a user can come back years later, on any device, and scroll up. Think of every channel as a diary:

• It only grows. New pages are added at the end, every second, forever. Nothing expires.
• Almost every read is the last page. Opening a channel means "show me the latest 50 messages". Scrolling up means "the 50 before this one".
• Diaries are wildly uneven. Discord described three kinds of servers in 2017: voice-heavy servers that send "a message or two every few days"; private text servers with 100,000 to 1 million messages a year; and large public servers with "thousands of members sending thousands of messages a day".

Synthesizing vector architecture diagram...

Read it as three eras. Each move answered a failure of the store before it: RAM (MongoDB), then hot partitions, garbage-collection pauses and maintenance toil (Cassandra).

A few words we'll use all page:

What Makes It Hard

• Reads are random. Discord found its reads were "extremely random" with a read/write ratio of "about 50/50" (2017). A quiet channel's 50 messages can be spread over months of disk.
• Traffic is concentrated. One announcement in a server with hundreds of thousands of members sends a crowd to the same few rows at the same moment.
• Deletes are expensive. A moderator deleting a million spam messages can hurt a database more than writing them did.

The Question the Whole Loop Answers

How do we store an ever-growing, very uneven pile of messages so that "latest in this channel" is always fast?

The answer grows every round:

• Round 1: match the store to the access pattern: a wide-column database, one partition per channel per 10-day time bucket, rows sorted newest first by Snowflake ID.
• Round 2: learn what that layout costs at scale: hot partitions, tombstones from deletes, compaction backlogs and GC pauses; tune consistency and repairs, and count the toil.
• Round 3: put a routing, request-coalescing data service in front of the database, move to ScyllaDB (no garbage collector, one shard per core), and migrate trillions of messages without downtime.

---

Round 1 · Mid-level · "Era 1: Outgrowing MongoDB"

~35 min · SDE II (L5) · 2015–2016 · 100M stored messages and growing fast · ~4,200 message writes/s at the peak (planning figure) · one replica set → a 12-node cluster · stop falling over as data grows

R1.1 Establish Design Scope

The interviewer sets the scene: "It's late 2015. We built the first version of our chat app in under two months. Everything, messages included, lives in one MongoDB replica set. We just passed 100 million stored messages and latency has become unpredictable. Design the message store we move to."

Out of scope for this round: trillions of messages, channels with hundreds of thousands of readers, mass deletes, search.

R1.2 Functional Requirements, Derived Step by Step

Discord's 2017 post adds that more random reads were coming: "view your mentions for the last 30 days then jump to that point in history", pinned messages, and full-text search. Every one of them is a jump to an arbitrary point in a channel.

Not yet: trillions of messages, mega-channels, bulk moderation, search (a separate system).

R1.3 Non-Functional Requirements: the Questions

We name each quality first; the numbers come in R1.7. Discord's own list (2017) is the best guide, and we keep its words where they are precise.

• Linear scalability. "We do not want to reconsider the solution later or manually re-shard the data." Adding capacity must mean adding machines.
• Automatic failover. Losing a node must not page anyone.
• Low maintenance. "We should only have to add more nodes as data grows."
• Predictable performance. The API's p95 alarm is 80 ms, and there is no cache to hide slow reads.
• Not a blob store. Appending to a per-channel blob thousands of times a second would mean constant deserializing and rewriting.
• Proven, open source. "We love trying out new technology, but not too new."

R1.4 The API

This API is illustrative: it is modeled on Discord's public API, whose GET /channels/{channel.id}/messages takes before, after or around (only one at a time) and a limit from 1 to 100, defaulting to 50, and returns messages newest first. The internal paths and fields are ours.

Fetch history

httpCopy
GET /v1/channels/175928847299117000/messages?before=175928847299117063&limit=50 HTTP/1.1
Authorization: Bearer <session token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

[
  { "id": "175928847299117062", "channel_id": "175928847299117000", "author": { "id": "80351110224678912" }, "content": "gg", "edited_timestamp": null },
  { "id": "175928847299117061", "channel_id": "175928847299117000", "author": { "id": "41771983423143937" }, "content": "one more?", "edited_timestamp": null }
]

Send a message

httpCopy
POST /v1/channels/175928847299117000/messages HTTP/1.1
Authorization: Bearer <session token>
Content-Type: application/json

{ "content": "one more?", "nonce": "c1-7f3a92" }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "id": "175928847299117061", "channel_id": "175928847299117000", "content": "one more?", "nonce": "c1-7f3a92" }

• IDs are strings in JSON. Discord's documentation says Snowflakes are "always returned as strings in the HTTP API to prevent integer overflows in some languages" (a JavaScript number is exact only up to 2^53).
• The message ID is the cursor. Because a Snowflake starts with its timestamp, before=<id> means "older than this message" and needs no separate page token. Discord's docs point out you can even build a cursor from a time: (timestamp_ms - DISCORD_EPOCH) << 22.
• The nonce is a value the client picks so it can match the server's echo to the message it showed optimistically. (Discord's public API has a nonce field; how Discord uses it for retries internally isn't something we rely on.)

Status codes

Recap

• Five operations, all scoped to one channel.
• Every read is "a slice of one channel, by message ID"; the message ID doubles as the page cursor.

R1.5 Design Evolution: From One Replica Set to a Partitioned Store

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

One MongoDB replica set (a primary that takes every write, plus secondaries that copy it) holds every collection. Messages have one compound index on channel_id and created_at.

Synthesizing vector architecture diagram...

One primary holds every message and the whole index. It works while the hot data and the index fit in that machine's RAM.

Discord published that this was intentional: build quickly, but "always with a path to a more robust solution", and it had decided not to use MongoDB sharding, which it considered "complicated to use and not known for stability".

Step 1.1: The Index No Longer Fits in RAM

Primitive: Database Sharding and Partition Keys · Loop: Design a Distributed Key-Value Store (partitioning, replication and quorums from the inside)

Step 1.2: Order Messages Without a Coordinator

Primitive: Distributed Unique ID Generators · Loop: Design a Distributed Unique ID Generator (clock steps, worker numbers and the exhaustion date, in depth) · Drill: The NTP Sync That Generated Duplicate Order IDs (answered here: a backward clock step and the fix above; why not UUIDv4 in the wrong answers above)

Step 1.3: One Channel's Partition Grows Forever

Where a quiet channel's read goes. A voice-heavy server sends, say, one message every two days (assumption: 0.5 a day). Fifty messages span 100 days, so "the latest 50" walks back through about 11 buckets, one query each, most of them nearly empty. Discord accepted this: "for active Discords enough messages are usually found in the first partition and they are the majority." Round 2 adds a fix.

Synthesizing vector architecture diagram...

A quiet channel's read walks backwards through buckets, one partition query each, until it has 50 messages or reaches the channel's creation bucket. A busy channel stops at the first box.

Step 1.4: Reads Must Be "Latest First"

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

Every request names a channel, so it lands on the three replicas that own that channel's current bucket. Any node can coordinate a request; data spreads over the ring by partition key.

The table (Discord's published minimal schema; the real one has 16 columns):

sqlCopy
CREATE TABLE messages (
   channel_id bigint,
   bucket int,
   message_id bigint,
   author_id bigint,
   content text,
   PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

Synthesizing vector architecture diagram...

One channel owns many small partitions, one per 10-day window; inside each, rows sort by Snowflake, newest first.

How we switched (published). Discord imported existing messages, then "set up our code to double read/write to MongoDB and Cassandra" (a dark launch: the new store gets real traffic while the old one still serves users). After it held up, Cassandra became the primary store and MongoDB was "phased out within a week". Measured in testing: writes "sub-millisecond" and reads "under 5 milliseconds", consistently for a week, and a jump back a full year in a channel with millions of messages stayed fast. The dark launch also surfaced the first surprise (a message with a null author_id), which is Round 2's step 2.4.

Trace: send, then open the channel (our design, with Discord's key layout)

Synthesizing vector architecture diagram...

The write is acknowledged by two of three replicas; the third catches up in the background. The read touches one partition when the channel is busy.

The consistency level (LOCAL_QUORUM, two of three copies) is our choice for this round; Discord published in 2023 that it performs "reads and writes with quorum consistency level", which Round 2 relies on.

Sources for this round

• Vishnevskiy, How Discord Stores Billions of Messages, Discord blog, January 2017: store history forever; MongoDB replica set and compound index; 100M messages and RAM in November 2015; the read patterns and server types; the requirements; the choice of Cassandra; KKV; (channel_id, message_id); 100 MB partitions; 10-day buckets and the bucket function; the dark launch; measured latencies; 12 nodes, RF 3, nearly 1 TB per node; 40M, 100M and 120M+ messages a day.
• Discord Developer Documentation, API Reference: Snowflakes and Message resource: Get Channel Messages (checked September 2026): the ID layout, the epoch, IDs as strings, before/after/around and limit.
• Ingram, How Discord Stores Trillions of Messages, Discord blog, March 2023: quorum reads and writes; the minimal schema; reverse queries.

Everything else in this round (the peak factor, row sizes, the API shapes beyond the public fields, the consistency level in the trace) is our design or our assumption.

R1.7 Numbers

Figures marked published come from the sources above; the rest are assumptions or derived from them.

Write and read rates

Cluster load at the peak (12 nodes, RF 3; reads at LOCAL_QUORUM, our choice)

Both are small for a Cassandra node. Round 1's problem is data size and layout, not request rate.

Storage: a sanity check against the published cluster. Discord reported "nearly 1TB of compressed data on each node" on 12 nodes: about 12 TB, or 12 ÷ 3 = 4 TB of unique data. How many messages is that? Discord didn't say. Draw daily volume piecewise through the published points (about 10M a day in January 2016, our assumption; 40M a day in July 2016; 100M in December 2016; 120M in January 2017) and add up the days: 2016 held roughly 20 billion messages. Then 4 TB ÷ 20B ≈ 200 bytes per message, compressed, all overhead included. That's our estimate, not a published figure; it only says the published numbers hang together.

Growth. At 120M messages a day × 200 B = 24 GB of new unique data a day; × 3 copies = 72 GB a day across the cluster, or 72 ÷ 12 = 6.0 GB per node a day. A node gains another terabyte in about 1,000 ÷ 6.0 ≈ 167 days, sooner as traffic grows. Discord's plan: newer Cassandra versions handle more data per node, so it expected to go from 1 TB to 2 TB per node, and Cassandra 3's storage format "can reduce storage size by more than 50%". Then keep adding nodes.

Buckets. From step 1.3: 10-day buckets hold up to 200,000 rows at 500 B before reaching 100 MB. The busy channel at 5,000 messages a day uses 25 MB per bucket; the quiet channel at 0.5 a day holds 5 messages per bucket and walks about 11 buckets for 50 messages.

Synthesizing vector architecture diagram...

A 10-day bucket keeps even a very busy channel at or under 100 MB, while a quiet channel's buckets are almost empty: that's the price we pay on reads.

R1.8 Trade-Offs

MongoDB (one replica set) vs Cassandra for this access pattern

Bucket size

The right bucket is set by the busiest channel (size) and paid for by the quietest (queries). One global size is a compromise; Round 2 reduces the quiet channels' cost instead of changing the size.

Row per message vs a blob per channel. A blob per channel (one document holding many messages) would mean deserializing and rewriting the blob on every send, thousands of times a second: Discord's "not a blob store" requirement.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Starts from the access pattern ("latest 50 in one channel") and picks a store whose layout matches it.
• Designs the key: partition by channel, cluster by a time-sortable ID, newest first.
• Knows why created_at can't be the clustering key and why random UUIDs can't be the cursor.
• Bounds partition growth with time buckets, and can derive the bucket from the ID.
• Names what buckets cost (multi-partition reads for quiet channels) and accepts it on purpose.
• Keeps the old store running while the new one proves itself.

Follow-up questions

1. "Why not keep MongoDB and shard it?" Answer: that was an option Discord explicitly rejected at the time as "complicated to use and not known for stability". The deeper reason: even sharded, the data layout (documents plus a secondary index) doesn't put a channel's recent messages next to each other on disk the way a clustered partition does.

2. "The bucket size is global. A single channel suddenly gets 50,000 messages a day. What happens?" Answer: that channel's buckets grow to 50,000 × 10 × 500 B = 250 MB, over the target. Nothing breaks immediately, but the partition becomes slower to compact and read. Options: a per-channel bucket size (stored with the channel, so reads can compute it), or a smaller global size. Discord kept one size and fixed the quiet-channel cost instead.

3. "When does a Discord Snowflake run out?" Answer: 42 bits of milliseconds is 2^42 ms ≈ 139 years after 2015, around 2154. But stored in a signed 64-bit bigint, the top bit becomes the sign bit after 2^41 ms ≈ 69.7 years, around 2084, when new IDs would turn negative and sort below old ones. (That's our observation from the published layout, not something Discord has written about.)

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Era 2: Billions of Messages on Cassandra"

~40 min · Senior SDE (L6) · 2016–early 2022 · 12 → 177 Cassandra nodes (published) · billions, then trillions, of messages · about 4B messages sent a day (published, 2022) · ~93,000 writes/s at the peak (planning figure) · survive hot channels, mass deletes and the cluster's own maintenance

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "Discord began with every message in one MongoDB replica set, indexed on channel and creation time. At 100 million messages in late 2015, the data and index outgrew RAM and latency became unpredictable. Reads are random and about half of all traffic, and Discord didn't want a cache in front, so the store itself had to match the access pattern. It moved to Cassandra: the partition key is (channel_id, bucket), where a bucket is a 10-day window derived from the message's Snowflake ID, and rows are clustered by message_id newest first. 'Latest 50' reads one partition for a busy channel and walks back through buckets for a quiet one. Snowflakes are 64-bit IDs with a millisecond timestamp since 2015, so they sort by time and work as page cursors. The move ran as a dark launch, writing both stores. By January 2017 it was 12 nodes with 3 copies of everything. Open costs: a very busy channel still lives on three replicas at a time, deletes are writes, and the cluster's maintenance is ours to run."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: every read is one channel's slice, newest first, from three replicas.

Round 1 step summary

Open costs: hot channels land on three replicas; deletes are writes; repairs, compaction and node operations are ours; eventual consistency is the default.

R2.1 The Scope Raise

Interviewer: "It's a few years later. We have hundreds of thousands of people in some servers, and when one of them posts an announcement, a crowd opens the same channel at once. Moderators delete spam by the thousands. The cluster has grown from 12 nodes to well over a hundred, and our on-call engineers are firefighting it every week. Keep reads fast and make it survivable."

Scope change

The "Not yet" list from R1.2 comes back: mega-channels and mass deletes are in scope now.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

Bulk delete (the shape of Discord's public endpoint; the internal behavior is ours):

httpCopy
POST /v1/channels/175928847299117000/messages/bulk-delete HTTP/1.1
Authorization: Bearer <moderator session token>
Content-Type: application/json

{ "messages": ["175928847299117061", "175928847299117062"] }

httpCopy
HTTP/1.1 204 No Content

Discord's public endpoint requires the MANAGE_MESSAGES permission, takes 2 to 100 IDs, and refuses messages older than 2 weeks with 400 Bad Request. That last rule is a product limit worth noticing: it keeps mass deletes inside the newest buckets.

Edit

httpCopy
PATCH /v1/channels/175928847299117000/messages/175928847299117061 HTTP/1.1
Content-Type: application/json

{ "content": "one more game?" }

The store writes only the changed columns (content, edited_timestamp). In Cassandra every write is an upsert (insert-or-update, with no read first), which matters in step 2.4.

Edit history (our design; Discord hasn't published whether or how it keeps one): each edit overwrites content, so keeping earlier versions would mean a separate message_edits table keyed by (channel_id, bucket) like messages and clustered by (message_id, edit_timestamp), written alongside the edit and deleted with the message.

Consistency per call (our design, around Discord's published quorum choice)

Read repair, in one line: when the replicas a quorum read contacts disagree, the coordinator writes the newest version to the stale replica before answering (Cassandra calls this blocking read repair), so a read also heals what it touched.

R2.4 Design Evolution: Hot Partitions, Tombstones, Pauses and Toil

Step 2.1: A Viral Channel Makes One Partition Hot

Primitive: Database Sharding and Partition Keys · Loop: Design a Chat and Instant Messaging System (gateways, fan-out and channels with 150,000 members; this page covers only the store)

Step 2.2: Deleting Spam Slowed Reads to 20 Seconds

Go deeper: the 2-day timing chain. Everything that can delay a delete reaching a replica has to fit inside the grace period.

The rule that follows: a node that has been down longer than the grace period must not rejoin with its old data. Wipe it and rebuild it from the others; that's the standard operating rule for Cassandra-style stores. A replica is a safe repair source only if it has seen every delete up to the moment the others purged theirs. At 10 days, that rule rarely bites; at 2 days, a long weekend outage triggers it.

Primitive: Write-Ahead Log and LSM Trees · Loop primitive: LSM-Trees & Compaction (Part 2 on versions and tombstones, and Part 5's worked example of a tombstone that must stay)

Step 2.3: Latency Spikes of Seconds

Primitive: Write-Ahead Log and LSM Trees · Loop primitive: LSM-Trees & Compaction (Part 5, "Why writes stall", with the numbers) · Drill: The Time-Series Database That Froze on Flush (both questions answered in this step)

Step 2.4: A Message With No Author

Primitive: Distributed Consensus: Raft and Paxos · Loop: Design a Distributed Key-Value Store (Round 2: tunable consistency, collisions, conditional writes, hinted handoff and anti-entropy)

Step 2.5: Operations Eat the Team

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The data path barely changed from Round 1; the operations box grew. The gateway (see the chat loop) delivers new messages; the store serves history.

Trace: a hot channel

Synthesizing vector architecture diagram...

Nothing limits how many identical reads reach the three replicas at once. Other channels whose data shares those nodes suffer too.

Trace: a bulk delete, then a read

Synthesizing vector architecture diagram...

A mass delete makes the next reads more expensive, not cheaper, until compaction catches up after the grace period.

Sources for this round

• Vishnevskiy, How Discord Stores Billions of Messages, Discord blog, January 2017: the dark launch and the author_id race; tombstones, 10-day default, 12 null tombstones per message; the 2016 incident (10 s GC, 20 s load, millions deleted); 2-day grace with nightly repairs; empty-bucket tracking; interest in Scylla for repair times.
• Ingram, How Discord Stores Trillions of Messages, Discord blog, March 2023: 177 nodes early 2022; hot partitions and quorum; compaction backlog and the gossip dance; GC tuning and reboots; p99 figures on Cassandra; other clusters' faults.
• Oakley, How Discord Supercharges Network Disks for Extreme Low Latency, Discord blog, August 2022: 4 billion messages a day; Google Cloud.
• Discord Developer Documentation, Bulk Delete Messages (checked September 2026).
• Apache Cassandra documentation, cassandra.apache.org/doc: tombstones and gc_grace_seconds, hinted handoff, compaction strategies, lightweight transactions and serial consistency.

Discord didn't publish its interim hot-partition measures, its consistency choices per job, or the toil figures; those are ours.

R2.6 Numbers and Cost

All figures are assumptions unless marked published.

Writes

Raw write rate isn't what hurt. The pain was concentration (hot partitions), deletes and the runtime.

The cluster

One hot channel (from step 2.1)

Tombstones from a bulk delete (illustrative)

GC pauses and p99 (illustrative)

The cost that matters here is effort (COST 11). The machines were affordable; the people weren't. Step 2.5's table is the bill: pages, dances, reboots and repairs that grow with every node.

R2.7 Trade-Offs

Consistency levels (RF 3)

Quorum's price is visible in step 2.1: when a replica is slow, a quorum request that needs it is slow. ONE would hide a slow replica but lets a fetch miss an acknowledged message.

Bucket size, revisited. Changing the bucket size now means rewriting trillions of rows into new partitions, which is a migration. Tracking empty buckets fixed the quiet-channel cost without touching the layout.

Soft delete vs hard delete

A soft delete doesn't escape tombstones: clearing content writes one. It only trades a row tombstone for a cell tombstone plus a permanent row. Discord kept hard deletes and fixed the read path.

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Explains a hot partition from first principles: one key, three replicas, reads costlier than writes, and quorum spreading the pain.
• Uses Little's law to show how a slow node turns into a queue.
• Knows deletes are tombstones, why the grace period exists (so repair can't resurrect deletes), and the rule for a node down past it.
• Connects GC pauses to p99 with arithmetic, and compaction backlog to GC.
• Knows last-write-wins is per column, and the LWT alternative with its cost and its SERIAL/LOCAL_SERIAL choice.
• Treats operator toil as a cost to measure.

Follow-up questions

1. "Why did Discord lower the grace period instead of raising it?" Answer: a shorter grace period purges tombstones sooner, so mass deletes stop costing reads sooner. It's safe only because repairs ran every night, well inside 2 days. The price is the rule that a node down for more than 2 days must be rebuilt.

2. "Would LOCAL_ONE reads fix hot partitions?" Answer: they'd halve the replica reads (one instead of two) and hide one slow replica, but a crowd still hits the same three nodes, and a fetch may miss a just-acknowledged message. It's a 2× trade, not a fix.

3. "Why not TTL old messages away?" Answer: the product promise is history forever. Expiring rows would also create tombstones (an expired cell behaves like one until compaction).

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Era 3: Trillions, ScyllaDB and Data Services"

~45 min · Principal (L7) · 2020–2023 · trillions of messages on 177 Cassandra nodes → 72 ScyllaDB nodes (published) · ~93,000 writes/s and 100,000 history fetches/s at the peak (planning figures) · migrate everything with no downtime · predictable p99, far less toil

R3.0 Where We Left Off

Round 2 in 60 seconds. "The layout from Round 1 held: (channel_id, bucket) partitions of 10 days, newest first. But at 177 nodes and trillions of messages, Cassandra became a high-toil system. A crowd opening one channel makes its partition hot; reads cost more than writes, and with quorum reads and writes, every request to those three nodes slows down. Deletes are tombstones that reads scan past until compaction purges them after the grace period; Discord cut the grace period from 10 days to 2 because repairs ran nightly, skipped empty buckets, and stopped writing nulls, which had been 12 tombstones per message. GC pauses dominated the p99, compaction fell behind, and the 'gossip dance' took nodes out to compact. On Cassandra, history fetches ran at a p99 of 40 to 125 ms and inserts 5 to 70 ms. Open costs: nothing limits concurrency on a hot partition, the runtime pauses, and maintenance grows with every node."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: the same data path as Round 1, with the operations box now the biggest thing on the page.

Round 2 step summary

Open costs: unbounded concurrency on hot partitions, a garbage-collected runtime, maintenance that no longer fits its windows.

R3.1 The Scope Raise

Interviewer: "It's early 2022. Every one of our other databases has moved to ScyllaDB. The messages cluster is the last one on Cassandra: nearly 200 nodes and trillions of messages. When a big server posts an announcement to everyone, thousands of clients ask for the same messages at the same moment. We want predictable p99s and fewer nodes, and we want to move trillions of messages without any downtime, quickly, because we're firefighting the old cluster."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

An internal data-service API. Discord's data services have "roughly one gRPC endpoint per database query and intentionally contain no business logic". The names below are ours:

Permissions and product rules stay in the API monolith; the data service only runs queries.

Migration controls (our design; the phases are Discord's):

yamlCopy
messages_store:
  primary: cassandra          # cassandra | scylla
  dual_write: true            # write both stores; the primary's result is returned
  shadow_read_percent: 1      # compare this share of reads across both stores
  backfill:
    source: cassandra
    target: scylla
    preserve_write_timestamps: true
    checkpoint: per-token-range
  config_version: 17

• primary decides which store's answer the client gets. Flipping it is the cutover.
• dual_write starts before the backfill, so everything the backfill doesn't copy arrives through live writes.
• shadow_read_percent is Discord's "small percentage of reads to both databases".
• config_version lets every data-service instance report which settings it runs, so we can see when all of them have flipped.

R3.4 Design Evolution: A New Engine, a Shield and a Migration

Step 3.1: Tail Latency From the Database Runtime

Primitive: Write-Ahead Log and LSM Trees (ScyllaDB is also an LSM engine: commit log, memtables, SSTables and compaction) · Loop primitive: Write-Ahead Log, fsync & Group Commit (what an acknowledged write has to survive, and why the durable half of the mirror matters)

Step 3.2: Thousands of Identical Reads Hit One Partition

Synthesizing vector architecture diagram...

Three requests, one query. Client 2 waits 8 ms and client 3 waits 5 ms: joiners wait only for the remainder of the query already in flight, never longer than one query.

Primitive: Distributed Cache Patterns and Eviction (coalescing, single-flight and stampedes) · Primitive: Consistent Hashing (routing by channel) · Drill: The Product Page That Melted Redis (answered here: a cold-cache stampede is merged by coalescing concurrent misses; "cache it forever" is the second wrong answer above)

Step 3.3: Migrate Trillions of Messages Without Downtime

Primitive: Change Data Capture and the Outbox Pattern · Loop primitive: Change Streams & the Transactional Outbox (Part 1: why writing to two systems can disagree; Part 7: refill from the stored source) · Loop primitive: Leases, Fencing Tokens & Distributed Locks (Part 9: when idempotent writes let one worker win without a lock)

Trace: one message during the migration

Synthesizing vector architecture diagram...

Order doesn't matter: whichever arrives first, each cell ends with its newest version, because both paths carry the source write time.

Notice the moment between the dual-written edit and the backfill: ScyllaDB holds a row for m with content and no author_id. That's exactly the shape Round 2's cleanup treats as "an edit raced a delete; remove it". So that cleanup stays off on ScyllaDB until the backfill is complete and shadow reads are clean, including after the flip. Shadow reads compare; they don't repair. And any delete the cleanup issues on Cassandra during the migration must be dual-written like every other delete, or ScyllaDB keeps the broken row.

Step 3.4: Deletes Still Hurt

Loop primitive: LSM-Trees & Compaction (Part 5: compaction merges and when a tombstone may go)

Step 3.5: Can We Archive Old Messages?

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Every query now passes through a data service that routes by channel and merges identical concurrent reads; only then does it reach ScyllaDB. The dotted parts are the migration (done) and the archive (an option Discord said it hoped to avoid).

Our AWS translation (not Discord's setup). ScyllaDB on storage-optimized EC2 instances with local NVMe instance storage (for example the i4i family), 24 nodes in each of 3 AZs with one replica per AZ; data services on EKS; S3 for backups; CloudWatch for alarms. The super-disk idea maps to a RAID 1 between a local NVMe stripe and an EBS volume marked write-mostly; that's a translation of Discord's GCP design that we haven't seen Discord or AWS publish, so it would need its own testing. EC2 instance storage has the same property that drove Discord's design: its data doesn't survive the instance being stopped or its host failing.

Trace: an old-message jump

Synthesizing vector architecture diagram...

A jump into history is two range reads in one old bucket, one of them in reverse order: the query Discord needed ScyllaDB to make fast.

Sources for this round

• Ingram, How Discord Stores Trillions of Messages, Discord blog, March 2023: reasons for ScyllaDB; all other databases migrated by 2020; reverse queries; data services, coalescing and consistent-hash routing by channel; the super-disk cluster; dual writes, the Spark estimate, the Rust migrator (SQLite checkpoints, 9 days, 3.2M/s), the tombstone stall; shadow-read validation; the May 2022 cutover; 72 nodes, 9 TB each; p99 results; the World Cup final.
• Oakley, How Discord Supercharges Network Disks for Extreme Low Latency, Discord blog, August 2022: Local SSD and persistent disk trade-offs on GCP; md RAID 0 plus RAID 1 write-mostly; 375 GB local SSDs; bad-sector handling.
• Ingram, How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB, ScyllaDB Summit 2023 talk: the same story, including the hybrid-RAID1 storage topology and the Rust data service library and migrator.
• Vishnevskiy, How Discord Stores Billions of Messages, January 2017: the archive idea "we want to avoid".
• ScyllaDB documentation: Data definition: tombstone_gc (modes timeout, repair, disabled, immediate), Compaction strategies, Replace a dead node, nodetool toppartitions, and the query_tombstone_page_limit setting in ScyllaDB's configuration source (checked September 2026).

Discord hasn't published its data-service fleet size, its compaction strategy, its per-AZ layout, the migrator's read consistency or how it handled backfill timestamps and tombstone GC during the load; everything on this page about those is our design.

R3.6 Numbers and Cost

Figures marked published come from the sources above; everything else is our assumption or derived from one.

Before and after (published)

Synthesizing vector architecture diagram...

The biggest win isn't the median but the range: the Cassandra figures were ranges (40–125, 5–70); the ScyllaDB ones are single numbers.

How full is the new cluster? (our estimate) Suppose ScyllaDB holds 150 TB of unique, compressed message data (assumption). Three copies are 450 TB: 450 ÷ 648 = 69% of the disk, leaving room for compaction to work. If the cluster held "trillions" (say 2 trillion) messages in that, that's 150 TB ÷ 2 trillion = 75 bytes per message on disk. Discord didn't publish the fill level or the message count, so treat this as a reminder of how compact the data must be, not a fact.

The migration (published figures, our arithmetic)

So the 9-day estimate is consistent with a data set in the low trillions; the post doesn't give the exact count.

The load test nobody planned. Discord's 2023 post shows message sends during the December 2022 World Cup final: one spike per goal and per break, with the store "handling it perfectly". Discord didn't publish the peak rate, so we don't use it as a number; it's the published evidence that the coalescing tier and the new cluster held up under a global, synchronized spike.

Coalescing for one announcement (from step 3.2): 5,000 identical requests/s become 98 queries/s when routed to one instance, 283 with per-AZ routing, 2,500 without routing.

Data-service fleet (assumptions: 100,000 fetches/s and 93,000 writes/s at the peak; 20,000 requests/s per instance; survive losing an AZ)

Load on ScyllaDB at the peak (72 nodes, 24 per AZ)

Cross-AZ traffic in the AWS translation. Data moving between AZs costs $0.01/GB in each direction, so $0.02 for every GB that crosses. Assume 1 KB per replicated write on the wire, 20 KB per 50-message fetch response, and an average of 50,000 fetches/s (half the peak).

AZ-aware coordinators save about $34,560 a month and a ring per AZ another $34,560: about $69,000 a month together. The two are independent: a rack- and token-aware driver makes a local replica the coordinator even with one global ring, at no cost; the ring per AZ costs 3 queries instead of 1 per coalesced group. The per-AZ ring is our variant; Discord didn't publish how its routing treats zones.

Why no load balancer in front of the data services. A Network Load Balancer would process the same ~1 GB/s: 3,600 GB an hour, which is 3,600 NLCUs at 1 GB processed per NLCU-hour for TCP, × $0.006 per NLCU-hour (us-east-1) = $21.60 an hour × 720 hours (30 days, the month used across this page) = ≈ $15,550 a month by the bytes dimension alone. Worse, it balances connections, not channels, so it would scatter a channel's requests and undo coalescing. Routing belongs in the client library.

Running two clusters at once (quota check). During the migration both clusters run. In our translation, 72 new nodes of, for example, i4i.8xlarge (32 vCPUs each) need 72 × 32 = 2,304 vCPUs on top of the old cluster. Storage-optimized instances count toward the account's "Running On-Demand Standard instances" vCPU quota, so we check that quota in the region before provisioning (R3.9, command 7). 72 nodes is 24 per AZ, a whole number, so each AZ holds exactly one replica of every partition.

The archive, priced (translation, list prices in us-east-1). Suppose 60% of the 150 TB is cold (assumption): 90 TB.

Glacier Instant Retrieval bills objects smaller than 128 KB as 128 KB, charges a 90-day minimum, and charges $0.03/GB to retrieve plus $0.01 per 1,000 GETs: hence one file per channel per year, not per bucket. The saving is real only if the cluster can actually shrink, which depends on throughput too: freeing 270 TB takes the cluster from 69% to (450 − 270) ÷ 648 = 28% full, but the 72 nodes may still be needed for requests. That's the quantitative version of "we want to avoid doing this one".

R3.7 Trade-Offs

Change the database, or change the access pattern?

Discord did both, in the cheaper order: the data services first (they helped even on Cassandra), then the database. Either alone would have left half the problem.

A data-service tier vs direct database access

Migration strategies

Spark migrator vs a custom one. Discord's case is the interesting one: the generic tool was the safer choice until its estimate (three months) was longer than the firefighting on the old cluster could wait. Owning a fast data library made a custom migrator an afternoon's work. Without that library, the right answer would have been the three months.

Run ScyllaDB ourselves, or a managed service? Discord runs its own. In an AWS translation, the managed Cassandra-compatible option is Amazon Keyspaces. The trade: no nodes, repairs or compaction to run, against less control over storage layout, compaction and tail behavior, and a pricing model per request instead of per node. For a team whose main complaint was toil, it's a question worth asking; for a team whose main complaint was p99 and who had already moved every other cluster to ScyllaDB, staying self-managed was consistent.

What changed from Round 1. Round 1 matched the layout to the query: a partition per channel and bucket, newest first. Round 3 keeps that layout exactly and changes everything around it: the engine underneath (no pauses, a core per shard, a cache of its own) and a tier above that shapes the traffic before it reaches the partition. The lesson of the whole loop: the data model was right in 2016; what failed was the runtime and the lack of any limit on how hard one partition could be hit.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6

Replace a dead node REL 11 · OPS 10

1. Confirm it's down, not slow: nodetool status shows it DN from several other nodes (command 1). Note its Host ID.

2. Decide: return or replace. If the table uses timeout tombstone GC and the node has been down longer than gc_grace_seconds, never let it rejoin with its data; replace it. In repair mode that particular risk is gone, but a node that missed a long stretch of writes is still quicker to replace than to repair.

3. Start a new node with the same configuration and this line in its scylla.yaml, naming the dead node's Host ID (ScyllaDB's documented procedure):

   yamlCopy
   replace_node_first_boot: 675ed9f4-6564-6dbd-ca08-43fddce952de

4. Let it stream its data from the other replicas, then repair it (command 6), unless repair-based node operations are enabled for replace, in which case the replace already repaired.

5. Watch p99, pending compactions and hints drain before calling it done.

Go deeper: CLI playbook

Commands an on-call engineer runs one at a time; replace names with real ones.

textCopy
# 1. Node states and Host IDs, seen from this node
nodetool status

# 2. Pending and running compactions on this node
nodetool compactionstats

# 3. The hottest partitions of messages.messages over 10 seconds
nodetool toppartitions messages messages 10000

# 4. Read and write latency percentiles for requests this node coordinated
nodetool proxyhistograms

# 5. Table statistics, including SSTable counts
nodetool tablestats messages.messages

# 6. Repair through ScyllaDB Manager
sctool repair -c messages-prod

# 7. (AWS translation) The Running On-Demand Standard instances vCPU quota, before provisioning a second cluster
aws service-quotas get-service-quota --region us-east-1 --service-code ec2 --quota-code L-1216C47A

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Separates what a new database fixes (runtime pauses, repair speed, isolation) from what it doesn't (hot partitions), and fixes both.
• Designs request coalescing with its routing, quantifies the gain, and states its staleness bound.
• Plans a live migration as dual writes, backfill, verification, cutover and rollback, and names the correctness rules: original timestamps, tombstones kept during the load, caught-up source replicas, idempotent re-copies.
• Reads published results critically: ranges vs single numbers, capacity vs usage, what a 9-day estimate implies.
• Prices the options (transfer, load balancers, archives) and knows when not to build something.
• Keeps Discord's published facts separate from its own design.

Follow-up questions

1. "Why Rust for the data services and the migrator?" Answer: Discord's stated reasons: "fast C/C++ speeds without having to sacrifice safety", the Tokio async runtime, driver support for Cassandra and ScyllaDB, and code that is easy to write safely under concurrency, which is exactly what a coalescer is. And because the migrator reused the data-service library, it took "an afternoon".

2. "Discord runs this in one place. What if we had to run it in two regions?" Answer: Discord hasn't published a multi-region layout for messages, so this is our design. ScyllaDB would replicate asynchronously between data centers, with LOCAL_QUORUM in each, so the RPO on losing a region is the replication delay, usually seconds, not zero. Deciding a region is lost must not depend on that region, so detection runs from the other region and from outside. Moving users (DNS or routing) should be a gated, human-approved step, checked against capacity in the survivor and any data-residency rules. Any conditional write that must be globally consistent pays SERIAL Paxos across regions; LOCAL_SERIAL is only consistent within one. For RPO 0, writes would have to wait for copies in both regions (EACH_QUORUM in Cassandra and ScyllaDB), adding a cross-region round trip to every send; the chat loop's Round 3 designs that synchronous alternative and prices it.

3. "Isn't request coalescing just a cache with a very short TTL?" Answer: close, but with one important difference: it never serves a result to a request that arrives after the query finished. So there's no expiry to tune, no invalidation, and no stale data beyond one query's flight time.

4. "The p99 went from 40–125 ms to 15 ms. How much was the database and how much the data services?" Answer: Discord didn't split it, and the data services arrived first, while still on Cassandra, where they helped but "don't solve all of our problems". An honest answer says the published numbers can't separate the two.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. For delivering messages to clients (gateways, fan-out, huge channels), see the chat loop; for quorums, hinted handoff and anti-entropy from the inside, the key-value store loop; for Snowflake clocks in depth, the unique ID loop.

The Two Sentences That Matter Most

• Opening any round: "Every read is 'the latest messages in one channel', so I'll partition by channel and a time bucket, sort by a time-ordered ID newest first, and size the bucket so the busiest channel stays small."
• When scale arrives: "A new engine fixes pauses but not hot partitions, so I'll put a data service in front that routes by channel and turns identical concurrent reads into one query, then migrate with dual writes, a timestamp-preserving backfill and shadow-read verification."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels

Sources

All sources used on this page, oldest first.

• Vishnevskiy, How Discord Stores Billions of Messages, Discord blog, January 2017.
• Oakley, How Discord Supercharges Network Disks for Extreme Low Latency, Discord blog, August 2022.
• Ingram, How Discord Stores Trillions of Messages, Discord blog, March 2023.
• Ingram, How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB, ScyllaDB Summit 2023.
• Discord Developer Documentation, API Reference: Snowflakes and Message resource, checked September 2026.
• Apache Cassandra documentation, cassandra.apache.org/doc: tombstones, gc_grace_seconds, hinted handoff, compaction, lightweight transactions.
• ScyllaDB documentation: Data definition (tombstone_gc), Compaction strategies, Replace a dead node, nodetool toppartitions, and the configuration source for query_tombstone_page_limit and max_hint_window_in_ms, checked September 2026.
• AWS, Amazon S3 pricing (us-east-1, checked September 2026), and the EC2 data transfer and Elastic Load Balancing pricing pages, for the translation's cost figures.