Design a News Feed System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a News Feed?

You Already Use One: a Mailbox of Friends' Updates

Open any social app and the first screen is your home feed: posts from the people you follow, newest first (or "best" first). Think of it as a mailbox. Every time someone you follow posts, a copy of the envelope lands in your mailbox. When you open the app, you read the top of the pile.

Three operations make up the whole product:

What Makes It Hard

Two numbers pull against each other:

• One post may need to appear in millions of mailboxes. A singer with 60 million followers posts once, and 60 million feeds should change.
• Millions of people open their mailbox at the same moment. Reads outnumber posts by more than ten to one, and every read wants an answer in a fraction of a second.

There are only two places to do the work. We can do it when someone posts (copy the post into every follower's mailbox now, so reading is cheap later), or when someone reads (collect the posts from everyone they follow at the moment they open the app). Every component in this loop moves work from one side to the other.

The Question the Whole Loop Answers

Do we do the work when someone posts, or when someone reads, and who decides for each post?

The answer gets sharper every round:

• Round 1: do it when someone posts. We precompute every user's feed, and a read becomes one cheap lookup.
• Round 2: that breaks for celebrities, for users who come back after a month, and for deleted posts. So it becomes a per-author decision, and ranking arrives.
• Round 3: the hard part is no longer speed. It is privacy, deletion, geography and abuse, where "precomputed" must never mean "allowed".

---

Round 1 · Mid-level · "A Feed for a Growing Social App"

~35 min · SDE II (L5) · 1 region, 3 AZs · 1M DAU · ~175 feed reads/s peak · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Design the news feed for our social app." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Celebrities with millions of followers.
• Ranking by relevance.
• Privacy controls (friends-only posts, blocks).
• Edits and deletes showing up in feeds. Posts can be deleted, but we don't yet promise how fast they vanish from other people's feeds.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into a requirement:

Not yet: ranking, edits and deletes that disappear from feeds, and celebrities. The interviewer will bring them back.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Read latency. The feed is the first screen of the app. If it is slow, the whole app feels slow.
• Freshness. How long between a post and a follower seeing it? "About a minute" lets us do work in the background.
• Availability. If the feed is down, the app is empty. But a feed that is a few seconds stale is still useful, so we can prefer "slightly stale" over "error".
• Read-heavy scale. Each user reads several times a day and posts about once. Reads outnumber posts, so we should make reads cheap even if posts cost more.

R1.4 The API

Three endpoints, all behind the app's normal login (a bearer token on every call).

1. Upload the image, then create the post. Images never pass through our servers. The client asks for a pre-signed URL: a short-lived S3 upload link, signed with our credentials, that allows exactly one PUT of one object.

httpCopy
POST /v1/media/upload-url HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>
Content-Type: application/json

{ "content_type": "image/webp", "size_bytes": 184320 }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "media_id": "m_7Qz2",
  "upload_url": "https://media-uploads.s3.amazonaws.com/u/401/m_7Qz2.webp?X-Amz-Expires=300&X-Amz-Signature=...",
  "expires_in_s": 300
}

The client PUTs the image to upload_url, then creates the post:

httpCopy
POST /v1/posts HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>
Idempotency-Key: 7b8e1f20-945a-497b-95eb-36b17c5b6102
Content-Type: application/json

{ "text": "Just shipped v2!", "media_ids": ["m_7Qz2"] }

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{
  "post_id": "97301141913751557",
  "author_id": "401",
  "created_at": "2026-09-26T12:00:00.000Z",
  "text": "Just shipped v2!",
  "media": [{ "url": "https://cdn.example-social.com/u/401/m_7Qz2.webp", "width": 1080, "height": 1080 }]
}

The Idempotency-Key lets a client retry a timed-out request without creating the post twice. IDs are strings in JSON, because a 64-bit ID is bigger than JavaScript can hold exactly (see the unique ID generator loop).

2. Read the feed.

httpCopy
GET /v1/feed?limit=20&cursor=eyJ2IjoxLCJzIjoyMzE5ODQwMDAwMCwicCI6Ijk3MzAxMTQxOTEzNzUxNTU3In0.k3Jd9w HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "items": [
    {
      "post_id": "97301141913751557",
      "author": { "user_id": "401", "name": "Alex", "avatar_url": "https://cdn.example-social.com/a/401.webp" },
      "created_at": "2026-09-26T12:00:00.000Z",
      "text": "Just shipped v2!",
      "media": [{ "url": "https://cdn.example-social.com/u/401/m_7Qz2.webp", "width": 1080, "height": 1080 }]
    }
  ],
  "next_cursor": "eyJ2IjoxLCJzIjoyMzE5ODM5OTg3NiwicCI6Ijk3MzAxMTQxMzk2OTY0MzIwIn0.Qm8aZ1",
  "has_more": true
}

The cursor is opaque: the client stores it and sends it back, and must never build or edit one. That leaves us free to change what's inside (step 1.3 puts a position and a signature in it) without breaking any app version.

3. Follow and unfollow.

httpCopy
POST /v1/users/802/follow HTTP/1.1
Authorization: Bearer <token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "user_id": "802", "following": true, "followed_at": "2026-09-26T12:01:10.000Z" }

DELETE /v1/users/802/follow undoes it. Both are idempotent: following someone you already follow returns the same 200.

Recap

• Three operations: post (text + image via a pre-signed S3 upload), follow, and read the feed 20 posts at a time.
• Chronological order; no repeats and no gaps while scrolling; a cursor the client can't read.
• Freshness of about a minute is fine.
• Reads outnumber posts, so reads should be the cheap side.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From "Query on Read" to Precomputed Timelines

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

Store posts and follow edges in a database. On each feed read, look up everyone I follow, query each one's recent posts, merge them by time, and return the top 20.

Synthesizing vector architecture diagram...

What's good about it: it is simple, it is always fresh (a post is visible the instant it's stored), and posting costs one write. This is called fanout on read: the work of spreading a post to its readers happens when they read.

What it costs us: every feed read is about 200 queries plus a merge, and the read is only as fast as the slowest of those 200 queries.

Step 1.1: Feeds Take Seconds to Load

Primitive: Distributed Cache Patterns & Eviction

Step 1.2: Memory Explodes, and Edits Never Show

The post store in DynamoDB. We key posts by their own ID, because hydration only knows IDs, and BatchGetItem needs the table's primary key (it cannot read through a secondary index):

Primitive: Distributed Cache Patterns & Eviction

Step 1.3: Page 2 Repeats Posts From Page 1

Step 1.4: Posting Blocks on Hundreds of Timeline Writes

textCopy
fanout_worker, for each post-created event from the stream:
    post   = event.new_item
    score  = time part of post.post_id          # step 1.6
    for each page of followers(post.author_id):  # Query on the follow table's index, 1,000 at a time
        for each follower in page:
            in the cache, as one atomic step:
                if timeline tl:<follower> exists:
                    add (score, post.post_id)
                    trim to the newest 800         # step 1.5
    checkpoint the stream position                 # only after the cache confirmed every write

Checkpointing only after the writes are confirmed makes the worker at-least-once: if it crashes mid-post, the next worker replays that post from the last checkpoint, and the duplicate adds are harmless.

Primitive: Message Queues vs Event Streams · Primitive: Change Data Capture & the Outbox Pattern · Drill: CDC outbox dual-write drift

Step 1.5: Timelines Grow Forever

Step 1.6: A Retry Inserted an Older Post Above a Newer One

Primitive: Distributed Unique ID Generators · Loop: Design a Distributed Unique ID Generator

Round 1 Step Summary

Two costs stay open for Round 2: a returning user has no timeline, and the whole design assumes nobody has millions of followers.

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Follow the two paths. Writing: the post service stores the post and returns; the table's change stream carries it to the fanout workers, which read the author's followers and add the post ID to each existing timeline. Reading: the feed service does one range read on the reader's timeline and one batch read of 20 posts. Images never touch our servers: they go up to S3 through a pre-signed URL and come down through CloudFront.

The pieces:

• ECS services in three AZs behind an Application Load Balancer. The fanout workers use the Kinesis Client Library (KCL), which assigns each stream shard to one worker and keeps its progress (the checkpoint) in a small DynamoDB lease table.
• DynamoDB, on-demand capacity: one table for posts, follow edges and profiles, replicated across three AZs by the service.
• ElastiCache (Valkey or Redis OSS), cluster mode: timelines spread over shards by key; each shard has a primary and a replica in another AZ, with automatic failover.
• S3 + CloudFront for images.

The table design (one table, SocialCoreTable, and one global secondary index, GSI1; a GSI is a second copy of chosen items, keyed differently, that DynamoDB keeps in sync asynchronously):

GSI1SK is a Number attribute. Stored as a string, IDs would sort as text, and our IDs grow from 17 to 18 digits around 2026-10-03 ($10^{17} \div 2^{22}$ ms ≈ 276 days after the ID epoch), so every newer 18-digit ID would sort below the older 17-digit ones. (A zero-padded 20-digit string would also work.)

• "Who do I follow?" → Query PK = USER#me, SK begins_with FOLLOWS#.
• "Who follows X?" (fanout) → Query GSI1 where GSI1PK = FOLLOWED_BY#X, 1,000 at a time.
• "X's latest posts" (fallbacks) → Query GSI1 where GSI1PK = AUTHOR#X, newest first.
• "Posts by ID" (hydration) → BatchGetItem on PK = POST#<id>.

The timeline layout (one sorted set per user):

Trace 1: a post is created and fanned out.

Synthesizing vector architecture diagram...

The author's request ends at the 201. Everything after it happens in the background, and a crash anywhere below the line replays from the last checkpoint.

Trace 2: a feed read.

Synthesizing vector architecture diagram...

Two network round trips to data stores, each a single batched call. If tl:802 doesn't exist (a returning user), the feed service falls back to the baseline read once and builds the timeline as it goes.

R1.7 Numbers

Traffic (assumptions: each user opens the feed 5 times a day, and posts once a day; peak is 3× average)

Fanout writes (average 200 followers per post):

$$11.6 \times 200 \approx 2{,}300 \text{ timeline writes/s on average} \qquad 35 \times 200 = 7{,}000 \text{ at peak}$$

The largest account has 10,000 followers, so the biggest single post is 10,000 writes. A worker that sends writes to the cache in pipelined batches does tens of thousands per second (an assumption we confirm by load test), so even that post is spread in well under a second.

What the baseline would have cost. At 174 reads/s and 200 queries each, the query-on-read design sends about 35,000 queries a second to the database at peak. The timeline design sends 174 range reads to the cache and 174 batch reads (of 20 items) to DynamoDB.

Timeline memory.

A cache.r7g.4xlarge node has 105.81 GiB of memory. ElastiCache reserves 25% of it by default for non-data use (reserved-memory-percent), which leaves about 79 GiB ≈ 85 GB, and we plan to fill that to 80%, about 68 GB per shard. $180 \div 68 \approx 2.6$, so 3 shards, each with a primary and a replica in another AZ: 6 nodes.

For comparison, full posts in the timelines (step 1.2's wrong answer) would add ~550 bytes of post to every entry: about $1.2 \times 10^{9} \times 670$ B ≈ 800 GB raw. Five times the memory, before a single edit goes wrong.

Post storage. A post item is about 550 bytes (text, media URLs, IDs, timestamps, attribute names). $1\text{M/day} \times 550 \text{ B} = 550$ MB a day, about 200 GB a year, plus the same again in the GSI copy of the key attributes.

Media. Assume 20% of posts carry one ~200 KB image: $1\text{M} \times 0.2 \times 200 \text{ KB} = 40$ GB a day into S3, ~1.2 TB a month.

Media delivery. Assume each feed page loads 4 images (20 posts × 20%) at a ~60 KB phone-sized version: $5\text{M pages/day} \times 240 \text{ KB} = 1.2$ TB a day, ≈ 36.5 TB a month out of CloudFront.

Availability budget. 99.9% over a 30.4-day month allows $43{,}776 \text{ min} \times 0.001 \approx 43.8$ minutes of downtime.

Monthly cost (us-east-1 on-demand list prices, 730 hours a month; rounded)

Say the headline in the interview: the timeline cache is already the biggest line. We are trading memory for read speed on purpose. Round 2 will have to look hard at that trade.

R1.8 Trade-Offs

Cache vs database for timelines. We could keep timelines in DynamoDB instead: durable, no memory bill. But every fanout write would be a billed write request (2,300 a second on average, ~6 billion a month at $0.625 per million ≈ $3,800), and the feed read would be a Query instead of an in-memory range read. We keep them in the cache because a timeline is derived data: it can always be rebuilt from posts and follows, so losing it costs time, not correctness. That one fact makes the cache safe.

Chronological vs ranked. Chronological is simple to page, easy to explain and cheap. Ranked feeds show "the best" first, which products usually want, but they need a model, features and a latency budget. Round 2 adds ranking, on top of the timeline, not instead of it.

R1.9 Failure Modes

Drill: Message queue order pipeline

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Starts from query-on-read, names its cost (about 200 queries per read), and moves the work to write time with a reason.
• Stores IDs, not posts, in timelines, and hydrates in one batch read.
• Pages with a cursor on (score, post ID) and explains why offsets repeat and skip.
• Takes fanout out of the post request, and knows why a second write after the database is risky.
• Bounds timelines by depth and by inactivity, and knows the timeline can be rebuilt.
• Derives the traffic, fanout writes and memory, and sees that memory is the big line.

Follow-up questions

1. "A user follows a new account. Does their timeline get that account's old posts?" Answer: not through fanout, which only carries new posts. The follow service can queue a small backfill: read the new followee's latest 20 posts and add them to the follower's timeline (the same "add, then trim to 800" step). It's asynchronous, so the first refresh after following may not show them yet.

2. "Why not a message queue like SQS instead of a stream?" Answer: SQS would work for fanout jobs. We chose a stream because the event source is the table's own change log (no dual write), because a stream can be replayed from a position after a bug, and because more consumers (search, notifications) can read the same events later. A queue deletes each message once it's handled.

3. "What happens if the cache is completely empty after a big failure?" Answer: every read falls back to query-on-read and rebuilds that user's timeline. That is roughly the baseline's load, about 35,000 queries a second at peak in Round 1, which DynamoDB on-demand can absorb for a while. We would also slow the fanout (there's little point writing into timelines that don't exist yet, and the "add if exists" rule already skips them).

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "300M Daily Users and Celebrities"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 300M DAU · ~52K feed reads/s peak · 99.99% · feed P99 < 200 ms

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built a chronological feed for 1M daily users: about 35 posts and 175 feed reads a second at peak, one region, three AZs, 99.9%. We moved the work from read time to write time. Posts live once in DynamoDB, keyed by post ID. Every post change flows through Kinesis Data Streams for DynamoDB to fanout workers, which read the author's followers from a GSI and add the post ID to each follower's timeline: a sorted set in ElastiCache, scored by the time part of the Snowflake post ID, trimmed to 800 entries, and expiring 7 days after its owner last read it. A feed read is one range read plus one BatchGetItem of 20 posts. Paging uses a signed cursor on (score, post ID). Images go to S3 through pre-signed URLs and out through CloudFront. About $12,300 a month, and the timeline cache is the biggest line. Four costs are still open: a celebrity post would mean millions of writes; returning users have no timeline; a deleted post can linger in timelines; and the feed is chronological only."

Architecture v1, compact

textCopy
 app ──► ALB ──► post service ──► DynamoDB (POST#id) ──change stream──► Kinesis ──► fanout workers (KCL)
                                                                             │ Query GSI1 FOLLOWED_BY#author
                                                                             ▼
                                        ElastiCache: tl:<user> sorted set, ≤ 800 IDs, 7-day expiry refreshed on read
 app ──► ALB ──► feed service ──► 1. range read tl:<me> at/below cursor   2. BatchGetItem 20 posts
 app ──► S3 (pre-signed PUT);  CloudFront ──► S3 for images

Round 1 step summary

Open costs: celebrity posts, returning users, lingering deleted posts, and no ranking.

R2.1 The Scope Raise

Interviewer: "The app took off. We have 300 million daily users now, posting 100 million times a day. Some accounts have 60 million followers. Product wants a ranked feed, not a chronological one. When someone deletes a post, it has to disappear from feeds, and edits must show. People who come back after a month shouldn't stare at a spinner. We want 99.99%, and the feed's P99 under 200 milliseconds."

A scope raise is not the end of scoping. Before we fix anything, we ask back, and we say what each answer changes, just as in R1.1.

Scope change

The "Not yet" list from R1.2 is now mandatory: ranking, edits and deletes in feeds, and celebrities.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

Ranked feed. The same endpoint, with a mode. The first call ranks a set of candidates once and stores the ranked list for this feed session; later pages read from that stored list, so paging is stable even though scores would change between calls.

httpCopy
GET /v1/feed?mode=ranked&limit=20 HTTP/1.1
Authorization: Bearer <token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "items": [ { "post_id": "97301141913751557", "...": "..." } ],
  "next_cursor": "eyJ2IjoyLCJtIjoicmFua2VkIiwic2lkIjoiZnNfOGsyUSIsIm8iOjIwfQ.Zp1xQe",
  "has_more": true,
  "ranking": "model"
}

The cursor now carries a version, a mode, a session ID and an offset into the stored ranked list:

jsonCopy
{ "v": 2, "m": "ranked", "sid": "fs_8k2Q", "o": 20 }

An offset is safe here, unlike in step 1.3, because the stored list never changes during the session. "ranking": "chronological" tells the app when ranking fell back (step 2.6). The chronological mode and its v1 cursor still work.

Edit and delete.

httpCopy
PATCH /v1/posts/97301141913751557 HTTP/1.1
Authorization: Bearer <token>
Content-Type: application/json

{ "text": "Just shipped v2.0.1!" }

httpCopy
DELETE /v1/posts/97301141913751557 HTTP/1.1
Authorization: Bearer <token>

Both return 200 with the post's new state ("status": "DELETED" for a delete). A delete sets a tombstone (a marker saying "this was deleted") on the post item instead of removing it at once, so every cache and timeline can recognize it. Both changes flow through the same change stream as new posts, as MODIFY records.

The celebrity registry. An internal list of accounts we don't push for, with an item per account:

The bucket (a hash of the user ID, 0–99) spreads 500K entries over 100 partition keys, so no single key gets hot when the list is rebuilt. Feed nodes never query it per request: a job writes a snapshot of all registry IDs (500K × 8 bytes ≈ 4 MB) to S3 every minute, and every feed node loads it into memory.

R2.4 Design Evolution: Hybrid Fanout, Ranking and Repair

Step 2.1: A Celebrity's Post Needs 60 Million Timeline Writes

Go deeper: the celebrity's follower list is one hot index key. Round 1's GSI puts every follower of an account under GSI1PK = FOLLOWED_BY#<id>. For a 60M-follower account that is one partition key with 60M items. We no longer read it for fanout, but two things still hurt:

• Writes. A DynamoDB partition serves at most 1,000 write units a second. When a celebrity goes viral and gains 10,000 followers a second, all those GSI writes target one key. A GSI that can't keep up throttles writes to the base table: follows of this account, and of any other account whose index key shares that partition, start failing.
• Counters. Keeping follower_count on the profile item means 10,000 updates a second to one item, over the same per-partition limit.

The fix: write-shard the index key for large accounts. The edge's GSI1PK becomes FOLLOWED_BY#<id>#<n>, where n = hash(follower_id) mod S, and the profile stores S (1 for almost everyone, 32 for the largest accounts). Anyone who needs all followers queries the S keys in parallel. Changing S for an existing account means a backfill job that re-keys its edges. Follower counts are summed from the change stream by a small aggregator and written to the profile once a second.

Primitive: Database Sharding & Partition Keys · Drill: Sharding tenant hotspot

Step 2.2: Eight Million People Refresh Right After a Celebrity Posts

Drill: Caching hot product page

Step 2.3: A Burst of Posts Overwhelms the Cache

Primitive: Circuit Breaker, Bulkhead & Fault-Tolerance Patterns · Primitive: Message Queues vs Event Streams

Step 2.4: A User Returns After 30 Days to an Empty Timeline

Step 2.5: A Deleted Post Still Shows for Some Followers

Step 2.6: The Feed Must Be Ranked, Not Chronological

Primitive: Distributed Cache Patterns & Eviction

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The timeline cluster holds only timelines. Everything short-lived (stored ranked lists, per-reader pull lists, rebuild locks) lives in a separate, smaller session cache, for a reason R2.8 explains. The write side now splits by audience size: authors under 1,000 followers and authors from 1,000 to 25,000 go through separate fanout pools, and authors at 25,000 or more skip fanout entirely. The read side runs four steps: gather candidates from the timeline and the celebrity pulls (1a and 1b in parallel), fetch features, score, and hydrate only what will be shown. A missing timeline triggers a background rebuild through SQS.

Trace 1: a regular user posts (1,200 followers)

textCopy
post service ─► PutItem via DAX ─► change stream ─► pool B (1,200 ≥ 1K)
pool B ─► registry says "push" ─► Query GSI1 FOLLOWED_BY#401#0 (2 pages)
       ─► 1,200 × "add if exists, trim to 800" in pipelined batches ─► checkpoint
about 30 ms of cache work; followers who read in the next few seconds see it

Trace 2: a celebrity posts (60M followers)

textCopy
post service ─► PutItem via DAX ─► change stream ─► both pools see "PULL" in the registry ─► skip, checkpoint
no timeline changes at all; the post is visible to anyone whose feed node pulls AUTHOR#celeb
(within ~1 s, the micro-cache TTL)

Trace 3: a ranked feed read

Synthesizing vector architecture diagram...

Page 2 skips everything before hydration: it reads offset 20–44 of the stored list and hydrates.

Trace 4: a user returns after 30 days

textCopy
feed service: tl:802 missing
  ├─ stage 1 (on the request): top-20 interacted accounts × latest 2 posts + celebrity pulls
  │                            → rank → hydrate → page 1 in the normal budget
  └─ stage 2 (SQS job): lock rebuild:802 (60 s) → create tl:802 = { sentinel "_" at score 0 } with 3-day expiry
                        → Query latest posts of all 600 follows → merge → add → trim to 800

R2.6 Numbers and Cost

Traffic (the current track's figures, re-derived)

Storage

The 100.4 TB is logical data. DynamoDB keeps it in three AZs as part of the service and bills the logical size, so we don't multiply by three. And a replication count is not a durability figure: the often-quoted "eleven nines" is S3's design target for objects, and it applies to the media in S3, not to the table.

Fanout writes. Assume 1% of posts come from accounts at or over the threshold (pulled), and pushed posts reach 200 followers on average (a planning assumption; we measure it):

$$1{,}157 \times 0.99 \times 200 \approx 229\text{K writes/s on average} \qquad 3{,}500 \times 0.99 \times 200 \approx 693\text{K at peak}$$

The worst single pushed post is 24,999 followers: at 50,000 writes/s per worker, about 0.5 s.

Timeline memory: the carried-over estimate, and two checks.

The track sizes timelines like this:

$$60\text{M timelines} \times 800 \text{ entries} \times {\sim}64 \text{ B} = 3.07 \text{ TB} \;\times\; 1.35 \;\approx\; 4.15 \text{ TB}$$

That is 51.2 KB per timeline, a 1.35 factor for allocator fragmentation and engine overhead, and 60M timelines described as the "active 20%". We don't copy it; we check both inputs.

Check 1: why 60M timelines, when there are 300M daily users? Timelines exist only for users who read recently (step 1.5). For 60M to be enough, a timeline would have to expire within a few hours of its owner's last read, since a daily user comes back every few hours. Then most sessions would start with no timeline, and at even one rebuild per user per day that is $300\text{M} \div 86{,}400 \approx 3{,}472$ rebuilds a second, each querying ~200 accounts: about 700,000 queries a second on average, more at peak. That is query-on-read for most sessions again, with step 2.4's less complete first page as the normal experience. So we choose the other side of the trade: every user who read in the last 3 days keeps a timeline (the expiry is 3 days, refreshed on read). We assume that is about 1.1 × DAU ≈ 330M timelines. Rebuilds are then only for people back after more than 3 days: assume 3% of DAU a day, 9M a day ≈ 104 a second, × 200 accounts ≈ 21,000 queries a second.

Check 2: is ~64 bytes an entry right? It is an estimate, and for the cache's default layout it is low. Past 128 members (the default zset-max-listpack-entries), a sorted set is stored as a hash table plus a skip list, and a bottom-up estimate per entry is:

Exact figures vary by engine and version, so we measure MEMORY USAGE on a sample of real timelines before buying nodes. At ~120 B, even the track's 60M timelines would need about $60\text{M} \times 800 \times 120 \text{ B} \times 1.35 \approx 7.8$ TB, not 4.15.

The lever: the compact encoding. Below the zset-max-listpack-entries limit, a sorted set is stored as a listpack: one packed block of bytes, where a member or score that looks like an integer is stored as a compact integer. The Redis documentation describes these small encodings as using up to 10 times less memory, and the limit as a CPU-for-memory trade you can tune. Our members (IDs) and scores (milliseconds) are both integers, so an entry is roughly 20–25 bytes. If we set the limit above our trim size (say 1,000, in a custom parameter group), every timeline stays compact. The price: an add or a trim scans and shifts the packed block, which is linear in its size (~20 KB here) instead of logarithmic. At our write rates that's affordable, but it's exactly the kind of change the Redis documentation says to benchmark first, so we load-test it.

$$330\text{M} \times (800 \times 25 + 80) \text{ B} = 330\text{M} \times 20{,}080 \text{ B} \approx 6.63 \text{ TB} \;\times\; 1.25 \approx 8.3 \text{ TB}$$

Cache nodes. A cache.r7g.8xlarge has 209.55 GiB; after the default 25% reserved memory, about 157 GiB ≈ 169 GB, and we fill to 80%, about 135 GB per shard. $8.3 \text{ TB} \div 135 \text{ GB} \approx 61.3$, so 62 shards, each a primary plus one replica in another AZ: 124 nodes. (The default encoding would need about 294 shards.)

Egress

Ranking compute. Assumptions, to be replaced by a load test: ~500 candidates per ranked request; one GPU instance (g5.xlarge, about $1.006/h) scores 20,000 candidates a second with our model; 60% of feed requests are first pages that need ranking (later pages read the stored list); instances run at 60% average utilization.

$$1.5\text{B} \times 30.4 \times 0.6 \approx 27.4\text{B ranked requests/month} \;\times\; 500 \div 20{,}000 \;=\; 684\text{M GPU-seconds} \approx 190\text{K GPU-hours}$$

At 60% utilization that's about 317K instance-hours, ≈ $319K a month, or about $0.012 per 1,000 ranked feeds. At peak: $52{,}000 \times 0.6 \times 500 \div 20{,}000 = 780$ GPUs busy.

DynamoDB capacity, and a quota. Hydration peaks at $52{,}000 \times 25 = 1.3$M item reads a second. With a 90% DAX hit rate (an assumption), 130K reach DynamoDB, 65K read units a second. On-demand tables have a default per-table throughput quota of 40,000 read units and 40,000 write units a second; we raise it well ahead of launch, and alarm on usage against it.

Monthly cost (us-east-1 on-demand list prices; rounded)

Why 78 feed nodes: we plan 1,000 feed requests a second per c7g.2xlarge (an assumption; the node mostly waits on network calls). Peak needs 52; losing an AZ must leave 52, so 26 per AZ × 3 = 78.

Read the table from the top: delivery (CloudFront), ranking and the timeline cache are almost everything. The first is what serving images to 300M people costs (and where large companies negotiate private pricing); the other two are the choices this round made. That's about half a cent per daily user per month.

R2.7 Trade-Offs

R2.8 Failure Modes

Drill: Circuit breaker cascading thread stall

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Makes fanout a per-author decision, and treats the threshold as a tuned choice with a method, not a magic number.
• Knows the limits are per partition key, and protects hot keys with coalescing and write sharding.
• Separates "how fast we read" from "how far we got": backpressure by reading slower, checkpoints for replay.
• Designs the returning-user path, including the race between rebuild and fanout (a sentinel member so the timeline exists before it is filled).
• Puts correctness at hydration (deletes, edits) instead of trying to keep every cache perfect.
• Gives ranking a budget and a fallback, and knows why ranked paging needs a stored session.
• Checks the carried-over sizing instead of trusting it, and finds the real cost drivers.

Follow-up questions

1. "A user follows 3,000 accounts, 400 of which are over the threshold. What happens to their reads?" Answer: 400 pulls per read breaks the 25 ms pull budget. We cap pulls per read (say the 50 pulled accounts they interact with most, rotating the rest across refreshes), and we can precompute a small "pulled posts" list for heavy followers in the background every minute. It's a per-reader decision, the mirror image of step 2.1's per-author one.

2. "Why not shard the timeline cache by region of the world, or by user popularity?" Answer: timelines are read by exactly one user, so we shard by that user's ID: every read is one shard, and every fanout write spreads across shards naturally. Sharding by anything else would make a read cross shards. Popularity matters for authors, and that is handled on the write side (the threshold and the pools), not by where timelines live.

3. "The ranking model team wants 2,000 candidates instead of 500." Answer: that is 4× the GPU cost (about $1.3M a month instead of $319K) and 4× the feature fetch. Before agreeing, we ask for the measured gain, and offer a cheaper shape: a light first-pass model over 2,000, and the heavy model only on the top few hundred. Round 3 builds exactly that.

Interview gotchas from this round

---

Round 3 · Architect · "Global, Private by Default, and Abuse-Resistant"

~45 min · Principal (L7) · 4 regions · ~1B DAU · ~174K feed reads/s peak worldwide · 99.99% per region · privacy changes effective in seconds

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run a ranked feed for 300 million daily users in one region: 100 million posts a day, about 52,000 feed reads a second at peak, 99.99%, P99 under 200 ms. Posts live once in DynamoDB, keyed by post ID. Fanout is hybrid: authors under 25,000 followers are pushed into their followers' timelines by two bulkheaded worker pools reading the table's change stream with backpressure; authors at or over the threshold are pulled at read time, with per-node single-flight and a one-second micro-cache so a celebrity's key sees about one read per node per second. Timelines are compact sorted sets of post IDs, 800 deep, for everyone who read in the last three days: about 330 million timelines, 8.3 TB, 62 cache shards. A read gathers ~500 candidates, fetches features, scores them on GPUs with a 60 ms fallback to chronological, stores the ranked list for the session, and hydrates 25 posts through DAX, dropping deleted ones and repairing the timeline. Returning users get a quick 20-account first page while the full timeline is rebuilt in the background. About $1.4 million a month, mostly delivery, ranking and the cache. Four costs are still open: it's one region; permissions are just 'public'; deletion is per post only; and ranking cost grows with every user."

Architecture v2, compact

textCopy
 post ─► DynamoDB (via DAX) ─► change stream ─► pool A (< 1K followers) / pool B (1K–25K) ─► timelines (62 shards)
                                                  authors ≥ 25K: no fanout (registry snapshot)
 read ─► timeline top 400 ∥ celebrity pulls (single-flight, 1 s micro-cache)
      ─► features ─► GPU ranker (60 ms, else chronological) ─► rk:<user>:<session> ─► hydrate 25 via DAX, drop DELETED
 missing timeline ─► stage 1 (top-20 accounts) now; stage 2 rebuild via SQS (create with a sentinel, then fill; 60 s lock)

Steps so far

Open costs: one region; everything is public; account deletion isn't designed; ranking cost per user.

R3.1 The Scope Raise

Interviewer: "We're global now: about a billion daily users on every continent, and people follow each other across continents. Posts can be friends-only or shared with a custom list. When someone blocks a person, that person must disappear from their feed immediately, and the other way round. When a user deletes their account, it must vanish from every feed and be purged everywhere. Spam rings are creating accounts and posting to millions. And we must survive losing a whole region."

Again, we ask back before we design:

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

An audience on every post, set at creation and changeable later:

httpCopy
POST /v1/posts HTTP/1.1
Authorization: Bearer <token>
Content-Type: application/json

{ "text": "Family dinner!", "media_ids": ["m_91xQ"], "audience": { "type": "LIST", "list_id": "l_fam" } }

audience.type is PUBLIC, FRIENDS or LIST. PATCH /v1/posts/{id} with a new audience changes who may see it from the next read on.

Blocks and mutes:

httpCopy
POST /v1/users/555/block HTTP/1.1
Authorization: Bearer <token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "user_id": "555", "blocked": true, "effective": "next feed load" }

A block works both ways: neither person sees the other's posts. A mute (POST /v1/users/555/mute) only hides 555 from the muter's feed, and 555 is not told.

Account deletion:

httpCopy
DELETE /v1/me HTTP/1.1
Authorization: Bearer <token>

httpCopy
HTTP/1.1 202 Accepted
Content-Type: application/json

{ "deletion_id": "del_3fK9", "status": "DELETING", "purge_deadline": "2031-03-31T00:00:00Z" }

GET /v1/deletions/del_3fK9 (internal, for support and audit) returns the workflow's state: DELETING (hidden from every feed, purge running) → PURGED, with counts of what was deleted in each store.

An integrity hook before fanout. The fanout workers call an internal decision service for every new post:

jsonCopy
{ "post_id": "683247088435237888", "author_id": "5550001", "decision": "HOLD", "reason": "new_account_link_burst", "review_by": "2031-03-01T09:44:00Z" }

decision is ALLOW, DELAY (fan out after a short wait), HOLD (visible only to the author until reviewed) or DENY.

R3.4 Design Evolution: Geography, Privacy, Deletion and Integrity

Step 3.1: Followers Live in Other Regions

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active · Drill: Replication multi-region consistency

Step 3.2: A Block Is Made, but the Posts Are Already in the Timeline

Drill: JWT revocation token blacklist

Step 3.3: A Deleted Account Must Vanish Everywhere

Primitive: Change Data Capture & the Outbox Pattern

Step 3.4: A Spam Wave Is Being Fanned Out to Millions

Primitive: Bot Defense, Sybil Resistance & Registration Abuse · Drill: Bot defense credential-stuffing surge · Loop: Design a Distributed Rate Limiter

Step 3.5: A Region Is Gone

Synthesizing vector architecture diagram...

The partner serves the lost region's users from data it already holds. Only the timelines are new, and only for users who show up.

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active

Step 3.6: Ranking Now Costs More Than Serving

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Every region is a Round 2 system that serves only its own users. The global layer is data, not traffic: tables that replicate, one CDN in front of every region's media, and DNS that knows each region's partner. Nothing on a feed read crosses a region.

Trace 1: a cross-region follow and post (Mumbai reader, Dublin author)

textCopy
Mumbai (ap-south-1):  POST /v1/users/<dublin_author>/follow
  └─ PutItem edge (written in the follower's home region): GSI1PK = FOLLOWED_BY#<author>#ap-south-1#0
     replicates to the other three regions (~1 s)
Dublin (eu-west-1):   author posts ─► PutItem POST#<id> (eu-west-1 writes it)
  └─ replicates to ap-south-1 (~1 s)
Mumbai:  replica's DynamoDB Streams ─► forwarder ─► Kinesis ─► integrity gate: ALLOW
  └─ pool A queries FOLLOWED_BY#<author>#ap-south-1#* ─► add to the Mumbai reader's timeline
Mumbai reader: GET /v1/feed ─► served entirely from ap-south-1

Trace 2: a block

textCopy
Priya (eu-west-1) blocks Sam (us-east-1):
  1. eu-west-1 writes BLOCKS edge (Priya → Sam) and deletes Priya's cached block set
  2. Priya's next read (same region): Sam's posts dropped at hydration. Effective immediately.
  3. edge replicates to us-east-1 (~1 s); a stream consumer there deletes Sam's cached block set
  4. Sam's next read: Priya's posts dropped. Effective within seconds.

Trace 3: a region failover (see the diagram in step 3.5)

textCopy
t = 0      us-east-1 health checks fail
t ≈ 2 min  Route 53 answers us-east.api with eu-west-1's endpoint (health-check interval × failure threshold, + record TTL)
t > 2 min  us-east-1 users hit eu-west-1: timeline miss → stage 1 page → 200-entry rebuild (6 h expiry)
           eu-west-1 fanout also processes FOLLOWED_BY#…#us-east-1 keys
           failed-over users: light ranker or chronological
return     replication resumes; late writes arrive; Route 53 fails back; the temporary timelines expire

R3.6 Numbers and Cost

Traffic per region (a planning split; each region is planned for its own peak at 3× its average)

Posts: $1\text{B} \times \tfrac{100\text{M}}{300\text{M}} \approx 333$M a day $= 3{,}858$/s average, $\approx 11{,}600$/s at peak. us-east-1 alone is exactly Round 2's system.

Cross-region replication

Prices above are us-east-1 list prices; other regions cost somewhat more.

Timeline cache per region (from R2.6: 330M timelines ≈ 8.3 TB, so ≈ 2.51 TB per 100M timelines; 135 GB usable per cache.r7g.8xlarge shard)

Failover headroom: 120M temporary timelines (step 3.5) × (200 entries × 25 B + 80 B) ≈ 610 GB × 1.25 ≈ 760 GB ≈ 6 shards, reserved in every region so any partner can take the other's users.

$462 \times \$3.49/\text{h} \times 730 \text{ h} \approx \$1.18\text{M/month}$

Ranking cost per 1,000 ranked feeds (planning assumptions to confirm by load test: a c7g.4xlarge, about $0.58/h, scores 500,000 candidates a second with the first-pass model; a g5.xlarge, about $1.006/h, scores 20,000 a second with the heavy model; 60% average utilization)

Ranked requests: $5\text{B} \times 30.4 \times 0.6 \approx 91.2$B a month. Round 2's design at this scale: 91.2\text{M} \times \0.0116 \approx $1.06$M a month. Tiered:$91.2\text{M} \times $0.0014 \approx $131$K a month.

Monthly total (list prices, us-east-1 rates where regional prices differ; rounded; storage a year in)

About half a cent per daily user per month. Delivering images is about half the bill, and at this volume it is negotiated, not paid at list price. The two lines this loop designed are the cache (a choice about who keeps a timeline and how it's encoded) and ranking (tiered, eight times cheaper than scaling Round 2's).

R3.7 Trade-Offs

Closing the loop. The opening question was: do we do the work when someone posts, or when someone reads, and who decides for each post? The answer is now:

• Speed work happens at write time for almost everyone: precomputed timelines, in the reader's region.
• It moves to read time where precomputing costs too much: celebrities (by a tuned threshold), returning users (a quick first page), and failed-over users (small temporary timelines).
• Correctness work always happens at read time: deletes, blocks, audiences, account tombstones and the kill switch are checked against current data on every page. Precomputed must never mean allowed.
• Who decides: the author's follower count decides push or pull, the reader's recency decides whether they keep a timeline, and the integrity gate decides whether a post fans out at all.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per region OPS 8 · REL 6

Kill-switch procedure (a post or an account must disappear from every feed now) SEC 10 · OPS 10

1. Add the post IDs or account ID to the deny list (CLI 5 below). Every feed node and fanout worker in every region reloads it within 5 seconds; hydration drops the post, fanout skips it.
2. Confirm. The deny-list size metric rises in all four regions; a test account that follows the author no longer sees the post.
3. Make it permanent. Set the post's status to DELETED (or start account deletion, step 3.3). Invalidate the media paths in CloudFront.
4. For a ring of accounts, list their posts (CLI 6) and add them in bulk; set the gate to HOLD for accounts that share the ring's signals.
5. Record it: who, what, why, when, and a link to the evidence, in the incident ticket. Deny-list entries carry the operator and the reason.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the IDs and ARNs with real ones.

textCopy
# 1. Fanout lag for pool B (enhanced fan-out consumers report MillisBehindLatest)
aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/Kinesis --metric-name SubscribeToShardEvent.MillisBehindLatest --dimensions Name=StreamName,Value=post-events Name=ConsumerName,Value=fanout-pool-b --statistics Maximum --period 60 --start-time 2031-03-01T09:00:00Z --end-time 2031-03-01T10:00:00Z

# 2. Cross-region replication lag into eu-west-1
aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/DynamoDB --metric-name ReplicationLatency --dimensions Name=TableName,Value=SocialCoreTable Name=ReceivingRegion,Value=eu-west-1 --statistics Average --period 60 --start-time 2031-03-01T09:00:00Z --end-time 2031-03-01T10:00:00Z

# 3. Scale fanout pool B
aws ecs update-service --region us-east-1 --cluster feed --service fanout-pool-b --desired-count 40

# 4. Timeline cache status (shards, failover state)
aws elasticache describe-replication-groups --region us-east-1 --replication-group-id timelines-use1

# 5. Kill switch: add a post to the deny list (a global table; any region)
aws dynamodb put-item --region us-east-1 --table-name IntegrityDenyList --item '{"PK": {"S": "POST#683247088435237888"}, "reason": {"S": "scam-link-wave"}, "added_by": {"S": "oncall-alice"}, "added_at": {"S": "2031-03-01T09:42:00Z"}}'

# 6. List an account's posts for a bulk kill
aws dynamodb query --region us-east-1 --table-name SocialCoreTable --index-name GSI1 --key-condition-expression "GSI1PK = :a" --expression-attribute-values '{":a": {"S": "AUTHOR#5550001"}}' --projection-expression "PK"

# 7. Account deletions still running
aws stepfunctions list-executions --region us-east-1 --state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:account-deletion --status-filter RUNNING

# 8. Remove a killed account's media from the CDN
aws cloudfront create-invalidation --distribution-id E2EXAMPLE12345 --paths "/u/5550001/*"

# 9. Is a region's API health check failing?
aws route53 get-health-check-status --health-check-id 11111111-2222-3333-4444-555555555555

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Separates speed from permission: fanout is an optimization, every read checks current data.
• Places data by who reads it (reader-home timelines) and who writes it (one writing region per item, handed to the partner during a failover), and knows what global tables do with conflicts.
• Designs deletion as a provable workflow with a fast hiding step, and refuses TTL as a deadline.
• Puts abuse controls before distribution, and builds the kill switch on the read-time filter that already exists.
• Treats timelines as a cache even in disaster recovery, and sizes the rebuild storm instead of doubling the cache.
• Finds where the money goes (delivery, cache, ranking) and changes the design where it's ours to change.

Follow-up questions

1. "A user moves from Europe to India. What happens to their home region?" Answer: their profile's home_region changes, written by the old home region (the single writer for that item) and then owned by the new one. Their follow edges keyed by region need re-keying: a background job rewrites their FOLLOWED_BY#…#<region> entries under the new region. Their timeline isn't moved; the new region rebuilds it (step 2.4). Until the re-keying finishes, both regions' fanouts may reach them, and the timeline's duplicate adds are harmless.

2. "Legal says some countries require their users' posts to stay in-country. Does this design allow it?" Answer: not as is: global tables replicate every item to every replica region. We'd split posts into a second table that is only replicated among allowed regions, and readers elsewhere would either not see those posts or fetch them across a region. That breaks "no cross-region call on a read" for those posts, which is a product and legal decision to make explicitly, not an engineering default.

3. "Why not rank with the heavy model and just buy more GPUs? Engagement matters more than cost." Answer: then measure it. Run both on a slice of traffic and compare engagement against the ~$0.9M a month difference. If the heavy-only model wins by enough, buy the GPUs. The tiered design isn't a rule; it's the default until the numbers say otherwise.

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: is the feed chronological or ranked, how many accounts does a user follow, and how fresh does it need to be?"
• When the scope is raised: "Here's what breaks in the current design, and I'll fix it in this order: anything that can show someone a post they shouldn't see, then anything that can take reads down, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels