Design a Scalable News Feed System

1. Problem Statement & Scope

System Mission

Design a massive-scale social news feed system (similar to Facebook News Feed, Twitter/X Timeline, or LinkedIn Feed) capable of publishing posts, aggregating updates in real-time, and rendering ranked feeds with sub-second latency for hundreds of millions of users.

Functional Requirements

1. Feed Publishing: Users can publish text posts, images, and videos.
2. Feed Generation (Fan-out): A user's timeline displays aggregated posts from followed users/friends, sorted chronologically and ranked by relevance.
3. Pagination: Seamless infinite scrolling pagination.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.99\%$ uptime SLA.
• Latency: Feed read latency $< 200\text{ ms}$ P99; post publishing $< 500\text{ ms}$.
• Consistency: Eventual consistency for timeline aggregation; immediate consistency for post creation.

2. Capacity & Scale Estimation

• Daily Active Users (DAU): 300 Million ($300\times 10^6$).
• Daily Posts: 100 Million posts/day.
• Feed Reads: 1.5 Billion feed requests/day.
• Throughput (QPS):
  • Read QPS: $17,360\text{ RPS}$ (Peak $3\times$: $\mathbf{52,000\text{ RPS}}$).
  • Write QPS: $1,157\text{ WPS}$ (Peak $3\times$: $\mathbf{3,500\text{ WPS}}$).
• 5-Year Metadata Storage: $\approx \mathbf{91.25\text{ TB}}$ (Post media stored in Amazon S3).

3. AWS-First High-Level Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

4. API Interface Design

httpCopy
POST /v1/posts
Host: api.social.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "content": "Designing high-scale distributed systems on AWS!",
  "media_urls": ["https://cdn.social.com/media/img_9981.webp"],
  "privacy": "PUBLIC"
}

Response: 201 Created
{
  "post_id": "718293847561029384",
  "author_id": "usr_1029",
  "created_at": 1718000000
}

5. Data Models & Storage Architecture

DynamoDB Single-Table Design (SocialCoreTable)

The core social graph and post metadata reside in a single-table DynamoDB cluster optimized for $O(1)$ point lookups:

sqlCopy
-- Partition Key (PK) | Sort Key (SK)           | Attributes / Payload
----------------------------------------------------------------------------------------
-- User Profile
USER#usr_102         | METADATA                | name, avatar_url, follower_count, is_verified
-- User Published Post
USER#usr_102         | POST#1718000000#p_9912  | content, media_urls[], created_at, likes_count
-- Follower Edge (Graph)
USER#usr_102         | FOLLOWS#usr_404         | followed_at, notification_level
-- Followee Inverted Index (GSI1: Inverted Follower Graph)
-- GSI1_PK = FOLLOWS#usr_404 | GSI1_SK = USER#usr_102 (Enables fast retrieval of all followers)

ElastiCache Redis Timeline Cache Layout

• Key: timeline:user:<user_id> (Redis Sorted Set - ZSET).
• Member: post_id (64-bit integer string).
• Score: created_at UNIX millisecond timestamp.
• Capacity Policy: Capped to latest $800\text{ post IDs}$ per user via ZREMRANGEBYRANK.