Design a Scalable News Feed System

1. Problem Statement & Scope

System Mission

Design a massive-scale, low-latency social news feed platform (similar to Facebook News Feed, Twitter/X Timeline, or LinkedIn Feed) capable of publishing multimedia updates, dynamically aggregating friend and followee activities, ranking items by personalized relevance, and delivering continuous timeline scrolling for hundreds of millions of daily active users with sub-200ms latency.

Functional Requirements

1. Feed Publishing (POST /v1/posts): Users can publish status updates containing text, rich metadata, and media attachments (images/videos uploaded via pre-signed S3 URLs).
2. Feed Generation & Retrieval (GET /v1/feed): Users view a real-time, aggregated timeline composed of posts from accounts they follow, sorted chronologically or ranked by a personalized machine learning recommendation score.
3. Hybrid Fanout Engine: Dynamically routes post distribution using fanout-on-write (push) for regular users and fanout-on-read (pull) for high-follower celebrity accounts.
4. Follow / Unfollow Graph (POST /v1/users/{id}/follow): Real-time management of bidirectional friendship edges and unidirectional follow graphs.
5. Infinite Cursor-Based Pagination: Seamless timeline scrolling using opaque cryptographic cursors preventing duplicate items or missed posts during concurrent insertions.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.99\%$ uptime SLA globally across multi-AZ AWS regions.
• Latency:
  • Read Path (GET /v1/feed): P95 $< 120\text{ ms}$, P99 $< 200\text{ ms}$.
  • Write Path (POST /v1/posts): P95 $< 100\text{ ms}$, P99 $< 250\text{ ms}$.
  • Fanout Propagation Latency: $99\%$ of followers receive standard user posts in $< 5\text{ seconds}$.
• Consistency: Eventual consistency for timeline aggregation ($< 5\text{ s}$ window); strict monotonic consistency and linearizability for post creation and author lookups.
• Durability: $99.999999999\%$ (11 Nines) zero-loss durability for post text, relational graph edges, and media assets.

---

2. Capacity & Scale Estimation

Traffic Calculations

• Daily Active Users (DAU): $300,000,000$ ($300\text{M DAU}$).
• Daily Post Ingestion: $100,000,000$ new posts/day.
• Average Post Write QPS: $\text{Write QPS}_{\text{avg}} = \frac{100,000,000\text{ posts}}{86,400\text{ sec}} \approx \mathbf{1,157\text{ posts/sec}}$
• Peak Post Write QPS ($3\times$ peak factor): $\text{Write QPS}_{\text{peak}} = 1,157 \times 3 \approx \mathbf{3,471\text{ posts/sec}} \approx \mathbf{3,500\text{ QPS}}$
• Feed Read Traffic: Each active user opens the app and refreshes their feed an average of $5$ times/day: $\text{Daily Feed Requests} = 300,000,000 \times 5 = \mathbf{1.5\text{ Billion requests/day}}$
• Average Feed Read QPS: $\text{Read QPS}_{\text{avg}} = \frac{1.5 \times 10^9\text{ requests}}{86,400\text{ sec}} \approx \mathbf{17,361\text{ reads/sec}}$
• Peak Feed Read QPS ($3\times$ peak factor): $\text{Read QPS}_{\text{peak}} = 17,361 \times 3 \approx \mathbf{52,083\text{ reads/sec}} \approx \mathbf{52,000\text{ QPS}}$
• Read-to-Write Ratio: $\approx 15:1$ (Read heavy).

Storage Calculations (5-Year Horizon)

• Post Metadata Schema Size:
  • post_id: $8\text{ Bytes}$ (64-bit Snowflake integer).
  • author_id: $8\text{ Bytes}$.
  • content: $280\text{ UTF-8 characters} \approx 300\text{ Bytes}$.
  • media_urls: $150\text{ Bytes}$ (JSON array of CDN pointers).
  • created_at: $8\text{ Bytes}$ (Unix epoch millisecond).
  • metrics (likes, shares, comments): $26\text{ Bytes}$.
  • DynamoDB attribute overhead: $\approx 50\text{ Bytes}$.
  • Total Metadata Size per Post: $\approx \mathbf{550\text{ Bytes}}$.
• 5-Year Post Metadata Storage: $\text{Volume}_{5\text{yr}} = 100\times 10^6\text{ posts/day} \times 365.25 \times 5 = \mathbf{182.6\text{ Billion posts}}$ $\text{Storage}_{\text{metadata}} = 182.6 \times 10^9 \times 550\text{ Bytes} \approx \mathbf{100.4\text{ TB}}$ With DynamoDB Multi-AZ replication ($\times 3$): $\mathbf{301.2\text{ TB}}$.
• Media Storage (Amazon S3):
  • Assume $20\%$ of posts contain an image ($200\text{ KB}$ WebP) and $5\%$ contain a video ($2.5\text{ MB}$ MP4): $\text{Avg Media / Post} = (0.20 \times 200\text{ KB}) + (0.05 \times 2,500\text{ KB}) = 40\text{ KB} + 125\text{ KB} = \mathbf{165\text{ KB}}$
  • Daily S3 Ingress: $100\times 10^6 \times 165\text{ KB} \approx \mathbf{16.5\text{ TB/day}}$.
  • 5-Year Media Storage: $16.5\text{ TB/day} \times 365.25 \times 5 \approx \mathbf{30.1\text{ PB}}$ on Amazon S3 Standard with lifecycle transitioning to Glacier Flexible Retrieval after 90 days.

Network Bandwidth

• Ingress Bandwidth (Writes): $\text{Bandwidth}_{\text{in}} = 3,500\text{ writes/sec} \times (550\text{ B} + 165\text{ KB}) \approx 580\text{ MB/sec} \approx \mathbf{4.64\text{ Gbps}}$ (Almost all of this is media landing directly in S3 via pre-signed URLs; the Post Service API itself sees only $3,500 \times 550\text{ B} \approx 2\text{ MB/sec}$.)
• Egress Bandwidth (Feed Reads):
  • Each feed request returns 20 hydrated post objects ($20 \times 550\text{ Bytes} \approx 11\text{ KB}$ payload): $\text{Bandwidth}_{\text{out}} = 52,000\text{ reads/sec} \times 11\text{ KB} \approx 572\text{ MB/sec} \approx \mathbf{4.58\text{ Gbps}}$
  • Media asset egress (images/videos) is served directly via Amazon CloudFront edge PoPs with an $88\%$ edge cache hit ratio, absorbing $\approx 40\text{ Gbps}$ of raw media traffic.

Memory & Cache Sizing (ElastiCache Redis Timelines)

• Applying the 80/20 Pareto Principle: $20\%$ of registered users represent the active daily core ($60\text{M}$ active user timelines cached concurrently).
• Timeline Key Structure: Each active user maintains an ElastiCache Redis Sorted Set (ZSET) containing the latest $800\text{ post IDs}$.
• Memory per ZSET entry: $8\text{ B}$ (member string post_id) + $8\text{ B}$ (score millisecond timestamp) + $48\text{ B}$ (SkipList node + Hash table entry) $\approx 64\text{ Bytes}$. $\text{RAM / User Timeline} = 800\text{ posts} \times 64\text{ Bytes} = \mathbf{51.2\text{ KB}}$ $\text{Total Timeline RAM} = 60,000,000\text{ users} \times 51.2\text{ KB} \approx \mathbf{3.07\text{ TB}}$ Accounting for Jemalloc fragmentation and Redis overhead ($\times 1.35$): $\text{Provisioned Cache Capacity} = 3.07\text{ TB} \times 1.35 \approx \mathbf{4.15\text{ TB RAM}}$
• Provisioned on an Amazon ElastiCache Redis Cluster with $40\times \text{cache.r6g.4xlarge}$ primary nodes ($105.8\text{ GiB RAM}$ per node $\to 4.2\text{ TB}$ usable), clustered mode enabled across 3 AZs, plus one replica per shard for failover (80 nodes total). A common sizing mistake is to quote the node count without multiplying it out: $24 \times 52.6\text{ GiB}$ would be only $1.26\text{ TB}$, a third of the working set.

---

3. AWS-First High-Level Architecture

Synthesizing vector architecture diagram...

Follow a post from creation to a reader's feed. The publishing service saves the post in DynamoDB (media in S3) and emits an event to Kinesis. In the "Asynchronous Fanout & Event Ingestion" panel, fanout workers read the author's followers and push the post ID into each active follower's timeline sorted set in Redis; this is fan-out on write, so reading later is cheap. When someone opens the app, the "Feed Generation & Ranking Engine" panel runs four steps: read their precomputed timeline (1), fetch recent posts from celebrities they follow (2), score the candidates with a SageMaker model (3), and load the top 20 posts through DAX (4). Celebrities are excluded from fan-out because pushing one post to millions of timelines would be a huge write spike; the mix of push for normal users and pull for celebrities is the key idea.

Data Flow Walkthrough

1. Post Creation Path:
   • The client requests an S3 pre-signed upload URL from the Post Service, uploads images/videos directly to Amazon S3, and dispatches POST /v1/posts.
   • Post Service validates the payload, generates a collision-free 64-bit Snowflake post_id, writes the primary post record into Amazon DynamoDB, and publishes a lightweight post event to Amazon Kinesis Data Streams.
2. Dynamic Fanout Execution:
   • The Fanout Worker Fleet consumes from Kinesis. If the author has $< 25,000$ followers (standard user), the worker retrieves the follower list from DynamoDB and pipelines ZADD commands to write the post_id into each active follower's Redis timeline.
   • If the author has $\ge 25,000$ followers (celebrity), push fanout is skipped entirely. The post is simply marked in the author's own post partition.
3. Feed Assembly & Read Path:
   • Client invokes GET /v1/feed. The News Feed Service fetches the calling user's precomputed timeline from ElastiCache Redis (Top 800 items).
   • Simultaneously, Feed Service retrieves the list of followed celebrities, queries their recent posts from DynamoDB, and executes a $K$-way timestamp merge in memory.
   • Merged candidates are passed to an Amazon SageMaker endpoint for ML scoring, the top 20 items are hydrated via DynamoDB Accelerator (DAX), and the response JSON is returned.

Concrete Step-by-Step Request Walkthrough: Tracing Read & Write Cycles

---

4. API Interface Design

1. Publish Post Endpoint

httpCopy
POST /v1/posts
Host: api.social.aws.internal
Content-Type: application/json
Authorization: Bearer <jwt_access_token>
X-Idempotency-Key: 7b8e1f20-945a-497b-95eb-36b17c5b6102

{
  "content": "Designing high-scale distributed systems on AWS! #SystemDesign",
  "media_attachments": [
    {
      "media_type": "IMAGE",
      "s3_url": "https://media.social.internal/uploads/2026/09/post_img_9912.webp",
      "width": 1920,
      "height": 1080
    }
  ],
  "visibility": "PUBLIC"
}

Response: 201 Created

jsonCopy
{
  "status": "SUCCESS",
  "data": {
    "post_id": "718293847561029384",
    "author_id": "usr_401",
    "created_at": 1718000000120,
    "content": "Designing high-scale distributed systems on AWS! #SystemDesign",
    "media_attachments": [
      {
        "media_type": "IMAGE",
        "url": "https://cdn.social.com/media/uploads/2026/09/post_img_9912.webp",
        "width": 1920,
        "height": 1080
      }
    ]
  }
}

2. Retrieve Feed Endpoint (Cursor Pagination)

httpCopy
GET /v1/feed?limit=20&cursor=eyJwb3N0X2lkIjoiNzE4MjkiLCJzY29yZSI6OTguMn0=
Host: api.social.aws.internal
Authorization: Bearer <jwt_access_token>
Accept: application/json

Response: 200 OK

jsonCopy
{
  "status": "SUCCESS",
  "data": {
    "items": [
      {
        "post_id": "718293847561029384",
        "author": {
          "user_id": "usr_401",
          "username": "alex_architect",
          "avatar_url": "https://cdn.social.com/avatars/usr_401.webp",
          "is_verified": true
        },
        "content": "Designing high-scale distributed systems on AWS! #SystemDesign",
        "media_attachments": [
          {
            "media_type": "IMAGE",
            "url": "https://cdn.social.com/media/uploads/2026/09/post_img_9912.webp"
          }
        ],
        "created_at": 1718000000120,
        "metrics": {
          "likes": 1420,
          "reposts": 88,
          "comments": 64
        }
      }
    ],
    "pagination": {
      "has_more": true,
      "next_cursor": "eyJwb3N0X2lkIjoiNzE4MjAxIiwic2NvcmUiOjg1LjR9"
    }
  }
}

3. Follow / Unfollow User Endpoint

httpCopy
POST /v1/users/usr_celeb99/follow
Host: api.social.aws.internal
Authorization: Bearer <jwt_access_token>

Response: 200 OK
{
  "status": "SUCCESS",
  "data": {
    "target_user_id": "usr_celeb99",
    "is_following": true,
    "target_is_celebrity": true,
    "followed_at": 1718000005000
  }
}

---

5. Data Models & Storage Architecture

DynamoDB Single-Table Design (SocialCoreTable)

The social graph, post metadata, and follower relationships are unified in a single high-performance DynamoDB table:

• Primary Partition Key (PK): Entity identifier.
• Primary Sort Key (SK): Hierarchical sub-type and timestamp.
• Global Secondary Index 1 (GSI1): Followee index for reverse follower fanout queries.
• Global Secondary Index 2 (GSI2): Feed candidate retrieval by category or hashtag.

ElastiCache Redis Timeline Cache Structure

• Data Structure: Redis Sorted Set (ZSET).
• Key: timeline:usr_<user_id>
• Member Value: post_id (64-bit integer formatted as string).
• Score: Millisecond timestamp (created_at) for chronological feeds, or scaled float score ($[0.0, 100.0]$) for ranked feeds.
• Eviction & Trimming Policy: User timelines are strictly bounded to the most recent $800$ entries. Atomic trimming is enforced on every insertion:

  textCopy
  ZADD timeline:usr_802 1718000000120 "718293847561029384"
  ZREMRANGEBYRANK timeline:usr_802 0 -801
  EXPIRE timeline:usr_802 604800  # 7-day inactivity TTL

Cursor Pagination Over the Sorted Set

The cursor in GET /v1/feed is an HMAC-signed Base64 tuple (last_score, last_post_id), never an offset:

• Page 1: ZREVRANGEBYSCORE timeline:usr_802 +inf -inf LIMIT 0 20.
• Page N+1: ZREVRANGEBYSCORE timeline:usr_802 (last_score -inf LIMIT 0 20 (the ( makes the bound exclusive), then drop any item with post_id >= last_post_id at the boundary score to break ties.
• Why not OFFSET: while the reader scrolls, fanout keeps inserting newer posts at the top of the set. An offset-based page 2 would shift and re-show items from page 1; a score-based cursor is anchored to a specific post, so concurrent inserts can never duplicate or skip an item.
• Signing: the HMAC stops clients from forging cursors into other users' score ranges, and a bad signature simply resets the client to page 1.

---