Design YouTube Video Streaming Platform

1. Problem Statement & Scope

System Mission

Design a global-scale video sharing, processing, and streaming platform capable of handling hundreds of thousands of daily video uploads, distributed parallel transcoding across heterogeneous resolutions and codecs (H.264/AVC, H.265/HEVC, AV1, VP9), Adaptive Bitrate (ABR) streaming via HLS and MPEG-DASH, and ultra-low latency global content delivery with high cache hit ratios.

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Functional Requirements

1. Resilient Video Upload: Support resumable, chunked multipart uploads for large video files ($> 50\text{ GB}$) over unstable networks.
2. Automated Transcoding Pipeline (DAG(Directed Acyclic Graph)): Ingest raw source files, split into GOP (Group of Pictures) chunks, transcode into multi-bitrate profiles ($1080\text{p}, 720\text{p}, 480\text{p}, 360\text{p}, 240\text{p}, 4\text{K}$), and package into HLS (.m3u8, .ts) and MPEG-DASH (.mpd, .m4s) formats.
3. Adaptive Bitrate Streaming (ABR): Deliver media segments dynamically adapting to client bandwidth, device screen resolution, and buffer occupancy without playback interruption.
4. Metadata, Search & Discovery: Real-time video metadata indexing, full-text search, view count tracking, likes, and comment threads.
5. Video Deduplication & Watermarking: Detection of duplicate uploads via perceptual and cryptographic hashing; dynamic forensic watermarking.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.99\%$ availability for video playback ($< 52.6\text{ minutes}$ downtime/year); $99.95\%$ for video ingestion.
• Playback Latency & TTFF (Time-To-First-Frame): $< 800\text{ ms}$ globally across CDN edge Points of Presence (PoPs).
• Zero Stall Rate (Rebuffering): $< 0.5\%$ of playback sessions experiencing mid-stream buffering events.
• Transcoding Turnaround SLA: Video ready for playback within $1.5\times$ the video duration for standard priority; $< 0.5\times$ for high-priority/breaking creators.
• Storage Durability: 11 9s ($99.999999999\%$) data durability for master and transcoded video segments.

Out-of-Scope

• Real-time ultra-low latency interactive live streaming (WebRTC / LL-HLS sub-second live streaming covered in dedicated blueprints).
• Rights Management & Content ID fingerprinting algorithms (e.g., automated copyright audio spectrogram analysis).
• In-stream targeted programmatic ad bidding auctions (RTB).

---

2. Capacity & Scale Estimation

Traffic & Throughput Calculations

• Daily Active Users (DAU): $100\text{ Million}$ active viewers.
• Video Views / Playback Sessions: Average $10\text{ videos/user/day} \implies 1\text{ Billion views/day}$. $\text{Average Playback QPS} = \frac{10^9 \text{ views}}{86,400 \text{ s}} \approx 11,574 \text{ views/sec}$ $\text{Peak Playback QPS (3}\times\text{)} = 35,000 \text{ views/sec}$
• Video Uploads: $500,000\text{ video uploads/day}$. $\text{Average Upload QPS} = \frac{500,000}{86,400} \approx 5.8 \text{ uploads/sec (Peak: } 25 \text{ uploads/sec)}$

Storage & Bandwidth Footprint

• Average Raw Video Size: Average duration = $10\text{ minutes}$ ($600\text{ seconds}$). High-quality source bitrate $\approx 40\text{ Mbps} \implies 3\text{ GB/raw upload}$. $\text{Daily Raw Ingestion} = 500,000 \times 3\text{ GB} = 1.5\text{ PB/day}$
• Transcoding Expansion:
  • Each video is transcoded into 6 resolution renditions: $1080\text{p}$ ($5\text{ Mbps}$), $720\text{p}$ ($2.5\text{ Mbps}$), $480\text{p}$ ($1.2\text{ Mbps}$), $360\text{p}$ ($700\text{ kbps}$), $240\text{p}$ ($400\text{ kbps}$), $144\text{p}$ ($250\text{ kbps}$).
  • Cumulative Transcoded Bitrate $\approx 10.05\text{ Mbps} \approx 1.25\text{ MB/sec}$.
  • Storage per 10-min transcoded video $= 600\text{ s} \times 1.25\text{ MB/s} = 750\text{ MB}$. $\text{Daily Transcoded Storage} = 500,000 \times 750\text{ MB} = 375\text{ TB/day} \implies \mathbf{136.875\text{ PB/year}}$
• CDN Egress Bandwidth:
  • Average stream watch time $= 5\text{ minutes}$ at $720\text{p}$ average ($2.5\text{ Mbps} \approx 312.5\text{ KB/s}$).
  • Data delivered per view $= 300\text{ s} \times 312.5\text{ KB/s} = 93.75\text{ MB}$. $\text{Total Daily Egress Data} = 10^9 \times 93.75\text{ MB} = 93.75\text{ PB/day}$ $\text{Average Egress Bandwidth} = \frac{93.75 \times 10^{15} \times 8 \text{ bits}}{86,400 \text{ s}} \approx \mathbf{8.68\text{ Tbps (Peak: } 25\text{ Tbps)}}$
• Edge Cache Hit Ratio (CHR): CloudFront Edge targets $\ge 98\%$ CHR for media segments (.ts/.m4s). $\text{Origin Shield / S3 Origin Egress} = 8.68\text{ Tbps} \times (1 - 0.98) = \mathbf{173.6\text{ Gbps}}$

---

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Component Responsibility Breakdown

---

4. API Interface Design & Wire Protocols

1. Initialize Multipart Video Upload

httpCopy
POST /v1/videos/uploads/initialize HTTP/1.1
Host: api.youtube.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "title": "Deep Dive into Distributed Consensus",
  "description": "Exhaustive exploration of Raft and Paxos algorithms.",
  "category_id": 28,
  "privacy": "PUBLIC",
  "file_name": "consensus_4k_master.mov",
  "file_size_bytes": 4294967296,
  "chunk_size_bytes": 10485760,
  "total_parts": 410,
  "content_type": "video/quicktime"
}

Response: 201 Created

jsonCopy
{
  "video_id": "vid_998124a87c1",
  "upload_id": "s3_upload_abc123xyz789",
  "part_urls": [
    {
      "part_number": 1,
      "presigned_url": "https://s3.us-east-1.amazonaws.com/raw-ingest/vid_998124a87c1?partNumber=1&uploadId=s3_upload_abc123xyz789&X-Amz-Signature=..."
    },
    {
      "part_number": 2,
      "presigned_url": "https://s3.us-east-1.amazonaws.com/raw-ingest/vid_998124a87c1?partNumber=2&uploadId=s3_upload_abc123xyz789&X-Amz-Signature=..."
    }
  ],
  "expires_in_seconds": 86400
}

2. Complete Multipart Upload & Trigger Transcoding DAG(Directed Acyclic Graph)

httpCopy
POST /v1/videos/uploads/complete HTTP/1.1
Host: api.youtube.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "video_id": "vid_998124a87c1",
  "upload_id": "s3_upload_abc123xyz789",
  "parts": [
    {"part_number": 1, "etag": "\"d41d8cd98f00b204e9800998ecf8427e\""},
    {"part_number": 2, "etag": "\"0cc175b9c0f1b6a831c399e269772661\""}
  ]
}

Response: 202 Accepted

jsonCopy
{
  "video_id": "vid_998124a87c1",
  "status": "PROCESSING",
  "estimated_transcode_seconds": 180,
  "status_check_url": "/v1/videos/vid_998124a87c1/status"
}

3. Master HLS Playlist Manifest (master.m3u8)

Delivered via CloudFront edge:

m3u8Copy
#EXTM3U
#EXT-X-VERSION:6

# 1080p Profile (AVC1 / High Profile @ Level 4.2)
#EXT-X-STREAM-INF:BANDWIDTH=5000000,AVERAGE-BANDWIDTH=4500000,RESOLUTION=1920x1080,FRAME-RATE=60.000,CODECS="avc1.64002a,mp4a.40.2"
1080p/index.m3u8

# 720p Profile (AVC1 / Main Profile @ Level 3.1)
#EXT-X-STREAM-INF:BANDWIDTH=2500000,AVERAGE-BANDWIDTH=2200000,RESOLUTION=1280x720,FRAME-RATE=30.000,CODECS="avc1.4d401f,mp4a.40.2"
720p/index.m3u8

# 480p Profile (AVC1 / Main Profile @ Level 3.0)
#EXT-X-STREAM-INF:BANDWIDTH=1200000,AVERAGE-BANDWIDTH=1000000,RESOLUTION=854x480,FRAME-RATE=30.000,CODECS="avc1.4d401e,mp4a.40.2"
480p/index.m3u8

# 360p Profile (AVC1 / Baseline Profile @ Level 3.0)
#EXT-X-STREAM-INF:BANDWIDTH=700000,AVERAGE-BANDWIDTH=600000,RESOLUTION=640x360,FRAME-RATE=30.000,CODECS="avc1.42e01e,mp4a.40.2"
360p/index.m3u8

---

5. Data Models & Storage Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

DynamoDB Single-Table Schema (YouTubeCoreTable)

To achieve single-digit millisecond latency at petabyte scale, video metadata, channel relationships, and rendition profiles are consolidated into DynamoDB with On-Demand capacity and DynamoDB Accelerator (DAX).

---