Design YouTube Video Streaming Platform

1. Problem Statement & Scope

System Mission

Design a global-scale video sharing, processing, and streaming platform capable of ingesting over 500 hours of video content uploaded per minute ($30,000\text{ hours/hour}$), executing parallel Group of Pictures (GOP) chunk splitting and GPU transcoding pipeline Directed Acyclic Graphs (DAG) across heterogeneous codecs (H.264/AVC, H.265/HEVC, VP9, AV1), serving Adaptive Bitrate (ABR) streams via HLS and MPEG-DASH, and delivering media globally with sub-second Time-To-First-Frame (TTFF) via multi-tier edge caching, CloudFront Origin Shield, and S3 lifecycle storage tiering.

Synthesizing vector architecture diagram...

Functional Requirements

1. Resilient Chunked Video Upload: Support resumable, chunked multipart uploads for large video files (up to $128\text{ GB}$) over unreliable client connections with pause/resume capability.
2. Parallel GOP Transcoding Pipeline DAG: Decompose uploaded raw videos at keyframe (I-frame) boundaries into 4-second Group of Pictures (GOP) chunks, transcode in parallel across 6 resolution tiers ($4\text{K}, 1080\text{p}, 720\text{p}, 480\text{p}, 360\text{p}, 240\text{p}$) and multiple codecs (H.264, VP9, AV1), and extract normalized audio streams.
3. Adaptive Bitrate (ABR) Streaming: Generate HLS (.m3u8 master and media playlists) and MPEG-DASH (.mpd) manifests supporting dynamic runtime switching based on client network bandwidth and forward buffer health.
4. Metadata, Real-Time Counters & Search: Ingest and query video metadata, channel subscriptions, and manage atomic video view counting buffering millions of concurrent playback events.
5. Storage Lifecycle & CDN Optimization: Tier raw master files to Amazon S3 Glacier Instant Retrieval and use CloudFront Origin Shield to eliminate origin thundering herds on trending uploads.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.99\%$ availability for video playback ($< 52.6\text{ minutes}$ downtime/year); $99.95\%$ for ingestion.
• Latency SLOs:
  • Time-To-First-Frame (TTFF): $< 800\text{ ms}$ ($P95$) globally across edge CDN Points of Presence (PoPs).
  • Rebuffer / Stall Rate: $< 0.5\%$ of active playback sessions experiencing mid-stream buffering.
  • Transcoding Turnaround SLA: Video ready for standard playback ($720\text{p}$) in $\le 0.5\times$ video duration; full multi-codec ladder ready in $\le 1.5\times$ video duration.
• Durability Guarantee: $99.999999999\%$ (11 Nines) zero-loss durability for master raw videos and transcoded segments in Amazon S3.
• CAP / PACELC Classification:
  • Playback & CDN Tier: AP / PA-EL (Availability and low latency prioritized; stale view counts and playlist versions tolerable).
  • Transcoding & Metadata Ingestion: CP / PC-EC (Consistent DAG state transitions, strict idempotency on upload chunks).

---

2. Capacity & Scale Estimation

Traffic Calculations

• Active Viewer Base: 2 Billion Monthly Active Users (MAU), 500 Million Daily Active Users (DAU).
• Video Ingestion Volume:
  • Global Ingest Velocity: 500 hours of raw video uploaded per minute. $\text{Daily Uploaded Video Hours} = 500\text{ hrs/min} \times 60\text{ min/hr} \times 24\text{ hrs/day} = \mathbf{720,000\text{ hours/day}}$
  • Average raw video duration $= 10\text{ minutes}$ ($600\text{ seconds}$).
  • Total Daily Uploads: $\text{Daily Video Uploads} = \frac{720,000\text{ hours} \times 60\text{ min/hr}}{10\text{ min/video}} = \mathbf{4,320,000\text{ uploads/day}}$
  • Upload QPS: $\text{Avg Upload QPS} = \frac{4.32 \times 10^6\text{ uploads}}{86,400\text{ sec}} = \mathbf{50\text{ uploads/sec}} \quad (\text{Peak } 3\times: \mathbf{150\text{ uploads/sec}})$
• Playback Traffic:
  • 5 Billion video views per day across global audience.
  • Playback Initiation QPS: $\text{Avg Playback QPS} = \frac{5 \times 10^9\text{ views}}{86,400\text{ sec}} \approx \mathbf{57,870\text{ views/sec}}$ $\text{Peak Playback QPS (3}\times\text{ surge)} = 57,870 \times 3 \approx \mathbf{173,610\text{ views/sec}}$

Storage Calculations (3-Year Horizon)

• Raw Ingest Footprint:
  • Average source upload bitrate $= 20\text{ Mbps} = 2.5\text{ MB/sec}$.
  • Size per 10-min raw video: $600\text{ s} \times 2.5\text{ MB/s} = 1.5\text{ GB}$.
  • Daily Raw Storage: $\text{Daily Raw Storage} = 4.32 \times 10^6 \times 1.5\text{ GB} = \mathbf{6.48\text{ PB/day}}$
• Transcoding Profile Ladder (Renditions & Codecs):
  • Transcoding ladder per video across H.264/AVC baseline and AV1/VP9 high-efficiency codecs:
    • $4\text{K}$ ($3840\times 2160$ @ $60\text{ fps}$): $15\text{ Mbps}$
    • $1080\text{p}$ ($1920\times 1080$ @ $60\text{ fps}$): $5.5\text{ Mbps}$
    • $720\text{p}$ ($1280\times 720$ @ $30\text{ fps}$): $2.5\text{ Mbps}$
    • $480\text{p}$ ($854\times 480$ @ $30\text{ fps}$): $1.2\text{ Mbps}$
    • $360\text{p}$ ($640\times 360$ @ $30\text{ fps}$): $600\text{ kbps}$
    • $240\text{p}$ ($426\times 240$ @ $30\text{ fps}$): $300\text{ kbps}$
    • Isolated Audio Stream (AAC/Opus @ $128\text{ kbps}$): $128\text{ kbps}$
  • Cumulative Transcoded Bitrate $\approx 25.2\text{ Mbps} = 3.15\text{ MB/s}$ for one codec family (H.264/HEVC ladder). Publishing a second family (VP9 or AV1, roughly $30\text{--}50\%$ smaller at equal quality) adds $\approx 0.6\times$ again; the figures below count a single ladder, which is also why AV1 is only generated for videos that earn views (Section 2, lifecycle).
  • Transcoded footprint per 10-min video $= 600\text{ s} \times 3.15\text{ MB/s} = 1.89\text{ GB}$.
  • Daily Transcoded Ingress: $\text{Daily Transcoded Storage} = 4.32 \times 10^6 \times 1.89\text{ GB} = \mathbf{8.16\text{ PB/day}}$
• 3-Year Cumulative Volume (Raw + Transcoded): $\text{Storage/Year} = (6.48\text{ PB} + 8.16\text{ PB}) \times 365 = \mathbf{5.34\text{ Exabytes/year}}$ $\text{3-Year Cumulative Storage} = 5.34 \times 3 = \mathbf{16.03\text{ Exabytes}}$
• S3 Lifecycle Tiering Strategy:
  • Raw source masters moved to S3 Glacier Instant Retrieval after 7 days ($\approx 65\%$ storage cost reduction).
  • Transcoded videos with zero views in 90 days moved to S3 Glacier Flexible Retrieval; cold renditions ($4\text{K}$) purged and regenerated on-demand if re-requested.

Network Bandwidth

• Ingress Bandwidth (Raw Uploads): $\text{Avg Ingress Throughput} = 50\text{ uploads/s} \times 1.5\text{ GB} = 75\text{ GB/s} = \mathbf{600\text{ Gbps}}$ $\text{Peak Ingress Throughput (150 uploads/s)} = 225\text{ GB/s} = \mathbf{1.8\text{ Tbps}}$
• CDN Egress Bandwidth (Video Playback):
  • Average stream watch time $= 5\text{ minutes}$ ($300\text{ seconds}$) at $720\text{p}$ average ($2.5\text{ Mbps} = 312.5\text{ KB/s}$).
  • Data delivered per view $= 300\text{ s} \times 312.5\text{ KB/s} = 93.75\text{ MB}$.
  • Total Daily Egress: $\text{Daily Egress Volume} = 5 \times 10^9 \times 93.75\text{ MB} = \mathbf{468.75\text{ PB/day}}$
  • Average and Peak Egress Bandwidth: $\text{Avg Egress Bandwidth} = \frac{468.75 \times 10^{15} \times 8\text{ bits}}{86,400\text{ sec}} \approx \mathbf{43.4\text{ Tbps}}$ $\text{Peak Egress Bandwidth (3}\times\text{)} = 43.4 \times 3 \approx \mathbf{130.2\text{ Tbps}}$
• Origin Shield Egress (Assuming $\ge 98.5\%$ Edge Cache Hit Ratio): $\text{Origin Bandwidth Demand} = 43.4\text{ Tbps} \times (1 - 0.985) = \mathbf{651\text{ Gbps (Peak: 1.95 Tbps)}}$

Memory & Cache Sizing (80/20 Pareto Working Set)

• Top $20\%$ of videos generate $80\%$ of all playback sessions.
• Daily active catalog working set $= 5 \times 10^9 \times 0.20 = 1\text{ Billion view sessions}$ referencing approximately 50 Million unique trending videos.
• Cache entry per video (manifests, metadata, initial segment headers): $4\text{ KB}$. $\text{ElastiCache Redis Hot Cache} = 50 \times 10^6 \times 4\text{ KB} = \mathbf{200\text{ GB RAM}}$
• View counter buffer (100 Million active counters): $100 \times 10^6 \times 64\text{ bytes} = \mathbf{6.4\text{ GB RAM}}$

Fleet Sizing & GPU Transcoder Provisioning

• Daily transcoding compute demand: $720,000\text{ hours of raw video per day}$.
• Target transcoding speed using AWS Batch with NVIDIA A10G GPUs (g5.2xlarge instances):
  • Hardware NVENC encodes a single rendition at $\approx 8\times$ real-time (a 4-second GOP chunk in $\approx 500\text{ ms}$, the figure used in Section 3).
  • 1 GPU therefore produces $8$ rendition-hours per real-time hour $\implies 192\text{ rendition-hours/day}$ per GPU.
  • Every uploaded hour becomes 6 rendition-hours (the 4K to 240p ladder), so the demand is $720{,}000 \times 6 = 4{,}320{,}000$ rendition-hours/day. Sizing the fleet against raw video-hours alone undercounts it by $6\times$, which is the most common mistake in this estimate.
• Base GPU Fleet Size: $\text{Required GPUs} = \frac{4,320,000\text{ rendition-hours/day}}{192\text{ rendition-hours/GPU/day}} = \mathbf{22,500\text{ GPUs}}$
• Provisioning for $1.5\times$ peak ingest surges: $\text{Peak GPU Fleet} = 22,500 \times 1.5 = \mathbf{33,750\text{ AWS Batch GPU Workers}}$ Decoding the source once and feeding all six encoders from that decode (NVDEC $\to$ 6 $\times$ NVENC on the same card) roughly halves this in practice; the ladder count, not the raw hours, is what dominates the bill. Operated as $80\%$ Spot instances backed by $20\%$ On-Demand buffer with auto-requeuing on spot reclamation.

---

3. AWS-First High-Level Architecture

Synthesizing vector architecture diagram...

Follow a video from upload to playback. In the "Ingestion & Upload Control Plane" panel, the creator uploads the raw file to S3 (with transfer acceleration) and its metadata goes to DynamoDB. The S3 upload event reaches EventBridge, which starts the "Chunked Transcoding Pipeline DAG" panel: Step Functions has the file split into roughly 4-second chunks at keyframes, each chunk is queued in SQS and transcoded in parallel by the GPU fleet, and the packager assembles HLS and DASH manifests, so a one-hour video takes about as long as its slowest chunk. Viewers stream through the "Multi-Tier Edge Acceleration & CDN Layer" panel, where Origin Shield merges identical cache misses from many edges into one request to S3. In the "Playback, Counters & Discovery Services" panel, view counts flow through Kinesis and Flink into Redis counters, which are flushed to DynamoDB every 60 s instead of costing one write per view. Raw uploads move to Glacier after 7 days, and old renditions after 90.

Data Flow Walkthrough

1. Resilient Ingestion: Creators initiate multipart uploads via the Upload Control Service, obtaining presigned S3 URLs. Files stream directly to S3 Raw Ingest with S3 Transfer Acceleration.
2. GOP Chunk Splitting: S3 triggers AWS Step Functions via EventBridge. The GOP Splitter decodes video keyframes (I-frames) and dispatches 4-second GOP chunk transcode jobs to Amazon SQS.
3. Parallel GPU Transcoding: AWS Batch EC2 g5.2xlarge instances poll SQS, transcode GOP chunks across the multi-resolution ladder using hardware NVENC, and write .ts/.m4s chunks to S3 Transcoded Segments.
4. Packaging & Manifest Generation: Once all chunks complete, the Packager Worker compiles the variant playlists (1080p/index.m3u8, etc.) and master manifest (master.m3u8), marking the video READY in DynamoDB.
5. Multi-Tier Edge Playback: Players fetch the master playlist via CloudFront. Origin Shield deduplicates concurrent requests, and edge PoPs serve media segments with byte-range optimization. View beacons stream to Kinesis for sharded Redis buffering before DynamoDB persistence.

Concrete Step-by-Step Request Walkthrough: Tracing Upload to Playback

---

4. API Interface Design

1. Initialize Multipart Video Upload (POST /v1/videos/uploads/initialize)

Initializes a resumable chunked upload session and generates presigned S3 URLs.

httpCopy
POST /v1/videos/uploads/initialize HTTP/1.1
Host: api.media.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "title": "Distributed Consensus in Practice: Raft & Paxos",
  "description": "Comprehensive architectural guide to state machine replication.",
  "category_id": 28,
  "privacy": "PUBLIC",
  "file_name": "consensus_master_4k.mov",
  "file_size_bytes": 10737418240,
  "chunk_size_bytes": 26214400,
  "total_parts": 410,
  "source_codec": "prores_422",
  "client_md5": "d41d8cd98f00b204e9800998ecf8427e"
}

Response: 201 Created

jsonCopy
{
  "video_id": "vid_998124a87c1",
  "upload_id": "s3_upload_987654321_abcde",
  "presigned_parts": [
    {
      "part_number": 1,
      "url": "https://s3.us-east-1.amazonaws.com/raw-ingest/vid_998124a87c1?partNumber=1&uploadId=s3_upload_987654321_abcde&X-Amz-Signature=...",
      "expires_at": 1773651600
    },
    {
      "part_number": 2,
      "url": "https://s3.us-east-1.amazonaws.com/raw-ingest/vid_998124a87c1?partNumber=2&uploadId=s3_upload_987654321_abcde&X-Amz-Signature=...",
      "expires_at": 1773651600
    }
  ],
  "expires_in_seconds": 86400
}

2. Complete Multipart Upload & Trigger Transcoding (POST /v1/videos/uploads/complete)

Verifies part ETags in S3 and activates the Step Functions DAG.

httpCopy
POST /v1/videos/uploads/complete HTTP/1.1
Host: api.media.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "video_id": "vid_998124a87c1",
  "upload_id": "s3_upload_987654321_abcde",
  "parts": [
    {"part_number": 1, "etag": "\"d41d8cd98f00b204e9800998ecf8427e\""},
    {"part_number": 2, "etag": "\"0cc175b9c0f1b6a831c399e269772661\""}
  ]
}

Response: 202 Accepted

jsonCopy
{
  "video_id": "vid_998124a87c1",
  "status": "PROCESSING",
  "estimated_completion_seconds": 120,
  "status_url": "/v1/videos/vid_998124a87c1/status"
}

3. Master HLS Playlist Manifest (GET /v1/videos/{video_id}/master.m3u8)

Served via CloudFront with ABR variant ladder:

m3u8Copy
#EXTM3U
#EXT-X-VERSION:6
#EXT-X-INDEPENDENT-SEGMENTS

# Audio Rendition (Demuxed AAC 128kbps)
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aac",NAME="English",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="en",URI="audio/128k/index.m3u8"

# 4K UHD Profile (HEVC / Main 10 @ Level 5.1)
#EXT-X-STREAM-INF:BANDWIDTH=15000000,AVERAGE-BANDWIDTH=14000000,RESOLUTION=3840x2160,FRAME-RATE=60.000,CODECS="hvc1.2.4.L153.B0,mp4a.40.2",AUDIO="audio-aac"
4k/index.m3u8

# 1080p Profile (AVC1 / High Profile @ Level 4.2)
#EXT-X-STREAM-INF:BANDWIDTH=5500000,AVERAGE-BANDWIDTH=5000000,RESOLUTION=1920x1080,FRAME-RATE=60.000,CODECS="avc1.64002a,mp4a.40.2",AUDIO="audio-aac"
1080p/index.m3u8

# 720p Profile (AVC1 / Main Profile @ Level 3.1)
#EXT-X-STREAM-INF:BANDWIDTH=2500000,AVERAGE-BANDWIDTH=2200000,RESOLUTION=1280x720,FRAME-RATE=30.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio-aac"
720p/index.m3u8

# 480p Profile (AVC1 / Main Profile @ Level 3.0)
#EXT-X-STREAM-INF:BANDWIDTH=1200000,AVERAGE-BANDWIDTH=1000000,RESOLUTION=854x480,FRAME-RATE=30.000,CODECS="avc1.4d401e,mp4a.40.2",AUDIO="audio-aac"
480p/index.m3u8

# 360p Profile (AVC1 / Baseline Profile @ Level 3.0)
#EXT-X-STREAM-INF:BANDWIDTH=600000,AVERAGE-BANDWIDTH=500000,RESOLUTION=640x360,FRAME-RATE=30.000,CODECS="avc1.42e01e,mp4a.40.2",AUDIO="audio-aac"
360p/index.m3u8

4. Status Codes & Error Contracts

---

5. Data Models & Storage Architecture

Physical Storage Layout: Amazon S3

DynamoDB Single-Table Schema (YouTubeCoreTable)

---