Netflix: Global Streaming, the API Edge and Regional Evacuation

This page is one interview loop in three rounds, built on a real company's public engineering history. All three rounds design the same system. Each round is an era of Netflix's history that answered a crisis, and each opens with the interviewer raising the scope, so the design from the round before has to evolve.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: Why Netflix?

Two Systems: a Small Brain and Huge Pipes

When you press Play on Netflix, two very different systems go to work.

• The brain (the control plane). A handful of API calls: who are you, what can you watch, where did you stop, and which server should send you the video? These requests are small, a few kilobytes each, but every one of them must work.
• The pipes (the data plane). Then the video itself arrives: gigabytes an hour, from a server as close to your home as possible.

Netflix published this split plainly: its control plane runs on AWS, and its video is delivered by Open Connect, Netflix's own content delivery network, whose servers sit inside internet providers' networks and at internet exchange points.

Synthesizing vector architecture diagram...

The thin arrows carry kilobytes and must never fail; the thick arrow carries almost all the bytes and must be close to the viewer. The two planes fail, scale and cost money in completely different ways.

A few words we'll use all page:

What Makes It Hard

• The brain must never be down. If the control plane is down, nobody can press Play, even though the video servers are fine.
• The pipes are enormous. Netflix gave as its reason for building its own CDN that it had grown to be "a significant portion of overall traffic" on consumer ISP networks. Evening is when everyone watches, and evening is when the internet is busiest.
• Everything changes all the time. Hundreds of services, hundreds of deploys a day and more than a thousand device types mean something is always broken somewhere.

The Question the Whole Loop Answers

How do we keep the control plane always available, and push the video bits as close to the viewer as possible?

The answer grows every round:

• Round 1: leave the single data center, make every service stateless and spread it across Availability Zones, keep member data in Cassandra, and let CDNs carry the video.
• Round 2: put one programmable front door (Zuul) in front of hundreds of services, find instances through a registry (Eureka), contain slow dependencies with circuit breakers and concurrency limits, cache hot data in every zone (EVCache), and break things on purpose (chaos engineering).
• Round 3: build our own CDN inside ISPs, filled overnight; run three AWS regions active-active; and practice moving all traffic out of a region in minutes.

---

Round 1 · Mid-level · "Era 1: From a Data Center to the Cloud"

~35 min · SDE II (L5) · 1 AWS region, 3 AZs · 1B API requests/day, ~35,000/s planned for the evening peak (assumption; Netflix reported ~20,000/s at its 2011 peak) · video through third-party CDNs · no single point of failure · 99.99% (Netflix's internal goal)

R1.1 Establish Design Scope

The interviewer sets the scene: "It's 2008. We rent DVDs by mail, and a year ago we started streaming. Everything runs as one big application on one relational database in our own data center. In August the database got corrupted. Design where we go from here."

Out of scope for this round: device-specific APIs, our own CDN, and running in more than one region.

What 99.99% allows. A 30-day month has 43,200 minutes; 0.01% of that is 4.32 minutes of downtime a month.

R1.2 Functional Requirements, Derived Step by Step

Not yet: a different API shape for each device family, our own CDN, several regions.

R1.3 Non-Functional Requirements: the Questions

We name each quality first; the numbers come in R1.7.

• Availability first. Can members press Play if one server, one database node or one data center fails? Today, the answer for the database is no.
• Scale for evenings. Traffic follows the evening: many times more people watch at 9 p.m. than at 5 a.m. Can we add capacity for the peak and give it back afterward?
• Keep video off the API. Video bytes and API calls must not share servers, links or budgets.
• Write availability for bookmarks. Position updates arrive constantly while people watch. If we can't save one, the member resumes at the wrong place tomorrow.

R1.4 The API

This API is illustrative: it's a design that fits the problem, not Netflix's actual interface.

Get a playback manifest

httpCopy
POST /v1/playback/manifest HTTP/1.1
Host: api.example-streaming.com
Authorization: Bearer <session token>
Content-Type: application/json

{ "profile_id": "p_3019", "title_id": "t_81049281", "device": { "type": "tv", "model": "brandX-2008", "drm": ["playready"] } }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "playback_id": "pb_01HZX894",
  "resume_position_s": 1452,
  "streams": [
    { "bitrate_kbps": 1000, "url": "https://cdn-a.example.net/t_81049281/1000.ismv?exp=1223476800&sig=..." },
    { "bitrate_kbps": 2200, "url": "https://cdn-a.example.net/t_81049281/2200.ismv?exp=1223476800&sig=..." }
  ],
  "fallback_cdn_base": "https://cdn-b.example.net/t_81049281/",
  "license": { "url": "https://license.example-streaming.com/playready", "challenge_token": "..." }
}

Save the position

httpCopy
PUT /v1/bookmarks/p_3019/t_81049281 HTTP/1.1
Authorization: Bearer <session token>
Content-Type: application/json

{ "position_s": 1512, "playback_id": "pb_01HZX894", "client_time": "2008-10-08T20:41:07Z" }

httpCopy
HTTP/1.1 204 No Content

• The manifest points to CDN URLs, signed with an expiry, so only this member can fetch the files and only for a while. The API never sends a video byte.
• The bookmark is a PUT: sending the same position twice leaves the same result, so the player can retry freely.

Status codes

Recap

• Four operations; the API returns where video lives, never the video.
• Playback and bookmark calls are safe to retry.

R1.5 Design Evolution: From One Database to Many Machines

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

One Java application behind a hardware load balancer, on a few large servers in our own data center, with one large relational database for everything: members, the catalog, DVD queues, viewing history. Video comes from third-party CDNs.

Synthesizing vector architecture diagram...

Every request, for every feature, ends at one database. When it's corrupted, the whole business stops.

It works until it doesn't, and in August 2008 it didn't.

Step 1.1: The Database Got Corrupted and the Business Stopped

Step 1.2: Should Video Go Through Our Servers?

Primitive: Distributed Cache Patterns and Eviction · Loop: Design a Video Streaming Platform (transcoding, segments and adaptive bitrate, which this page doesn't re-teach)

Step 1.3: Viewing History Grows Forever and Must Always Be Writable

Primitive: Database Sharding and Partition Keys · Loop: Design a Distributed Key-Value Store (quorums and last-writer-wins, from the inside)

Step 1.4: Evenings Bring Three Times the Traffic

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

Read it as two planes. The control plane is stateless servers in three AZs with state in a replicated database; the data plane is somebody else's CDN. Nothing in the picture exists only once.

Sources for this round

• Izrailevsky, Vlaovic and Meshenberg, Completing the Netflix Cloud Migration, Netflix, February 2016: the August 2008 corruption and three days without DVD shipments; the move away from vertically scaled single points of failure; completion in January 2016.
• Ciancutti, Four Reasons We Choose Amazon's Cloud as Our Computing Platform, Netflix TechBlog, December 2010.
• Netflix, Q4 2008 results press release, January 2009: about 9,390,000 subscribers.
• Schmaus, Making the Netflix API More Resilient, Netflix TechBlog, December 2011: "a billion requests a day", and around 20,000 requests/s at peak.
• Duvedi, Li, Garg and Fisher-Ogden, Scaling Time Series Data Storage, Part I, Netflix TechBlog, January 2018: Cassandra for viewing history, the 9:1 write-to-read ratio, the EVCache layer, the redesign.
• Netflix, How Netflix Works With ISPs Around the Globe, March 2016: streaming launched in 2007; third-party CDNs before Open Connect.
• Meshenberg, Gopalani and Kosewski, Active-Active for Multi-Regional Resiliency, Netflix TechBlog, December 2013: the 99.99% internal availability goal.
• Gonigberg et al., Open Sourcing Zuul 2, Netflix TechBlog, May 2018: slow warm-up of new instances.

Everything else in this round (instance counts, bitrates, the table schemas, the API shapes) is our design or our assumption.

R1.7 Numbers

Every figure here is an assumption except the ones marked published. The point is the arithmetic, not the exact values.

API traffic

API fleet. We assume one instance handles 500 requests/s at our target CPU.

Autoscaling moves the fleet between 12 and 105, about 9 to 1.

Bookmarks. We assume 300,000 people watch at the evening peak, and each player saves its position every 60 seconds.

• Writes: 300,000 ÷ 60 s = 5,000 bookmark writes/s, about 14% of the peak API traffic (5,000 ÷ 35,000).
• With 3 copies each, Cassandra performs 5,000 × 3 = 15,000 replica writes/s. On 6 nodes (2 per AZ), that's 2,500 writes/s per node, well within what one Cassandra node handles (a comfortable budget of 5,000/s per node, assumption).
• Reads: at the published ratio of 9 writes per read, about 5,000 ÷ 9 ≈ 560 history reads/s, and most hit the cache.

History storage. Assume 10M members, 1,000 records each over the years, 200 bytes a record: 10,000,000 × 1,000 × 200 B = 2 TB. Three copies make 6 TB, 1 TB per node on 6 nodes.

Bytes: why the API must not carry video

Video is more than 99% of the bytes (600 ÷ 602.8 = 99.5%). That's the whole argument for step 1.2 in one line.

R1.8 Trade-Offs

Our own data center vs AWS

Netflix's own summary of the outcome in 2016: "the cloud allows one to build highly reliable services out of fundamentally unreliable but redundant components".

Relational database vs Cassandra for viewing history

Third-party CDNs vs our own CDN (not yet)

At 2008's scale, buying is right. Round 3 revisits it when Netflix is a significant share of ISP traffic.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Names the single point of failure and removes it with replication and statelessness, not with better backups.
• Separates the control plane from the data plane, and backs it with a byte count.
• Picks a store for viewing history from the access pattern: write-heavy, keyed by member, availability over consistency.
• Designs Cassandra tables per screen, with the partition key the app already has.
• Sizes the fleet for losing an AZ, not just for the peak.
• Autoscales, and knows new instances need warm-up.

Follow-up questions

1. "Why not keep the relational database and add read replicas?" Answer: replicas scale reads, but viewing history is 9 writes for every read, and every write still goes to one primary. A primary failure still stops writes until failover finishes.

2. "Two devices on the same profile save different positions at the same moment. What happens?" Answer: Cassandra keeps the write with the later timestamp. The member resumes at one of the two positions, which is acceptable for a bookmark. It would not be acceptable for money, which is why we don't keep money this way.

3. "Why sign the CDN URLs if the video is encrypted anyway?" Answer: encryption (DRM) stops people from watching without a license; signing stops strangers from using our CDN bill to download files at all. Different jobs.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Era 2: Hundreds of Microservices Behind One Front Door"

~40 min · Senior SDE (L6) · one region per geography, 3 AZs · 500,000 edge requests/s per region at the peak (assumption) · 10M device connections per region (assumption) · hundreds of services and deploys a day · contain every failure

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "In August 2008 Netflix's one relational database was corrupted and DVDs didn't ship for three days. Netflix decided to leave vertically scaled single points of failure and moved to AWS, a migration that finished in January 2016. We made every service stateless and spread it across three Availability Zones, sized so two AZs carry our planned evening peak of 35,000 API requests a second, 35 instances per AZ, autoscaling down to 12 instances at night. Member data like bookmarks and viewing history lives in Cassandra, keyed by profile, with three copies (one per AZ) and quorum writes, because history is written nine times for every read and availability matters more than instant consistency. Video never touches our servers: the API returns a manifest with signed URLs on third-party CDNs, because video is about 214 times the API's bytes. Open costs: every device calls one generic API, services find each other by configuration, a slow dependency can stall everything, one cache cluster, and we learn about failures from members."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: nothing exists once, and video goes around our servers.

Round 1 step summary

Open costs: one generic API for every device, hand-configured service addresses, no isolation between dependencies, one cache cluster, failures found by members.

R2.1 The Scope Raise

Interviewer: "It's a few years later. The monolith has become hundreds of services owned by different teams. We support well over a thousand device types, and each wants different data in a different shape. Last week a service nobody considers critical slowed down and took the home page with it. Teams deploy many times a day. And we want devices to keep a connection open, so we can push to them."

Scope change

The "Not yet" list from R1.2 comes back: device-specific APIs are in scope now.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

A front door for all device traffic. Every device request enters through one gateway fleet, which authenticates it, applies routing rules, and sends it to the right service. Netflix published the gateway's structure, which the diagram shows: inbound filters run before the request is proxied (authentication, routing, decorating the request); an endpoint filter either answers directly or proxies to the backend ("origin" in Zuul's terms); outbound filters run on the response (compression, metrics, headers).

Synthesizing vector architecture diagram...

Netty handlers do the network work on both sides; filters hold all the logic. Change the filters and the same gateway solves a different problem, which is why Netflix could deploy the same core at the edge and, with fewer filters, for internal traffic.

Device-specific endpoints. In 2012 Netflix moved from a one-size-fits-all REST API to "a platform for API development" where each UI team writes endpoints for its own devices. An illustrative endpoint (our shapes):

httpCopy
GET /tv/v2/home?profile_id=p_3019&rows=12 HTTP/1.1
Host: api.example-streaming.com
Authorization: Bearer <device token>

jsonCopy
{
  "rows": [
    { "id": "continue_watching", "items": [ { "title_id": "t_81049281", "box_art": "https://img.example.net/t_81049281/tv_1080.jpg", "resume_position_s": 1512 } ] },
    { "id": "top10_country", "items": [ { "title_id": "t_80100172", "box_art": "https://img.example.net/t_80100172/tv_1080.jpg" } ] }
  ],
  "degraded": ["ratings"]
}

One request builds the whole TV home screen with TV-sized artwork; a phone's endpoint returns different fields. degraded tells the UI that ratings came from a fallback (step 2.3).

Registration and discovery. Every instance registers itself in a registry and renews a lease; callers look services up by name. Eureka's REST interface looks like this (simplified):

httpCopy
POST /eureka/v2/apps/RATINGS HTTP/1.1
Content-Type: application/json

{ "instance": { "instanceId": "i-0a1b2c3d", "app": "RATINGS", "ipAddr": "10.1.2.3", "vipAddress": "ratings", "port": { "$": 8080, "@enabled": "true" }, "status": "UP" } }

httpCopy
PUT /eureka/v2/apps/RATINGS/i-0a1b2c3d HTTP/1.1

The first registers; the second is the heartbeat, sent every 30 seconds.

Every dependency declares its fallback (our configuration shape):

yamlCopy
dependencies:
  ratings:
    timeout_ms: 300
    concurrency_limit: adaptive
    fallback: fail_silent        # home page shows no stars
  subscriber:
    timeout_ms: 200
    concurrency_limit: adaptive
    fallback: custom             # use the member data in the signed request identity
  playback_license:
    timeout_ms: 800
    concurrency_limit: adaptive
    fallback: fail_fast          # no safe default: return 503 fast

R2.4 Design Evolution: A Front Door, a Registry and Containment

Step 2.1: Devices Talk to Dozens of Services

Primitive: API Gateway and Reverse Proxy

Step 2.2: Service Addresses Change Constantly

Primitive: Gossip Protocol and Failure Detection (heartbeats, timeouts and why a registry must not panic)

Step 2.3: One Slow Service Took Down the Home Page

Primitive: Circuit Breaker, Bulkhead and Fault Tolerance · Drill: The slow recommendation service that took down checkout · Loop: Distributed rate limiter, step 3.4 (load shedding vs rate limiting)

Step 2.4: Hot Data Must Be Fast in Every AZ

Primitive: Distributed Cache Patterns and Eviction

Step 2.5: Failures Surprise Us in Production

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Follow a request left to right: one front door, a device-shaped API, then direct calls between services, each wrapped in a breaker and a limit, over addresses from Eureka. Dotted lines are lookups, misses and injected failures, not the request path. The load balancer at the front is our AWS mapping; Netflix fronted Zuul with AWS load balancers, and the exact type is not something we rely on.

Sources for this round

• Meshenberg, Gopalani and Kosewski, Active-Active for Multi-Regional Resiliency, Netflix TechBlog, December 2013: Zuul opened to the community in June 2013.
• Jacobson, Embracing the Differences: Inside the Netflix API Redesign, Netflix TechBlog, July 2012: 800+ device types; device-specific endpoints.
• Cockcroft, A Closer Look at the Christmas Eve Outage, Netflix TechBlog, December 2012: "over a thousand different streaming devices".
• The Cloud Gateway team, Zuul 2: The Netflix Journey to Asynchronous, Non-Blocking Systems, Netflix TechBlog, September 2016: Netty, connection scaling, the measured results, 83M members.
• Gonigberg et al., Open Sourcing Zuul 2, Netflix TechBlog, May 2018: 80+ clusters, ~100 origins, more than 1M requests/s, filter types, self-service routing, load balancing features.
• Netflix/zuul on GitHub (checked September 2026): active, Netty-based zuul-core.
• Netflix TechBlog, Edge Authentication and Token-Agnostic Identity Propagation, February 2021: authentication in Zuul filters, the Passport, the move to NodeQuark backends for frontends.
• Eureka wiki, Understanding Eureka client/server communication and Server self-preservation mode: 30 s heartbeats, 90 s eviction, 30 s delta fetches, the 15% threshold.
• Vroom, Mulcahy, Yuan and Gulewich, Zero Configuration Service Mesh with On-Demand Cluster Discovery, Netflix TechBlog, August 2023: why client-side load balancing, Eureka's degraded mode, the move to Envoy.
• Schmaus, Making the Netflix API More Resilient, December 2011: breakers, bounded thread pools, 10 s windows, three fallback kinds.
• Netflix/Hystrix README: maintenance mode; resilience4j and adaptive limits.
• Landau, Thurston and Bozarth, Performance Under Load, Netflix TechBlog, March 2018: adaptive concurrency limits.
• The EVCache team, Caching for a Global Netflix, Netflix TechBlog, March 2016: EVCache scale, per-region redundancy, "hundreds of microservices".
• Netflix/EVCache wiki: reads from the client's own zone, writes to all zones, zone fallback.
• The Netflix Simian Army, July 2011; Basiri, Hochstein, Thosar and Rosenthal, Chaos Engineering Upgraded, September 2015; ChAP: Chaos Automation Platform, July 2017.

The fleet sizes, request rates, cache sizes, the fallback configuration and the AWS load balancer mapping are ours.

R2.6 Numbers and Cost

All figures are assumptions unless marked published.

The gateway fleet for one region. Assume 500,000 edge requests/s at the peak and 5,000 requests/s per Zuul instance at our target CPU.

For scale: Netflix's published total in 2018 was more than 1 million requests/s across all its Zuul clusters.

Connections: why Zuul 2. Assume 10M devices hold a connection to this region at the peak.

• Per instance: 10,000,000 ÷ 150 = 66,667 connections.
• Thread-per-connection (Zuul 1's model) would need 66,667 threads per instance. The JVM's default stack reservation on 64-bit Linux is 1 MB a thread, so that's about 65 GB of reserved stack address space before any work, plus the scheduler cost of switching among them.
• An event loop per core (Zuul 2's model): on a 16-core instance, 16 threads serve all 66,667 connections, about 4,167 each; each idle connection costs a socket and a small buffer.

Little's law in the gateway. At 3,333 requests/s per instance and a 50 ms origin, 3,333 × 0.05 = 167 requests are in flight. If an origin slows to 2 s, the same arrivals mean 3,333 × 2 = 6,667 in flight. Zuul 1 needs a thread for each; Zuul 2 needs only memory for each, which is why Zuul 2 still needs concurrency limits per origin (Netflix open-sourced "origin concurrency protection" with Zuul 2): memory runs out too, just later.

EVCache for one region. Assume 10M cache operations/s at the peak (9M reads, 1M writes), 1 KB average item, and a 30 TB hot set. Three regions at this rate make 30M/s, the same order as Netflix's published 2016 peak.

Data moving between AZs in one region is charged $0.01/GB in each direction, so $0.02 for every GB that crosses. Reading from our own AZ's copy is worth about $315,000 a month here; the cost of writing every copy is the price of having one.

The load balancer in front of Zuul (our AWS mapping). A Network Load Balancer with a TCP listener, passing TLS through to Zuul. NLB bills the largest of three dimensions in NLB capacity units (NLCUs) at $0.006 per NLCU-hour in us-east-1. For TCP, one NLCU is 800 new connections/s, 100,000 active connections, or 1 GB processed an hour. Assume 20,000 new connections/s and 4 KB per request and response together.

Bytes dominate: 7,200 × $0.006 = $43.20 an hour, × 730 hours ≈ $31,500 a month, plus $0.0225 an hour for the load balancer itself (about $16).

A second copy of the same bytes. If we also put an internal Application Load Balancer between Zuul and the device APIs, it processes the same 2 GB/s again: 7,200 LCUs (1 GB an hour per LCU for EC2 targets) × $0.008 ≈ $57.60 an hour, ≈ $42,000 a month more, plus one more box on every request. Client-side load balancing through Eureka avoids both, which is the choice Netflix made for traffic between services.

R2.7 Trade-Offs

One programmable gateway vs a service mesh

They aren't rivals: Netflix published both, for different traffic. Netflix gave the reason for the mesh: keeping the same features "in more languages, in more clients" consistent was the hard part.

Client-side vs server-side load balancing

The cost of chaos engineering

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Puts one programmable front door in front of the services and explains its filter model.
• Explains non-blocking honestly: connection scaling yes, CPU savings only when work per request is small.
• Uses a registry with leases and client-side load balancing, and knows how stale a crashed instance can stay (3 to 5 minutes, not the documented 90 s).
• Uses Little's law to show why a slow optional dependency exhausts a shared pool, and contains it with bulkheads, breakers, fallbacks and adaptive limits.
• Knows Hystrix is in maintenance and what replaced its role.
• Designs a per-AZ cache and prices the cross-AZ bytes.
• Proposes chaos engineering as experiments with a steady state and an automatic stop.

Follow-up questions

1. "If Zuul 2 didn't make the API cluster cheaper, why did Netflix do it?" Answer: connection scaling. Persistent connections from every device enable push and fewer chatty requests. CPU-bound work costs the same CPU either way; idle connections are where thread-per-connection loses.

2. "Eureka is down for ten minutes. What breaks?" Answer: calls between running services keep working from each caller's cached copy. New instances can't be found, crashed ones aren't removed, and deploys should pause. That's why the design treats Eureka as important but not on the request path.

3. "Why fail silent for ratings but fail fast for the playback license?" Answer: a page without stars is still a page; a Play without a license can't play. A fake license would be worse than a quick, honest error the device can retry.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Era 3: Own the CDN, and Survive Losing a Region"

~45 min · Principal (L7) · 190 countries · more than 325M paid memberships · about 536M hours watched a day · 3 active-active AWS regions · Open Connect partnerships with more than a thousand ISPs · 3M edge requests/s worldwide (assumption) · evacuate a region in under 10 minutes

R3.0 Where We Left Off

Round 2 in 60 seconds. "Hundreds of services now sit behind one programmable front door, Zuul. Its filters authenticate each request and create a Passport, an HMAC-protected identity object that travels to every service; its routing rules run canaries; its load balancer avoids cold, failing and busy instances. Zuul moved to Netty in 2016: the win was cheap persistent connections, not CPU. Behind it, device-specific APIs build each screen in one call. Services find each other through Eureka (30-second heartbeats, a documented 90-second lease that in practice expires in 3 to 5 minutes, cached registries) and call each other directly, with bulkheads, circuit breakers, fallbacks and adaptive concurrency limits; Hystrix is in maintenance mode since 2018. EVCache keeps a copy of hot data in every AZ and reads from the local one. Chaos Monkey, chaos experiments and ChAP exercise our failure paths continuously. Per region: 500,000 requests a second, 150 Zuul instances, 10M connections. Open costs: third-party CDNs carry every video byte, and each region is a world of its own: if a region fails, its members can't play."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: one front door, direct calls with containment, a cache in every zone, all inside one region.

Round 2 step summary

Open costs: someone else's CDN for every byte, far from viewers; one region per member, with nothing to fail over to.

R3.1 The Scope Raise

Interviewer: "Two things happened. On Christmas Eve 2012, a problem in AWS's load balancer service in us-east-1 stopped playback on many TV devices for hours. And we're about to launch in nearly every country, while in the evening we're already a significant share of every ISP's traffic. Deliver the video yourselves, and make the control plane survive the loss of a whole region."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

The manifest now points at our own servers. The playback service asks the steering service for appliances that have the files, are healthy, and are close to this client in network terms (our response shape):

jsonCopy
{
  "playback_id": "pb_01J2K7Q9",
  "resume_position_s": 1512,
  "servers": [
    { "rank": 1, "base_url": "https://oca-17.isp-a-sea.example.net/", "site": "embedded in ISP A" },
    { "rank": 2, "base_url": "https://oca-04.ix-sea.example.net/", "site": "internet exchange, Seattle" },
    { "rank": 3, "base_url": "https://oca-22.ix-pdx.example.net/", "site": "internet exchange, Portland" }
  ],
  "files": [ { "stream": "video_3000k", "path": "t_81049281/v3000.mp4" }, { "stream": "audio_en", "path": "t_81049281/a_en.mp4" } ],
  "url_expires_at": "2026-09-28T23:40:00Z"
}

Each appliance reports to the control plane. Netflix published what an appliance reports: its health, the BGP routes it has learned from the ISP router it peers with, and which files it stores. An illustrative report:

jsonCopy
{ "oca_id": "oca-17", "site": "isp-a-sea", "healthy": true, "egress_utilization": 0.62,
  "bgp_prefixes_learned": ["198.51.100.0/22", "2001:db8:1200::/40"],
  "files_changed": { "added": 1834, "removed": 912 }, "reported_at": "2026-09-28T19:40:00Z" }

Evacuation controls (ours). One internal request records the decision; automation carries it out, and the call is refused unless its checks pass:

httpCopy
POST /traffic/v1/evacuations HTTP/1.1
Content-Type: application/json

{ "evacuate": "us-east-1", "targets": { "us-west-2": 1.0 }, "reason": "SPS -38% in us-east-1; probes failing from us-west-2 and eu-west-1",
  "requested_by": "oncall:traffic", "idempotency_key": "evac-2026-09-28-01" }

httpCopy
HTTP/1.1 202 Accepted
Content-Type: application/json

{ "evacuation_id": "evac-2026-09-28-01", "state": "CAPACITY_ATTACHING",
  "checks": { "targets_capacity_ready": "pending", "ec2_vcpu_quota_headroom": "ok", "residency_rules": "ok", "at_least_two_regions_on": "ok" } }

Replicate member data everywhere. Each Cassandra keyspace for member data keeps three copies in every region (CQL, a SQL-like schema language):

sqlCopy
CREATE KEYSPACE member_playback WITH replication = {
  'class': 'NetworkTopologyStrategy', 'us-east-1': 3, 'us-west-2': 3, 'eu-west-1': 3 };

R3.4 Design Evolution: Our Own CDN and Three Regions

Step 3.1: Video Is Expensive and Far Away

Synthesizing vector architecture diagram...

Fill runs at night in each time zone. Only the fill master pulls from S3; everything else copies from a nearby OCA, so a new episode crosses the ocean once, not once per viewer.

Loop: Design a Video Streaming Platform (multi-CDN steering and ISP caches from the buyer's side, step 3.2)

Step 3.2: A Region Failed and Members Couldn't Play

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active

Step 3.3: Move All Traffic Out of a Region, Fast

Step 3.4: Evacuation Hits Replication Lag and Clock Trouble

Drill: The booking that existed in Frankfurt but not in Virginia · Loop: Distributed key-value store, steps 3.1 and 3.2 (multi-region replication and conflicts from the inside)

Step 3.5: Did the Evacuation Work?

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Two planes again, now fully separated. The control plane runs in three regions that can each serve anyone, with data replicated among them and traffic steered by controls that don't live in any one region. The data plane is Netflix's own network inside ISPs, told what to serve by the control plane. Netflix published the regions, the replication, Zuul's role and Open Connect; the ARC and Route 53 mapping, the probes and the dark-capacity labels on the diagram are our AWS rendering of the published ideas.

Sources for this round

• Netflix, Open Connect Overview (versions 2016 to 2026): OCAs at IXPs and embedded in ISPs, what OCAs report, the steering flow, fill windows.
• Netflix, Open Connect (checked September 2026): "over a thousand ISPs".
• Netflix, How Netflix Works With ISPs Around the Globe, March 2016: built from 2011, 100% of video, tens of Tbps, close to 90% direct, 8 to 90+ Gbps per server, the Australia example, 190 countries.
• Costello and Livengood, Netflix and Fill, Netflix TechBlog, August 2016: proactive caching, fill tiers, fill masters.
• Gallatin, Serving Netflix Video at 400Gb/s on FreeBSD, EuroBSDCon 2021.
• Cockcroft, A Closer Look at the Christmas Eve Outage, December 2012.
• Meshenberg, Gopalani and Kosewski, Active-Active for Multi-Regional Resiliency, December 2013: the three rules, geo-DNS and Denominator, Zuul's misrouting and shedding, the 500 ms test, Chaos Kong and split-brain.
• The EVCache team, Caching for a Global Netflix, March 2016: three regions, global replication design and latencies.
• Kosewski, Ramanujam, Behnam, Blohowiak and Probst, Project Nimble: Region Evacuation Reimagined, March 2018: the 50-minute breakdown, dark capacity, 8 minutes.
• Fisher-Ogden, Sanden and Rioux, SPS: the Pulse of Netflix Streaming, February 2015.
• Basiri, Hochstein, Thosar and Rosenthal, Chaos Engineering Upgraded, September 2015: Chaos Kong, the September 2015 event.
• Netflix, Q4 2025 shareholder letter, January 2026: more than 325M paid memberships.
• AWS, Enabling accelerated recovery for managing public DNS records: opt-in, 60-minute target.
• Netflix, What We Watched the First Half of 2026: more than 97B hours.

Netflix has not published its current traffic-steering tools, its present number of AWS regions, per-instance capacities, cache sizes or evacuation detection rules. Everything about those on this page is our design, and so are the probes, the ARC mapping, the scorecard and the consistency rules in step 3.4.

R3.6 Numbers and Cost

Figures marked published come from the sources above; everything else is our assumption or derived from one.

The data plane: how much video

A sanity check against 2016. Netflix then delivered more than 125M hours a day at "tens of terabits per second" of peak. The same model gives 125M ÷ 24 = 5.2M streams × 3 Mbps = 15.6 Tbps on average, × 1.5 = 23.4 Tbps at the peak: tens of terabits. The model is in the right range.

Why own the CDN: a made-up price. Suppose a CDN charged half a cent per GB, a round number we made up for illustration, not a quote. 21.7 billion GB × $0.005 = $108.5M a month, rising with every hour watched. Money is only part of Netflix's published reasoning: the other parts are working directly with ISPs, keeping bytes off backbones and undersea cables, and caching proactively instead of on demand.

Servers are not the limit; placement is. At the 400 Gb/s per server that Netflix engineers described in 2021, 100 Tbps is 100,000 ÷ 400 = 250 servers running flat out. Open Connect has thousands of appliances because each ISP site is sized for its own local peak and storage, and most sites are small. The design problem is putting the right files in the right thousand places, not raw throughput.

The control plane in each region. Assume 3M edge requests/s worldwide at the peak (Netflix published more than 1M in 2018 with 125M members; 325 ÷ 125 = 2.6 times the members, rounded up), 1M per region, and 5,000 requests/s per Zuul instance, as in Round 2.

The same 50% headroom that protects a region from losing one of three AZs also covers an even split of one of three regions, because in both cases a third of the capacity disappears and the other two-thirds must carry it. We accept not surviving both at once. The US pair evacuates to each other at full load, so each US region keeps about a third more capacity dark (102 ÷ 300 = 34%) in every tier on the streaming path, sized by time of day as Nimble did. Netflix reported Nimble as cost neutral without saying how, so we don't guess at the saving.

Quotas. 402 Zuul instances at 16 vCPUs each (assumption) is 6,432 vCPUs, and every other tier on the streaming path grows by the same third. The target region's EC2 On-Demand vCPU quota must cover the evacuation fleet for all of them, not today's peak. Dark instances already count against the quota; the check is headroom for autoscaling beyond them. We check it monthly and before every exercise (R3.9).

Caches during an evacuation. Round 2 had 10M cache operations/s for 500,000 edge requests/s: 20 per request. At 1M/s per region that's 20M operations/s, and the target of a paired evacuation must serve 40M/s. RAM doesn't need to grow, because globally replicated caches already hold every region's members; throughput does. This holds only for fully replicated caches. Caches that replicate invalidations only are empty for moved members, so we use invalidate-only replication only where Cassandra can absorb the moved members' miss rate. Caches on the streaming path replicate data.

A cold cache would be a stampede. The evacuated region's members make about 18M cache reads/s (90% of 20M). At a 99% hit ratio (assumption), Cassandra sees 18M × 1% = 180,000 reads/s. With cold caches, all 18M/s would go to Cassandra: 100 times its normal load, at the worst moment.

Replication between regions (list price $0.02/GB for data sent from us-east-1, us-west-2 or eu-west-1 to another region; the receiver pays nothing; other regions charge more).

Traffic control. One ARC cluster for the routing controls costs $2.50 per cluster-hour × 730 h ≈ $1,825 a month, whether or not we ever flip it.

At the peak, bookmarks run at 33.5M streams ÷ 60 s ≈ 558,000 writes/s, the figure step 3.4 used. Replication is cheap next to the fleet; what it costs in design effort is the no-cross-region-calls rule.

What a failure costs in availability. Netflix's internal goal is 99.99%, 4.32 minutes a month. A real region failure evacuated in 9 minutes hurts about a third of members for those 9 minutes: 9 × ⅓ = 3 member-weighted minutes, about 69% of the month's budget from one event. This is why cutting 50 minutes to 8 matters; Netflix's stated reason was that "even short or partial outages affect many of our customers."

R3.7 Trade-Offs

Build our own CDN vs buy

Active-active vs active-passive

Consistency, chosen per kind of data (ours)

Automatic vs approved evacuation. Fully automatic saves about a minute and risks evacuating a healthy region on a bad signal. We automate everything except the decision, and make the decision one action with the evidence in front of the on-call engineer. The capacity step starts during that minute anyway.

What changed from Round 1. Round 1 asked how to survive losing a database and answered with statelessness, replication and three AZs. Round 3 asks the same question one level up, about a whole region and about the internet itself, and answers with the same ideas at a bigger scale: nothing exists once (three regions, thousands of OCAs), state is replicated and its consistency chosen on purpose, and the control plane and data plane never share a fate. The split we made in step 1.2 to keep video off our API servers is the reason a region failure in Round 3 costs Play buttons, not video already playing.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6

Evacuation procedure REL 13 · OPS 10

1. Confirm it's a region. SPS for the region is below its band in the views of the other regions, and probes from both other regions fail. One region's probes alone, or a drop across every region, is not a regional failure.
2. Start capacity at once. In each target region, raise the production groups' maximum size, then move dark instances into production (commands 1 to 3). They reach UP in about a minute. This is safe to undo.
3. Check the gate. Targets' capacity UP; vCPU quota headroom (command 4) (dark instances already count against the quota; the check is headroom for autoscaling beyond them); the target map satisfies residency rules; two regions will remain on.
4. Approve and flip. Turn the failed region's routing control off through any ARC cluster endpoint; the five endpoints are in five regions, so try them in turn until one answers (commands 5 and 6).
5. Drain what DNS won't. If the failed region's Zuul answers, switch it to redirect mode so persistent connections close gradually and misrouted requests are sent to the targets.
6. Watch SPS, not the DNS change (command 7): the worldwide SPS should return to its band within about 5 minutes of the flip, and the failed region's should fall toward zero.
7. Return later, slowly. Once the region is healthy, replication has caught up and its caches are checked (R3.8), turn its routing control back on and move traffic back in steps, watching SPS all the way.

Go deeper: CLI playbook

Commands an on-call engineer runs one at a time. Replace names, IDs, ARNs and endpoints with real ones. attach-instances takes at most 20 instance IDs per call.

textCopy
# 1. Production and dark group sizes in the target region
aws autoscaling describe-auto-scaling-groups --region us-west-2 --auto-scaling-group-names zuul-prod zuul-dark --query "AutoScalingGroups[].[AutoScalingGroupName,DesiredCapacity,MaxSize,length(Instances)]"

# 2. Make room in the production group (attach fails if it would exceed the maximum)
aws autoscaling update-auto-scaling-group --region us-west-2 --auto-scaling-group-name zuul-prod --max-size 420

# 3. Move dark instances into production, in batches of up to 20
aws autoscaling detach-instances --region us-west-2 --auto-scaling-group-name zuul-dark --instance-ids i-0abc1234def567890 i-0abc1234def567891 --should-decrement-desired-capacity
aws autoscaling attach-instances --region us-west-2 --auto-scaling-group-name zuul-prod --instance-ids i-0abc1234def567890 i-0abc1234def567891

# 4. The Running On-Demand Standard instances vCPU quota in the target region
aws service-quotas get-service-quota --region us-west-2 --service-code ec2 --quota-code L-1216C47A

# 5. Read the routing control state through one of the cluster's five endpoints
aws route53-recovery-cluster get-routing-control-state --region us-west-2 --endpoint-url https://<cluster-endpoint-us-west-2>/v1 --routing-control-arn arn:aws:route53-recovery-control::123456789012:controlpanel/0123456789abcdef/routingcontrol/abcdef0123456789

# 6. Turn off the failed region (the safety rule refuses it if fewer than two regions would stay on)
aws route53-recovery-cluster update-routing-control-state --region us-west-2 --endpoint-url https://<cluster-endpoint-us-west-2>/v1 --routing-control-arn arn:aws:route53-recovery-control::123456789012:controlpanel/0123456789abcdef/routingcontrol/abcdef0123456789 --routing-control-state Off

# 7. Stream starts per minute served by each region (our own metric)
aws cloudwatch get-metric-statistics --region us-west-2 --namespace Streaming/Playback --metric-name StreamStarts --dimensions Name=ServingRegion,Value=us-west-2 --start-time 2026-09-28T20:00:00Z --end-time 2026-09-28T20:30:00Z --period 60 --statistics Sum

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Separates the control plane from the data plane, and designs each for its own failure and cost model.
• Explains why proactive, ISP-embedded caching beats on-demand caching for a predictable catalog, and what the ISP relationship adds.
• Chooses active-active over active-passive and states the rules that make it work: stateless services, local resources only, no cross-region calls, asynchronous replication.
• Breaks evacuation time into phases, finds the long poles (boot time, DNS), and removes them with dark capacity; counts every delay.
• Detects and moves traffic without depending on the failed region, and gates the move on capacity, quotas and residency.
• Chooses consistency per kind of data, knows last-writer-wins depends on clocks, and knows local conditional checks can't give one winner across regions.
• Proves it works with a member-facing metric and regular exercises.
• Keeps Netflix's published facts separate from its own design.

Follow-up questions

1. "Why didn't Netflix use latency-based DNS routing?" Answer: Netflix wrote that latency-based routing "could cause unpredictable traffic migration effects". With geographic routing, which users go where is decided by us, so capacity plans and evacuations are predictable.

2. "A new blockbuster launches worldwide at the same minute. What does Open Connect do differently from a normal CDN?" Answer: it doesn't wait for the first viewer. The files are copied to appliances during the preceding nights' fill windows, based on predicted popularity, peer to peer within each area, so at launch every nearby appliance already has them.

3. "What if the region you need to evacuate is the one that hosts your DNS provider's control plane?" Answer: that's exactly why the move uses ARC routing controls, whose data plane has endpoints in five regions, rather than editing records through the Route 53 API in us-east-1 (whose opt-in accelerated recovery targets 60 minutes), and why detection comes from probes in the other regions rather than health-check metrics stored in us-east-1.

4. "Is Hystrix still how Netflix does resilience?" Answer: no. Hystrix is in maintenance mode; Netflix pointed to adaptive concurrency limits and resilience4j, and has been moving features like adaptive limits, circuit breaking and hedging into an Envoy-based service mesh.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. For transcoding and adaptive bitrate, see the video streaming loop; for load shedding and limits, the rate limiter loop; for why retries must be budgeted and what a provider's 429 means, the notification system loop.

The Two Sentences That Matter Most

• Opening any round: "Netflix is two systems: a small control plane that decides what you can play and where to fetch it, and an enormous data plane that moves the video; I'll keep them apart so they never share a failure or a bill."
• When scale arrives: "I'll put one programmable front door in front of the services, contain every dependency with limits and fallbacks, run the control plane active-active in three regions with capacity waiting for an evacuation, and put the video inside the viewers' ISPs, filled before anyone asks."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels

Sources

All Netflix sources used on this page, oldest first.

• Netflix, Q4 2008 results press release, January 2009.
• Ciancutti, Four Reasons We Choose Amazon's Cloud as Our Computing Platform, December 2010.
• The Netflix Simian Army, July 2011.
• Schmaus, Making the Netflix API More Resilient, December 2011.
• Jacobson, Embracing the Differences: Inside the Netflix API Redesign, July 2012.
• Cockcroft, A Closer Look at the Christmas Eve Outage, December 2012.
• Meshenberg, Gopalani and Kosewski, Active-Active for Multi-Regional Resiliency, December 2013.
• Fisher-Ogden, Sanden and Rioux, SPS: the Pulse of Netflix Streaming, February 2015.
• Basiri, Hochstein, Thosar and Rosenthal, Chaos Engineering Upgraded, September 2015.
• Izrailevsky, Vlaovic and Meshenberg, Completing the Netflix Cloud Migration, February 2016.
• The EVCache team, Caching for a Global Netflix, March 2016.
• Netflix, How Netflix Works With ISPs Around the Globe, March 2016.
• Costello and Livengood, Netflix and Fill, August 2016.
• The Cloud Gateway team, Zuul 2: The Netflix Journey to Asynchronous, Non-Blocking Systems, September 2016.
• ChAP: Chaos Automation Platform, July 2017.
• Duvedi, Li, Garg and Fisher-Ogden, Scaling Time Series Data Storage, Part I, January 2018.
• Kosewski, Ramanujam, Behnam, Blohowiak and Probst, Project Nimble: Region Evacuation Reimagined, March 2018.
• Landau, Thurston and Bozarth, Performance Under Load, March 2018.
• Gonigberg et al., Open Sourcing Zuul 2, May 2018.
• Edge Authentication and Token-Agnostic Identity Propagation, February 2021.
• Gallatin, Serving Netflix Video at 400Gb/s on FreeBSD, EuroBSDCon 2021.
• Vroom, Mulcahy, Yuan and Gulewich, Zero Configuration Service Mesh with On-Demand Cluster Discovery, August 2023.
• Netflix, Q4 2025 shareholder letter, January 2026.
• Netflix, What We Watched the First Half of 2026, 2026.
• Open-source documentation: Netflix/zuul, Netflix/Hystrix, the EVCache wiki, the Eureka wiki, and Netflix's Open Connect Overview and Open Connect site, checked September 2026.