Design a Web Crawler at Scale

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Web Crawler?

You Already Know One: a Librarian Following References

A librarian reads a book. At the back is a list of references: other books. She writes each one on her reading list, takes the next book from the list, reads it, and adds its references. Over time the list grows faster than she can read.

A web crawler does the same with web pages. It fetches a page, pulls out every link on it, and adds those links to a list of pages to fetch next. That list is called the frontier: the set of URLs we know about but haven't fetched yet (or need to fetch again). Search engines (Googlebot, Bingbot), web archives (the Internet Archive) and open datasets (Common Crawl) all run crawlers.

What Makes It Hard

• The list never ends. Every page adds dozens of new links. There are far more URLs than we could ever fetch.
• Much of it is repeated. The same page appears under many URLs, and many pages are near-copies of each other (the same product with a different footer).
• We are guests. Every fetch uses someone else's server. A crawler that sends too many requests to a small site is, from the site's point of view, an attack.
• Some of the web is hostile or broken. Pages that generate infinite links, responses that decompress to gigabytes, servers that redirect us to our own internal network.

The Question the Whole Loop Answers

What do we fetch next, when are we allowed to, and is it worth fetching at all?

The answer gets sharper every round:

• Round 1: a durable queue, a fleet of workers, one request at a time per site, robots.txt rules, and pages stored in archive files.
• Round 2: the open web: a probabilistic "seen" filter, near-duplicate detection, a two-tier frontier that separates importance from politeness, traps, bombs, cheap machines that vanish, and recrawls.
• Round 3: a fixed budget spent where it buys the most freshness, fetchers near the hosts, politeness that listens to the server, and controls for the people who run the websites.

---

Round 1 · Mid-level · "Crawl 10,000 News and Shop Sites Daily"

~35 min · SDE II (L5) · 1 region · 10K sites, 1M pages/day · 23 pages/s in a 12-hour window · ≤ 1 request/s per site

R1.1 Establish Design Scope

The interviewer says: "We run a price-and-news aggregator. Every day we need a fresh copy of the pages on about 10,000 news and shopping sites. Design the crawler." Before drawing anything, we ask questions and say what each answer changes.

A robots.txt file is a plain-text file a website publishes at /robots.txt that tells crawlers which paths they may fetch. It is standardized as the Robots Exclusion Protocol, RFC 9309 (September 2022).

Out of scope for this round:

• The open web. Following links to sites not on our list.
• Near-duplicates. Pages that are almost, but not exactly, the same.
• Smart recrawl timing. Every page is simply fetched once a day.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into something the system does:

Not yet: links to other sites, near-duplicate detection, recrawl scheduling, JavaScript rendering.

R1.3 Non-Functional Requirements: the Questions

We state each quality in words first, in the order we'd defend it. The numbers come in R1.7.

• Politeness first. A crawler that knocks over a small shop's server has failed, however fast it is. No more than one request at a time and one request per second to any host, and robots.txt always obeyed.
• Then throughput: finish the day's crawl. A million pages inside a 12-hour window, with time left for retries.
• Then robustness to bad pages and bad sites. Slow servers, dead servers, huge responses and broken HTML must not stall the crawl or crash a worker.
• Durability of the work. A worker crash must not lose URLs from the frontier or pages we already fetched.

R1.4 The API

Everything in this round is internal. There are three contracts.

1. The crawl job message (one per URL, sitting in the frontier queue)

jsonCopy
{
  "crawl_date": "2026-09-27",
  "url": "https://shop.example.com/products/red-shoes?color=red&size=42",
  "host": "shop.example.com",
  "depth": 2,
  "discovered_from": "https://shop.example.com/products",
  "attempt": 0
}

depth is the number of links followed from the home page. It helps us notice a site that goes unexpectedly deep.

2. The crawl result event (published after the page is safely stored)

jsonCopy
{
  "crawl_date": "2026-09-27",
  "url": "https://shop.example.com/products/red-shoes?color=red&size=42",
  "http_status": 200,
  "content_type": "text/html; charset=utf-8",
  "fetched_at": "2026-09-27T03:14:07Z",
  "fetch_ms": 840,
  "body_bytes": 201344,
  "content_sha256": "3f9a1c...e07b",
  "links_found": 61,
  "links_new": 4,
  "warc": { "s3_uri": "s3://agg-crawl-archive/2026/09/27/worker-b-0007.warc.gz", "offset": 81234567, "length": 40211 }
}

The price and news extractors subscribe to these events. The warc pointer tells them exactly which bytes to read (step 1.5).

3. The admin API (ops manages the site list)

httpCopy
POST /admin/sites HTTP/1.1
Content-Type: application/json

{ "host": "shop.example.com", "seed_url": "https://shop.example.com/", "max_pages_per_day": 25000, "notes": "partner since 2024" }

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{ "host": "shop.example.com", "status": "ACTIVE" }

PATCH /admin/sites/shop.example.com with {"status": "PAUSED"} stops a site at once; GET /admin/sites/shop.example.com/runs/2026-09-27 shows the day's progress: pages fetched, errors, and whether the site finished.

Recap

• Three contracts: a job per URL, a result event per page, and an admin API for the site list.
• About 1M pages a day from 10K known sites; one region.
• Politeness first; finish within a 12-hour window; survive crashes and bad sites.

Let's build it, starting with the simplest version that works.

R1.5 Design Evolution: From One Script to a Polite Fleet

Every step follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

One script on one machine. It holds a list of URLs in memory, starting with the 10,000 home pages. It takes the first URL, downloads it, finds the links, appends the same-site ones to the list, saves the HTML to a local disk, and loops.

Synthesizing vector architecture diagram...

The whole crawler is one loop. It works for a few hundred pages.

What's good about it: it's an afternoon of work, and it shows the core loop every crawler has. What's wrong: one download at a time, taking about 1.5 seconds each, gets through 86,400 ÷ 1.5 ≈ 57,600 pages a day, not a million. And if the machine restarts, the list is gone.

Step 1.1: One Machine Is Too Slow, and a Crash Loses the List

Primitive: Message Queues vs Event Streams · Drill: Message queue order pipeline

Step 1.2: We Fetched the Same Page 50 Times

This table does more than dedup. Its row is where the URL's state lives for the day: QUEUED when inserted, FETCHED with the archive pointer once stored (R1.6). That makes it the source of truth for "did we get this page today?"

Step 1.3: Ten Workers Hit the Same Small Site at Once

The alternative is host-affine workers: each worker owns a fixed set of hosts, keeps a local queue per host, and enforces the delay with a timer. It gives each worker warm DNS and connection caches for its hosts, but a crash leaves its hosts orphaned until someone reassigns them. We compare the two in R1.8, and Round 3 comes back to host affinity at a much larger scale.

Primitive: Distributed Rate Limiting

Step 1.4: We Crawled Pages the Site Forbids

Step 1.5: Where Do a Million Pages a Day Go?

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Every morning the control service seeds each site's home page into the FIFO queue. Workers pull one URL per host at a time, check and record URLs in the pages table, store pages as WARC files in S3, and announce each stored page to the extractors.

Why these pieces:

• ECS Fargate runs the workers as containers with no servers to manage. At 23 pages a second, three small tasks are plenty (R1.7), one per AZ so that losing an AZ costs a third of our capacity, not all of it.
• Public subnets, no NAT gateway. Workers reach the internet directly through public IPs, with a security group that allows no inbound traffic at all. A NAT gateway would charge $0.045 for every GB it carries, and we pull about 6 TB a month through it (R1.7). S3 and DynamoDB are reached through free VPC gateway endpoints.
• SQS FIFO gives us the per-host lanes of step 1.3. Its basic (non-high-throughput) mode handles 300 API calls a second per action, or 3,000 messages a second with batching; we need about 23.
• DynamoDB holds the site list and the per-day URL table. Both are key lookups with conditional writes, which is what DynamoDB does well.
• S3 holds WARC files, with a lifecycle rule (R1.7).
• SNS fans out result events to the extractors, each through its own SQS queue.

The data model

sites: one row per site, partition key host

pages: one row per URL per day. Partition key site_day, sort key url_hash

TTL here is only housekeeping. DynamoDB deletes expired items eventually, typically within a few days, not at the second. That is fine because nothing depends on the delete happening on time: every key includes its date, so an old row can never be mistaken for today's.

Tracing one page

Synthesizing vector architecture diagram...

The host's lane stays closed from receive to delete, so the next shop.example.com request starts at 03:14:07.2 at the earliest. The row becomes FETCHED only once the WARC file is in S3.

R1.7 Numbers

Targets

Traffic

The 1.5-second fetch time (DNS, TCP and TLS setup, server think time and transfer) is an assumption for mixed small sites; we measure it on day one.

The biggest site sets the crawl length. One lane per host means a host's pages go one at a time, and each takes max(fetch time, 1 s). Suppose the largest shop has 20,000 pages (an assumption about our list): 20,000 × 1.5 s = 30,000 s ≈ 8.3 hours. That fits in 12 hours. A site with 40,000 pages would need 16.7 hours and could not finish; that's the follow-up in R1.11.

Page size. We fetch HTML only: no images, scripts or stylesheets. Common Crawl's August 2025 crawl is a good public reference: 2.44 billion pages were 424 TiB uncompressed and 88.24 TiB as compressed WARC files. That is 424 × 1,099.5 GB ÷ 2.44B ≈ 191 KB per page, and a compression ratio of 88.24 ÷ 424 ≈ 0.21. We plan with 200 KB per page and a ratio of 0.2, so 40 KB per page compressed. (A figure like 500 KB is closer to a whole page with its images and scripts, which a crawler that stores HTML doesn't download.)

Bytes

DynamoDB calls

Rough monthly cost (us-east-1 list prices; check the AWS Pricing Calculator before quoting)

Two lessons hide in this table. A NAT gateway would have added 3 × $0.045 × 730 ≈ $99 in hourly charges plus 6,000 GB × $0.045 = $270 in data processing: about $370, more than the workers and the queue together. And the seen-set reads are the biggest line, because links outnumber pages 60 to 1. Round 2 has about 33 times the pages (1B a month against ~30M) but on the open web, where links point anywhere and billions of URLs are already known, so checking links, not fetching pages, is what decides its design.

R1.8 Trade-Offs

One shared queue vs a lane per host

Stateless workers vs host-affine workers

At 10K hosts and 23 pages a second, the stateless design is simpler and plenty fast. Host affinity wins when per-host state gets expensive to share, which is Round 3.

Pages in S3 vs in a database

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Asks which sites, how deep, how often and what to keep, and says what each answer changes.
• Makes the frontier durable and the workers stateless, and explains what happens when a worker dies mid-fetch.
• Normalizes URLs before deduplicating, and makes "queue once" correct with a conditional write.
• Treats politeness as a per-host property, not a global rate.
• Knows what robots.txt allows and forbids, and that Crawl-delay isn't in the standard.
• Stores pages in archive files with offsets, and finds the NAT gateway line in the bill.

Follow-up questions

1. "A site on the list has 40,000 pages. What happens?" Answer: at 1.5 s a page, one lane takes 16.7 hours, which misses our 12-hour window. Options, in order: ask the site for a higher rate (many large sites allow a few requests a second, and robots.txt may already say so); crawl only the pages that change daily (their listing and category pages) and the rest on a rotation; or accept a two-day cycle for that site. We never solve it by opening a second lane to the same host.

2. "Why not a standard SQS queue plus a lock per host?" Answer: it can work, but every worker that receives a locked host's URL must put it back, and at scale most receives would be wasted on hosts someone else holds. The FIFO group does the "one at a time per host" check inside SQS, for free.

3. "Two workers discover the same new link at the same moment. Walk me through it." Answer: both batch-read the pages table and see no row. Both try a conditional put with attribute_not_exists. DynamoDB applies one; the other gets ConditionalCheckFailedException and drops the link. Only the winner sends it to the queue.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "A Billion Pages a Month, Politely"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 1B pages/month: 386/s average, 1,200/s planned peak · 5B URLs tracked · Spot fleet · zero politeness incidents

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We crawl 10,000 known news and shop sites every day: a million pages in a 12-hour window, about 23 pages a second. The frontier is an SQS FIFO queue where each host is its own message group, so only one worker at a time works on a host, and the worker deletes the message only one second after the fetch started, which spaces requests to every host at least a second apart. Workers are three stateless Fargate tasks in public subnets with no inbound access and no NAT gateway. URLs are normalized, then deduplicated through a per-day DynamoDB table with conditional puts, so each URL is queued once. robots.txt is parsed per host and cached for 24 hours, as RFC 9309 allows. Pages go into WARC files in S3, compressed per record, with offsets in the table, and a row becomes FETCHED only after its file is uploaded. About $620 a month. Open costs: the seen set costs a database call per link, every page is treated as equally important, and we only know how to crawl a list of friendly sites once a day."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: the FIFO queue enforces politeness, the pages table enforces "queue once" and records what's durable, and S3 holds the archive.

Round 1 step summary

Open costs: a database call per discovered link; no idea which pages matter; nothing that handles an infinite site, a hostile server, or a page that changed.

R2.1 The Scope Raise

Interviewer: "The aggregator worked. Now we're building a search engine. Start from a seed list and follow links anywhere on the web. We want a billion pages a month. You'll discover billions of URLs, and a lot of the web is duplicated. Some sites are effectively infinite, and some servers are hostile. Pages change, so we need to refetch them, but we can't afford to refetch everything. Compute must be cheap: use Spot. And a few percent of the sites we care about show nothing without JavaScript."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

R2.2 What Breaks in the Round 1 Design

We fix them in this order: the seen filter (2.1), near-duplicates (2.2), priority (2.3), traps (2.4), Spot (2.5), recrawls (2.6) and bombs (2.7).

R2.3 New Requirements and API Additions

The job message gains fields (sent to the per-host FIFO queue):

jsonCopy
{
  "url": "https://news.example.org/world/2026/09/27/storm-update",
  "url_hash": "9c1e0f4b7a2d3e58",
  "host": "news.example.org",
  "registered_domain": "example.org",
  "priority_class": 1,
  "depth": 3,
  "is_recrawl": true,
  "due_version": 1790503200,
  "conditional": { "etag": "\"a7f3-5e2b\"", "last_modified": "Sat, 26 Sep 2026 22:10:04 GMT" }
}

• priority_class is 1 (high), 2 or 3 (low), from step 2.3.
• is_recrawl says we've fetched it before, so the worker sends a conditional request (step 2.6).
• due_version is the scheduling time the message was created for. It lets a worker spot a stale copy of a message (step 2.3).
• registered_domain is the part of the host a person can register (example.org for news.example.org), found with the Public Suffix List, the public list of suffixes like .co.uk under which anyone can register names. It's the key for the domain-level politeness cap (step 2.3).

The result event gains a content fingerprint and a verdict, and moves to a Kinesis data stream the indexer reads:

jsonCopy
{
  "url": "https://news.example.org/world/2026/09/27/storm-update",
  "http_status": 200,
  "fetched_at": "2026-09-27T08:00:04Z",
  "changed": true,
  "simhash64": "e3a1c07f5b2d9410",
  "near_duplicate_of": null,
  "links_found": 84,
  "links_new": 9,
  "rendered": false,
  "truncated": null,
  "warc": { "s3_uri": "s3://crawl-archive/2026/09/27/use1-b-w07-000213.warc.gz", "offset": 41943040, "length": 38112 }
}

A 304 recrawl produces "http_status": 304, "changed": false and no warc. A near-duplicate produces near_duplicate_of with the URL hash of the page it copies, and no warc.

A per-host policy record in a DynamoDB hosts table holds everything about how we treat one host:

R2.4 Design Evolution: The Open Web Fights Back

Step 2.1: Checking 5 Billion URLs in a Database Is Too Slow and Costly

Primitive: Bloom Filters & Counting Filters · Drill: Bloom filter crawler dedup

Step 2.2: Thousands of Pages Are the Same Page With a Different Footer

A tiny worked example with 8-bit fingerprints. Page A has four tokens:

Counter for bit position 1 (leftmost): cheap has 1 (+2), red has 0 (−1), shoes has 1 (+3), sale has 0 (−1): total +3. Doing all eight positions:

Page B is the same page with a footer token 2026 (weight 1, hash 0 1 0 1 0 1 0 1) added. Each counter moves by exactly 1:

Only position 3 flipped (its counter was +1 and dropped to 0, which counts as "not above 0"). The Hamming distance, the number of differing bits, is 1: a near-duplicate. Two unrelated pages differ in about half their bits, about 32 of 64.

Finding near matches: bit-block tables. Split each 64-bit fingerprint into 4 blocks of 16 bits. If two fingerprints differ in at most 3 bits, those 3 bits fall in at most 3 blocks, so at least one block is identical (the pigeonhole principle). So we keep 4 tables; table i maps "the value of block i" to the fingerprints that have it. To check a new fingerprint:

1. Look up each of its 4 blocks in its table.
2. For every candidate returned, compute the full distance: XOR the two fingerprints and count the 1 bits.
3. Any candidate at distance ≤ 3 is a near-duplicate.

A table keyed by 16 bits spreads fingerprints over 65,536 buckets, so a host with 50,000 pages gives about 50,000 ÷ 65,536 ≈ 0.8 candidates per table, 3 in all. Our choice: we check near-duplicates within the same host, where most copies come from (session parameters, sort orders, printer versions). Cross-site copies (syndicated articles) are found by a daily batch job over all fingerprints and marked for the indexer.

Go deeper: for a global index, 16-bit keys give too many candidates (1B fingerprints ÷ 65,536 ≈ 15,000 per bucket). The Manku paper uses more tables with longer keys: split 64 bits into 6 blocks and build one table for each choice of 3 blocks, C(6,3) = 20 tables. Three differing bits spoil at most 3 blocks, so some table's 3 key blocks all match. Each key is about 32 bits, so a bucket holds about 10^9 ÷ 2^32 ≈ 0.2 fingerprints. The price is 20 copies of the index.

Step 2.3: We Spend Our Budget on Junk While Important Pages Wait

Synthesizing vector architecture diagram...

The front tier picks which URL is worth fetching next; the back tier holds each host to one request at a time. The router fills the back tier only as fast as workers drain it, so priority applies whenever we're short of capacity.

Primitive: Distributed Locks & Leases (the domain slots are short leases that expire on their own)

Step 2.4: An Infinite Calendar Ate a Million Fetches

Step 2.5: Spot Instances Vanish Mid-Crawl

Synthesizing vector architecture diagram...

Everything the worker must save is saved in under a minute, well inside the two-minute notice. If the notice never comes, leases and visibility timeouts recover the same work, just later.

Step 2.6: Pages Change, but the Filter Says "Seen"

Synthesizing vector architecture diagram...

A URL cycles between QUEUED and FETCHED for as long as it lives. The 6-hour lease brings back any URL whose message was lost; nothing depends on the queue remembering it.

Step 2.7: A Response Decompressed to 10 GB and Killed a Worker

Round 2 Step Summary

R2.5 Architecture v2

The design splits into two paths: the frontier path decides what to fetch, and the fetch path fetches it.

Synthesizing vector architecture diagram...

The frontier path: links become rows in the urls table only once, and rows become messages only when they're due. The queues hold minutes of work; the table holds the frontier.

Synthesizing vector architecture diagram...

The fetch path: a worker checks the host's rules and takes a domain slot, resolves through a local cache, checks the IP before connecting, fetches with limits, parses in a separate process, and only then records the page as fetched.

What's new, and why:

• Spot workers on EC2. At 1,200 pages a second, a worker instance runs up to 150 fetches at once (R2.6). Each worker runs Unbound, a caching DNS resolver, on the instance itself: a host fetched once a second resolves from memory for the life of its DNS record, and the VPC resolver's hard limit of 1,024 packets a second per network interface stays far away.
• SSRF guard in the fetch path. SSRF (server-side request forgery) is getting our own server to make a request to somewhere it shouldn't, like our internal network. Every resolved address is checked before connecting (R2.8).
• High-throughput FIFO. Basic FIFO allows 300 calls a second per action (3,000 messages with batching); we need about 1,200 sends, receives and deletes a second at peak. High-throughput mode allows up to 70,000 calls a second per action in the largest regions. Workers keep Round 1's rule for batched receives: keep one message per host and return the other same-host messages at once with the host's delay, so no message's visibility timeout can run out while it waits behind another.
• Headless pool. A headless browser is a real browser engine with no screen, which runs a page's JavaScript and gives us the resulting HTML. Only hosts flagged needs_render go there (R2.7).
• SimHash service. A few memory-optimized instances holding the block tables, sharded by host. Its state is rebuildable: each fingerprint is also stored in its URL's row.
• ElastiCache for Valkey with noeviction: the Bloom filters must never be evicted, and the slot keys expire on their own. Nothing in this cluster is a cache that can be refilled on a miss, so eviction would only lose data.

Trace 1: a new URL, end to end

Synthesizing vector architecture diagram...

The URL is stored once (the conditional put), queued when due (the scheduler's lease), fetched under both politeness levels, and recorded only after its bytes are in S3.

Trace 2: a recrawl that returns 304

Synthesizing vector architecture diagram...

An unchanged page costs about 200 bytes and one row update instead of 200 KB and a WARC record, and its next visit moves further out.

Trace 3: a calendar trap being cut off

1. events.example.com/calendar?month=2026-10 is fetched; its "next" link leads to ?month=2026-11, and so on.
2. From about 2027 onward, the months have no events, and every page renders the same empty grid. Pages 5 to 9 under /calendar are each within 3 bits of the one before.
3. After the 5th near-duplicate in a row under the prefix /calendar, the parser adds /calendar to the host's trap_rules.
4. Already-queued /calendar URLs are dropped when received (BLOCKED); new links under it are dropped at normalization.
5. The host's growth alarm had not fired yet; the near-duplicate stop caught it after a few dozen fetches, not a million. If it hadn't, the 50,000-page budget would have.

R2.6 Numbers and Cost

Targets

Rates

Page size, verified and recomputed. The fact source for this track used 500 KB a page and a 0.35 compression ratio. We fetch HTML only, and Common Crawl's August 2025 numbers (R1.7) give about 191 KB per page and a 0.21 ratio. We use 200 KB and 0.2:

These are upper bounds: every 304 and every near-duplicate stores nothing, and servers that compress their responses send fewer bytes on the wire.

The Bloom filter (from step 2.1): m = 55.14 × 10^9 bits = 6.9 GB (6.42 GiB), k = 8, as 64 filters of 108 MB. During a monthly rebuild two sets exist: 12.84 GiB. On 3 shards of cache.r7g.large (13.07 GiB each), that's 4.3 GiB per shard at worst. Commands: a page's 80 links fall into about 50 of the 64 filters, so about 60,000 pipelined Bloom commands a second at peak, 20,000 per shard (an estimate we'd load-test).

Why the filter pays for itself: without it, every link is a DynamoDB conditional put: 80 × 10^9 × $0.625 per million writes = $50,000 a month. With it, only the ~5B new URLs reach DynamoDB.

Fetch fleet

SimHash service: fingerprints of the last 30 days, about 1B pages. Each fingerprint appears in 4 tables at 12 bytes (8-byte fingerprint + 4-byte page reference): 10^9 × 4 × 12 B = 48 GB. We use 4 shards by host, each with a replica, on r7g.xlarge (32 GiB) instances.

Rough monthly cost (us-east-1 list prices; Spot prices change constantly, so check the Spot price history before quoting)

Three lessons:

• The frontier table is the biggest line, bigger than all the compute. Every URL we know costs writes, not just every URL we fetch. Round 3 attacks this.
• Spot makes compute cheap. The fetch fleet on-demand would be 16 × $0.34 × 730 ≈ $3,970; Spot saves about $2,100 a month here, roughly half off at current prices (our assumed $0.16/h; Spot prices move).
• Storage grows forever. Deep Archive adds about 480 TB a year, ~$475 a month more each year, and retrieving it takes hours (up to 12 hours standard, 48 hours bulk), which is fine for archives we rarely reread.

R2.7 Trade-Offs

How to build the per-host back queues

The in-flight limit is worth a sentence in an interview: with one message in flight per active host, 120,000 in flight caps how many hosts we can crawl at the same moment. We use about 1,800. At 20 times the rate (Round 3), fetches alone would hold about 23,000/s × 1.5 s ≈ 34,500 messages, 29% of the limit; the pressure comes from messages parked in flight during backoffs and domain-slot waits (R2.8), which can grow without bound when many hosts slow down at once. That, the 14-day retention and the per-call bill are why Round 3 builds its own frontier.

Bloom filter vs an exact set

The Bloom filter wins because a false positive costs little (a page we skip, with the table as the backstop) and the exact set costs 6× or more memory. If losing any new URL were unacceptable, we'd keep the exact set on disk-backed storage and accept slower checks.

Headless rendering: when

We render only hosts on the list, and only when a plain fetch returns an empty shell. The renderer blocks images, video and fonts, caches script files between pages, and every request it makes goes through the same host lanes and domain slots.

R2.8 Failure Modes

R2.9 Production Gotchas

1. A visited set only in a database

• Symptom: the database bill and latency climb with the number of links, not pages.
• Cause: a lookup per discovered link, 80 per page.
• Fix: a Bloom filter in front (step 2.1); the database stays the source of truth.

2. Rendering JavaScript everywhere

• Symptom: the fleet costs many times more, and sites complain about request volume.
• Cause: a headless browser for every page, fetching every script, style and API call.
• Fix: plain fetches by default; render only listed hosts, with the same politeness applied to every sub-request.

3. No robots.txt caching

• Symptom: twice the request volume to every host; some block us.
• Cause: fetching /robots.txt before each page.
• Fix: parse once, cache for up to 24 hours (RFC 9309), and treat a 5xx robots.txt as "disallow everything for now".

4. No content-size limits

• Symptom: workers die from out-of-memory errors at random.
• Cause: reading and decompressing whole responses into memory.
• Fix: streaming limits on wire bytes, decompressed bytes, ratio, time and speed (step 2.7).

5. Letting the crawler reach internal addresses

• Symptom: a security review finds fetched pages containing instance credentials or internal dashboards.
• Cause: fetching whatever a link resolves to.
• Fix: check every resolved IP, pin it for the connection, re-check redirects, require IMDSv2 (R2.8).

6. Clearing the seen filter to force recrawls

• Symptom: the queues flood with billions of known URLs after each reset.
• Cause: using one structure for "is it new?" and "is it due?".
• Fix: the filter answers only "new?"; the scheduler decides "due?" from the table (step 2.6).

R2.10 Pillar Check

Security in detail. Workers have one IAM role: send and receive on the crawler queues, read and write the urls and hosts tables, put objects under one S3 prefix, put records to one stream. They can't read other prefixes or change any queue or table settings. The parser process runs as a separate, unprivileged user with memory and CPU limits and no network access; it gets bytes from the fetcher and returns links and a fingerprint. Worker instances accept no inbound traffic, and are replaced (not patched in place) from a new image every week.

Operations in detail.

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer adds over L5

• Sizes the Bloom filter with the formula, explains what a false positive costs, and knows how it degrades past capacity.
• Explains SimHash well enough to compute a tiny one, and how to find near matches without comparing against everything.
• Separates priority from politeness (the Mercator two tiers), and knows the frontier can't live only in a queue with a 14-day retention.
• Defends against traps, bombs and SSRF with specific limits.
• Designs for Spot so that correctness never depends on the interruption notice.
• Recrawls from the table, with adaptive intervals and conditional requests.
• Checks the fact source's numbers (the 500 KB page) instead of copying them.

Follow-up questions

1. "Your Bloom filter was sized for 5B. The crawl grows to 15B URLs in the window. What happens?" Answer: at 15B in a filter sized for 5B, (1 − e^(−8 × 15 ÷ 55.14))^8 = (1 − 0.114)^8 ≈ 38% of new URLs would be skipped, and nothing would error: fewer new pages would just quietly appear. The 80% fill alarm fires long before. We rebuild at the new size, m scales linearly with n, so 15B at 0.5% is 20.7 GB, 192 filters of 108 MB. Valkey's scaling filters add layers as they fill, but every layer counts toward the 128 MB per-object limit, so with ~108 MB filters we'd rather split into more non-scaling filters than rely on scaling.

2. "Why not check near-duplicates before fetching, to save the download?" Answer: SimHash needs the content, so it can't. What we can do before fetching is avoid URLs that lead to duplicates: stripped session parameters, sorted query strings, and trap rules learned from earlier near-duplicates. And rel="canonical" in a fetched page tells us which URL the site considers the original, so we can lower the priority of the others.

3. "A worker received a message, fetched the page, then Spot took it before the upload. Walk me through recovery." Answer: the page's row still has the lease the scheduler set, due_at 6 hours ahead, and was never marked FETCHED. The message itself was deleted after the politeness delay, so SQS won't return it. In at most 6 hours the row comes due again, the scheduler re-queues it, and it's fetched once more. Nothing is lost; the cost is a delay of up to 6 hours and one extra fetch.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Web Scale, Fresh, and a Good Citizen"

~45 min · Principal (L7) · 3 regions · 20B fetches/month: 7,716/s average, ~23,000/s peak · ~400M hosts, 200B known URLs · news fresh within 15 min · under $5 per million pages

R3.0 Where We Left Off

What the candidate says in the first 60 seconds of Round 3, and everything you need if you start here.

Round 2 in 60 seconds. "We crawl the open web from seeds: a billion pages a month, 386 a second on average, planned for 1,200, in one region on a Spot fleet. New links are checked against 64 Bloom filters in Valkey, 6.9 GB in all, sized for 5 billion URLs at 0.5% false positives, and only new ones get a conditional put into the urls DynamoDB table, which is the real frontier. A scheduler reads due URLs from an index on that table, leases them for 6 hours, and feeds three priority queues; a router picks from them 60/30/10 and moves URLs into SQS FIFO lanes, one per host, while 5 slots per registered domain in Valkey cap subdomain farms. Workers resolve DNS through a local cache, block private addresses, stream responses through hard limits, fingerprint pages with 64-bit SimHash and treat 3 bits or fewer as duplicates. Recrawls come from the table: intervals halve on change and double otherwise, between an hour and 30 days, with conditional requests that often return 304. Spot interruptions lose nothing: we checkpoint on the notice, and leases catch what's left. About $13.6 per million pages; the frontier table is the biggest line. Open costs: one region far from most hosts, a crude priority, a frontier that costs writes per URL known, a fixed politeness rate, and nothing for webmasters."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: the table decides what's due, the queues decide order and politeness, and the workers fetch safely and record results.

Rounds 1–2 step summary

Open costs: one region; a priority by host type; a frontier that pays per known URL and a queue whose in-flight limit caps active hosts; politeness that doesn't listen to the server; no identity or controls for webmasters.

R3.1 The Scope Raise

Interviewer: "We're now a major search and research index. We want 20 billion fetches a month across hundreds of millions of hosts. Breaking news must be in the index within minutes, while old archives barely change. Most hosts are far from our one region. Webmasters are complaining: some say we're too aggressive, some want to set our rate, and some want to stay in search but opt out of their content being used for AI training. The indexing team wants a clean stream of what's new. And the budget is fixed."

We ask back, and say what each answer changes.

Scope change

R3.2 What Breaks in the Round 2 Design

We fix them in this order: geography (3.1), the frontier (3.2), what to fetch (3.3), how fast (3.4), webmasters (3.5), and the handoff and budget (3.6).

R3.3 New Requirements and API Additions

Webmaster verification and rate control. A site owner proves they control a host, then sets our rate.

httpCopy
POST /webmaster/v1/sites HTTP/1.1
Authorization: Bearer <webmaster account token>
Content-Type: application/json

{ "host": "shop.example.com" }

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{ "host": "shop.example.com", "verification": { "method": "dns_txt", "record": "example-search-verify=3f9a1c07e5b2" }, "status": "PENDING" }

The owner adds that TXT record to their DNS and calls POST /webmaster/v1/sites/shop.example.com/verify. We look up the record; if it matches, the site is verified for that account.

httpCopy
PUT /webmaster/v1/sites/shop.example.com/crawl-rate HTTP/1.1
Content-Type: application/json

{ "max_requests_per_second": 2, "until": "2026-10-31T00:00:00Z" }

The rate is a ceiling our adaptive controller never exceeds (step 3.4). Raising it above our default needs verification; lowering it only needs verification too, but a site can always lower us without an account through robots.txt or by returning 429.

The opt-out registry holds one record per subject and purpose:

jsonCopy
{
  "subject": "example.org",
  "scope": "registered_domain",
  "purposes": { "crawl": "allow", "search_index": "allow", "ai_training": "deny" },
  "source": "webmaster_console",
  "verified": true,
  "effective_at": "2026-09-27T10:00:00Z",
  "registry_version": 48211
}

registry_version increases by one with every change, so every component can report which version it has applied.

The index-handoff event is a versioned contract:

jsonCopy
{
  "schema": "crawl.handoff.v3",
  "event_id": "b41c9e0a7d2f5e63",
  "url": "https://news.example.org/world/2026/09/27/storm-update",
  "canonical_url": "https://news.example.org/world/2026/09/27/storm-update",
  "fetched_at": "2026-09-27T08:00:04Z",
  "change": "CHANGED",
  "simhash64": "e3a1c07f5b2d9410",
  "language": "en",
  "importance": 0.73,
  "usage": { "search_index": true, "ai_training": false },
  "usage_registry_version": 48211,
  "warc": { "s3_uri": "s3://crawl-archive-use1/2026/09/27/use1-f031-000877.warc.gz", "offset": 41943040, "length": 38112 },
  "text": { "s3_uri": "s3://crawl-text-use1/2026/09/27/use1-f031-000877.txt.gz", "offset": 1048576, "length": 9214 }
}

event_id is a hash of the URL and fetched_at, so a consumer that sees the same event twice can drop the copy. change is NEW or CHANGED only; unchanged pages and duplicates never appear.

R3.4 Design Evolution: Spending a Fixed Budget Well

Step 3.1: Hosts Are Everywhere; Our Fetchers Are in One Region

One more wrinkle: hosts are not the same size. Politeness caps each host's fetch load (a host can't take more than its rate), but not its frontier size: one encyclopedia host may know 100M URLs, a small blog 50. With plain hashing, a node that draws a few giant hosts holds far more data than its neighbors. We use bounded-load consistent hashing: a host goes to its ring owner unless that node already holds more than 1.25× the average, in which case it goes to the next node clockwise. The same bound handles traffic skew, not just data size: we measure a node's load as leases handed out per second plus stored URLs, so a node that drew several hot hosts (big news sites at their full adaptive rate) also sheds new hosts to its neighbor. The cost is that placement now depends on current loads, so it needs the controller's view of every node's load, and a load change can move hosts.

Primitive: Consistent Hashing · Drill: Consistent hashing vnode skew

Step 3.2: The Frontier Has 200 Billion Entries

Synthesizing vector architecture diagram...

The frontier costs disk, not writes: all URLs sit sorted on local disk, and only the next few URLs per host and a heap of "who's next" live in memory. The change log and snapshots rebuild a lost node.

Primitive: Database Sharding & Partition Keys · Write-Ahead Log & LSM Trees

Step 3.3: We Have Budget for 20B Fetches: Which Ones?

Step 3.4: A Site's Server Is Struggling Because of Us

Primitive: Distributed Rate Limiting · Circuit Breaker, Bulkhead & Fault Tolerance (pausing a failing host is a circuit breaker per host)

Step 3.5: Webmasters Want Control, and to Opt Out of Some Uses

We keep legal claims general here: which purposes need consent and how quickly opt-outs must apply depend on jurisdiction, and legal decides the policy. The design's job is to make any policy they choose enforceable and auditable.

Step 3.6: The Indexer Needs Clean Input, and the Budget Is Fixed

Primitive: Change Data Capture & the Outbox Pattern

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each region fetches the hosts assigned to it, from frontier nodes that own those hosts' politeness. The control plane writes host placement, opt-outs and blocks from one region, and every frontier node reads replicated copies. Handoff events from all regions reach one indexer.

Inside one region

Synthesizing vector architecture diagram...

The fetcher is a pair of hands; the frontier node is the brain for its hosts. Every check that protects a site happens before the lease, and every signal the site sends back changes the next lease.

Trace 1: a news article recrawled within minutes

Synthesizing vector architecture diagram...

The feed, not the crawl order, finds the article. From discovery to handoff takes seconds; the 10-minute feed poll sets the worst case.

Trace 2: a struggling host slows us down

Synthesizing vector architecture diagram...

We cut our load by 75% within two windows and pause entirely on a 503, then climb back slowly. The owner never had to contact us.

Trace 3: an opt-out applied

1. 10:00: the owner of example.org, verified, sets ai_training: deny in the console. The registry, written in the control region, goes to version 48,211.
2. 10:00–10:01: the global table replicates to all regions (typically within seconds); every frontier node picks up version 48,211 on its next 60-second refresh and reports it.
3. From then on, every handoff event for example.org carries "ai_training": false and usage_registry_version: 48211. Search indexing is unchanged.
4. The usage-restrictions table now lists example.org for AI training. The next training-data selection job, reading pages fetched last month, checks it at read time and excludes them.
5. An alarm would fire if any frontier node were still on an older version 5 minutes later.

R3.6 Numbers and Cost

What 20B fetches a month are (a plan built from the class budgets in step 3.3, with assumed outcomes)

Fetches and bandwidth by region (bandwidth from full bodies: 14B × 200 KB ÷ 2,592,000 s = 1.08 GB/s = 8.64 Gbps in total)

Each instance now handles 200 pages a second, up from Round 2's cautious 100: load tests showed 300 concurrent fetches fit comfortably, 300 ÷ 1.5 s = 200/s, using about half the CPU at our assumed ~20 ms per page. Fetches also got faster where hosts are close, but we keep 1.5 s for sizing. With host-affine leases, DNS lookups mostly hit the fetcher's cache; even with none, 200 × 2 = 400 packets a second per instance stays under the 1,024 limit.

The frontier on disk

Storage per tier (11.9B pages × 40 KB = 476 TB a month, 5.71 PB a year)

Each further year adds 5.71 PB to Deep Archive, about $5,650 a month more. Objects are ~1 GB WARC files, so per-object charges (transition requests, Deep Archive's ~40 KB of overhead per object) are negligible.

Rough monthly cost (list prices; Spot and commitment discounts are assumptions to check)

Synthesizing vector architecture diagram...

Storage is a third of the bill and grows every year; the frontier, which was Round 2's biggest line, is now a fixed fleet.

Round 2 cost $13.6 per million; Round 3 costs $3.25. The frontier moved from per-row writes to disks we own, the fleet scales with load instead of sitting at peak, each fetcher does twice the work, and the cheaper Glacier Instant Retrieval tier replaced Standard-IA for months 2–4.

R3.7 Trade-Offs

Freshness vs coverage

The value formula makes this a single dial: a new URL's value is its importance, a known URL's value is importance times the chance it changed. They compete in the same ranking, so the split falls out of the data.

Adaptive vs fixed politeness

Build a crawler vs buy data (COST 11)

Closing the loop. Round 1's question was "how do we fetch politely?" and the answer was a queue with one lane per host. Round 2 asked "what's worth fetching?" and split priority from politeness. Round 3 asks "what's worth fetching with this budget, and are we welcome?", and the answers are the value model, adaptive politeness and webmaster controls. The one-lane-per-host idea from Round 1 never went away; it moved from SQS into the heap of a frontier node.

R3.8 Failure Modes

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active (our regions are active-active for fetching, but each host is active in exactly one)

R3.9 Runbook

Golden signals, per region OPS 8 · REL 6

Procedure: emergency block of one host (a webmaster reports overload, or a violation alarm fires)

1. Add the host to the crawler-host-blocks table in the control region, with an expiry and a reason (command below). Every frontier node stops leasing it within about 60 seconds; leases already out finish within their 60-second deadline.
2. Confirm in the fetch logs that requests to the host stop within 2 minutes.
3. Reply to the webmaster with what we did and when we'll resume.
4. Find the cause: AIMD state, the host's recent ceilings, recent deploys. Fix it before the block expires, or extend it.

Commands (AWS CLI; check names and regions before running)

textCopy
aws dynamodb put-item --region us-east-1 --table-name crawler-host-blocks --item '{"host":{"S":"shop.example.com"},"blocked_until":{"N":"1790640000"},"reason":{"S":"webmaster overload report 2026-09-27"}}'

aws dynamodb get-item --region eu-west-1 --table-name crawler-host-blocks --key '{"host":{"S":"shop.example.com"}}'

aws autoscaling set-desired-capacity --region eu-west-1 --auto-scaling-group-name crawler-fetch-euw1 --desired-capacity 0

aws ec2 describe-spot-price-history --region us-east-1 --instance-types c6i.2xlarge c6a.2xlarge c7i.2xlarge --product-descriptions "Linux/UNIX" --start-time 2026-09-26T00:00:00Z

aws s3api head-object --bucket crawl-archive-use1 --key 2026/09/27/use1-f031-000877.warc.gz --checksum-mode ENABLED

The second command reads the block from another region's replica, to confirm it has replicated. The third is the regional kill switch: it stops all fetching in one region. The last shows the object's stored checksum when the file was uploaded with one.

Incident flow OPS 10

Synthesizing vector architecture diagram...

The first question is always the blast radius. Every branch has a prepared first action that makes things safer before we understand the cause.

Game days. Monthly: kill a frontier node and time its recovery; block a test host and time the stop. Quarterly: fence a whole region in a staging copy and move its hosts; restore a sample of Deep Archive objects end to end. REL 12 · OPS 11

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Turns "freshness" into a budget problem with a value per fetch, and knows why "recrawl in proportion to change rate" is a trap.
• Moves the frontier out of managed queues when the numbers say so, and keeps it recoverable with snapshots and an idempotent log.
• Guarantees one owner per host across regions, including during a region failure, and knows why global tables need a single writer here.
• Makes politeness a feedback loop with ceilings and kill switches.
• Treats webmasters as customers: verifiable identity, controls, per-purpose opt-outs enforced at use as well as fetch.
• Knows the unit cost, what drives it, and when buying data beats crawling.

Follow-up questions

1. "A site's owner says we sent 50 requests a second yesterday. Our logs say 2. What now?" Answer: first, check whether the traffic was ours: their logs' source IPs against our published list, and reverse DNS. Crawler user agents are often faked. If it's not our IPs, we tell them how to verify and block the impostor. If it is ours, we block the host at once and treat it as a P1 politeness violation: find which frontier node owned it, whether two regions had it during a move (epochs and fencing should prevent this), and whether the headless pool's sub-requests were counted.

2. "Why not let every region fetch every host, and coordinate politeness through a global store?" Answer: every fetch would need a cross-region round trip to the store, 100 ms or more, and during a network split the regions would have to choose between stopping and risking double traffic. One owner per host keeps politeness a local decision; the only cross-region work is moving ownership, which is rare and fenced.

3. "The budget is cut by half. What do you drop?" Answer: the value model decides: we raise the cut-off, and the lowest-value fetches go first, which are mostly tail recrawls and low-importance new URLs. News feeds and the head keep their budgets. We'd also look at the biggest line: storage. Archiving only the head and news (and keeping fingerprints and text for the rest) would cut storage growth far more than any compute change.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design, let me ask a few scoping questions."
• When the scope is raised: "Here's what breaks in the current design, and here's the order I'll fix it in."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels