The Web Crawler That Forgot Its History
The Web Crawler That Forgot Its History
Your enterprise search crawler processes 2 billion discovered URLs per month, extracting content and indexing fresh web pages. To prevent crawling the same page repeatedly, worker nodes check whether a URL has already been visited before fetching it. Storing 2 billion 80-byte URLs in an exact database index (PostgreSQL/MongoDB) consumes 300 GB of storage and requires a 15ms database network roundtrip per URL, which bottlenecks the 25,000 URL/sec parser pipeline down to 4,000 URLs/sec. An engineer implements an in-memory Bloom filter with 10 bits per item and 7 hash functions to do instant sub-millisecond pre-filtering in RAM. Three months later, as the total discovered URLs pass 4.5 billion, crawl throughput mysteriously plummets by 40%: the crawler is falsely marking unvisited new URLs as "already visited" and dropping them from the crawl frontier. You are asked to diagnose why the Bloom filter degraded, redesign the membership verification tier with a scalable counting or layered structure, and handle URL deletions/updates.
The Web Crawler That Forgot Its History
Your enterprise search crawler processes 2 billion discovered URLs per month, extracting content and indexing fresh web pages. To prevent crawling the same page repeatedly, worker nodes check whether a URL has already been visited before fetching it. Storing 2 billion 80-byte URLs in an exact database index (PostgreSQL/MongoDB) consumes 300 GB of storage and requires a 15ms database network roundtrip per URL, which bottlenecks the 25,000 URL/sec parser pipeline down to 4,000 URLs/sec. An engineer implements an in-memory Bloom filter with 10 bits per item and 7 hash functions to do instant sub-millisecond pre-filtering in RAM. Three months later, as the total discovered URLs pass 4.5 billion, crawl throughput mysteriously plummets by 40%: the crawler is falsely marking unvisited new URLs as "already visited" and dropping them from the crawl frontier. You are asked to diagnose why the Bloom filter degraded, redesign the membership verification tier with a scalable counting or layered structure, and handle URL deletions/updates.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).