Design an Ad Click Event Aggregation System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is Click Aggregation?

You Already Know One: a Turnstile Counter

A subway turnstile clicks once for every person who walks through, and at the end of the day the operator reads the counter and knows how many fares to collect. Now picture a turnstile on every ad on the internet. Every time someone clicks an ad, a counter ticks, and the advertiser pays for each tick. The same counter tells the advertiser, minute by minute, how the campaign is doing.

That's the whole system: count clicks per ad, per time period, and turn the counts into a bill and a dashboard.

A few words we'll use all the way through:

What Makes It Hard

• Volume. By Round 2 it's a billion clicks a day, with spikes during big live events.
• Clicks arrive late, and twice. A phone clicks in a subway tunnel and uploads 40 minutes later. A user double-clicks. A network retry sends the same click again.
• Some clicks are fake. Bots and click farms click ads to drain a competitor's budget or to inflate a publisher's earnings.
• Every count is money. Counting one click twice overcharges a customer. Losing one underpays us. Both end up in a dispute.

The Question the Whole Loop Answers

How do we count a firehose of clicks fast enough to act on, and exactly enough to bill?

The answer gets sharper every round:

• Round 1: count through a queue instead of the database, once an hour, exactly.
• Round 2: count in a stream within seconds, and keep a slower, exact recount for billing.
• Round 3: feed the fast count back into ad serving so budgets stop on time, across regions, under privacy rules.

---

Round 1 · Mid-level · "Click Reports for a Small Ad Network"

~35 min · SDE II (L5) · 1 region, 3 AZs · 10M clicks/day, 500/s peak · 5,000 active ads · redirect 99.9%

R1.1 Establish Design Scope

The interviewer says: "Design a system that counts ad clicks for a small ad network." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Real-time dashboards. Hourly is what was asked.
• Fraud detection. Beyond forged links, not yet.
• Budgets. Ads keep running whatever they've spent; the sales team handles limits by hand.

R1.2 Functional Requirements, Derived Step by Step

Not yet: minute-level counts, fraud scoring, and a separate "final for billing" flag.

R1.3 Non-Functional Requirements: the Questions

We name each quality in words first. The numbers come in R1.7.

• A fast redirect. The user is waiting to see the advertiser's page. Our hop must add only a few milliseconds of server time, and recording the click must never make the user wait on a slow database.
• No lost clicks. A lost click is money we can't bill. Once we've redirected someone, the click must be recorded somewhere durable, even if a downstream part is broken.
• Correct counts. Each real click counted exactly once, in the hour it happened.
• Cheap storage of raw events. We keep every click for recounts and disputes, so raw storage must cost almost nothing per click.

How they pull against each other. The fastest redirect records nothing. The most careful one writes to a database and waits for it. The design below finds the middle: hand the click to a durable log that is built to take writes, then redirect.

R1.4 The API

The click link

The ad server, at the moment it shows an ad, builds a link that carries a signed token:

httpCopy
GET /c/v1.k7.Y2xrXzJiOWU0MWQwN2E2ZjRjMTE.Jx9fQ2mB0s1PzK4dRw7aNg HTTP/1.1
Host: c.adnet.example

httpCopy
HTTP/1.1 302 Found
Location: https://shoes.example/spring-sale?utm_source=adnet
Cache-Control: private, no-store

We answer 302 with no-store, so every click reaches us and nothing is cached by the browser. The URL shortener loop explains that choice in detail: URL shortener, step 1.4.

The token's parts (the payload is base64url-encoded; step 1.4 explains the signature):

The advertiser's destination URL is not in the token. The redirect service looks it up by ad_id in an in-memory copy of the ad catalog. Taking the target from the request would make us an open redirect: anyone could craft our trusted domain into a link to a phishing site.

The click event the redirect service records (about 500 bytes with the user agent):

jsonCopy
{
  "request_id": "r-7f3a9c2e51d0",
  "click_id": "c-2b9e41d07a6f4c1188d3e0a5b7f21c64",
  "ad_id": "ad_1042",
  "campaign_id": "cmp_88",
  "publisher_id": "pub_311",
  "cpc_micros": 450000,
  "impression_time": "2026-09-28T10:14:03.120Z",
  "event_time": "2026-09-28T10:14:09.874Z",
  "ip": "203.0.113.24",
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) ...",
  "redirect_host": "redir-b-2"
}

request_id is unique per HTTP request; click_id is unique per impression. A double-click produces two request_ids with one click_id. event_time is stamped by the redirect server when the request arrives; for a web click, that is the moment of the click.

The report

httpCopy
GET /v1/ads/ad_1042/metrics?from=2026-09-28T08:00Z&to=2026-09-28T11:00Z&granularity=hour HTTP/1.1
Authorization: Bearer <advertiser token>

jsonCopy
{
  "ad_id": "ad_1042",
  "granularity": "hour",
  "rows": [
    { "hour": "2026-09-28T08:00Z", "clicks": 81,  "spend_micros": 36450000 },
    { "hour": "2026-09-28T09:00Z", "clicks": 94,  "spend_micros": 42300000 },
    { "hour": "2026-09-28T10:00Z", "clicks": 88,  "spend_micros": 39600000 }
  ]
}

Status codes

Recap

• One click path (GET /c/{token} → 302) and one report path.
• 10M clicks a day, 500/s at peak, 5,000 active ads, one region.
• Counts per ad per hour, exact, with duplicates removed, bucketed by when the click happened.
• Never block the redirect on anything slow.

R1.5 Design Evolution: From a Table of Clicks to Hourly Counts

Every step follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is usually the next problem.

Step 1.0: The Baseline

The redirect handler inserts a row per click into a relational table, then redirects. Reports run SELECT ad_id, COUNT(*) FROM clicks WHERE ts BETWEEN ... GROUP BY ad_id.

Synthesizing vector architecture diagram...

One table does two jobs: it takes every click on the redirect path and answers every report.

It works on day one. Everything below exists because this table is on the user's path and is being asked to be both a write log and an analytics engine.

Step 1.1: Recording Clicks Slows the Redirect

Why a stream and not a direct call to a counting service? A direct call couples the redirect to that service's health and speed, the same problem as the database. The log absorbs bursts, keeps records while consumers are down, and lets us add consumers later without touching the redirect.

Primitive: Message Queues vs Event Streams · Drill: Message queue order pipeline

Step 1.2: Reports Scan Millions of Rows

Synthesizing vector architecture diagram...

Raw clicks flow left to right into cheap files; one small table holds the only numbers reports ever read.

Step 1.3: A Double-Click Is Counted Twice

Step 1.4: Tampered Links Fake Clicks for Other Ads

Step 1.5: Which Hour Does a Click Belong To?

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

The click path touches only the load balancer, the redirect service and Kinesis. Everything after Kinesis can fail or run late without the user noticing.

Why these pieces:

• Application Load Balancer terminates TLS and spreads clicks over the redirect tasks.
• Redirect service on ECS (Fargate), one task per AZ at minimum. Each task holds the ad catalog (5,000 ads, a few MB) and the signing keys in memory, so the click path has no lookups.
• Kinesis Data Streams, one provisioned shard. The partition key is click_id, which spreads records evenly (there is only one shard now, but Round 2 will add more).
• Firehose reads the stream, converts to Parquet using a schema in the AWS Glue Data Catalog, and writes to s3://clicks-lake/arrival_hour=2026-09-28-11/... every 300 seconds. Records that fail conversion go to an errors/ prefix instead of disappearing.
• The hourly job: EventBridge Scheduler starts a Step Functions workflow at 10 minutes past each hour. It runs one Athena query, then a Lambda function writes the result rows to DynamoDB. A second schedule at 02:10 recounts the whole previous day.
• DynamoDB holds the only numbers reports read.

The data model

click_hourly (DynamoDB):

The raw lake table (Glue Data Catalog, read by Athena):

The hourly query. At 11:10, the job recounts event hours 08:00, 09:00 and 10:00. It reads arrival hours 07:00 through 11:00: 07:00 is the lookback hour, and 11:00 holds clicks from 10:xx that arrived after 11:00.

sqlCopy
WITH first_copy AS (
  SELECT click_id,
         min(event_time)        AS event_time,
         arbitrary(ad_id)       AS ad_id,
         arbitrary(cpc_micros)  AS cpc_micros
  FROM clicks_raw
  WHERE arrival_hour BETWEEN '2026-09-28-07' AND '2026-09-28-11'
  GROUP BY click_id
)
SELECT ad_id,
       date_trunc('hour', event_time) AS hour,
       count(*)                        AS clicks,
       sum(cpc_micros)                 AS spend_micros
FROM first_copy
WHERE event_time >= TIMESTAMP '2026-09-28 08:00:00'
  AND event_time <  TIMESTAMP '2026-09-28 11:00:00'
GROUP BY 1, 2;

GROUP BY click_id keeps one row per click, so a double-click counts once, in the hour of its first copy. The lookback hour matters for a double-click at 07:59:59 and 08:00:01: without arrival hour 07:00, the job would see only the second copy and count it as a new click in hour 08:00.

The Lambda writer then does, for each result row, an UpdateItem with SET clicks = :c, spend_micros = :s, computed_by = :run. It sets the value; it never adds to it. R1.9 shows why.

Tracing the two flows

1. A click

Synthesizing vector architecture diagram...

The user waits for the signature check and one log append, about 20 ms of server time. Counting happens later, off the path.

2. A report, and when its numbers appear

1. The click above has event_time 10:14:09 and arrives in arrival hour 10.
2. At 11:10 the hourly job recounts event hours 08:00–10:00; the 10:00 row now includes it: clicks = 88.
3. At 11:13 (about 1 minute for the query, 1–2 minutes for writing up to 15,000 items) the new rows are in DynamoDB.
4. The advertiser's GET /v1/ads/ad_1042/metrics?... is one DynamoDB Query on ad_id = ad_1042 and hour BETWEEN '2026-09-28T08' AND '2026-09-28T10': 3 items, a few milliseconds.
5. The 10:00 row is recomputed again at 12:10 and 13:10, and once more by the daily run at 02:10 the next day. If a late copy or a late click arrived, the number changes; otherwise the same value is written again.

R1.7 Numbers

Targets

Traffic and bytes

Kinesis

Freshness, step by step. These happen one after another, so the worst cases add:

Rough monthly cost (us-east-1 list prices; 30.4 days a month, 730 hours; check the AWS Pricing Calculator before quoting)

Two lessons:

• Firehose's 5 KB rounding makes us pay for 10× the bytes we send (1,520 GB billed for 152 GB of data). Packing several clicks into one record would cut the two Firehose lines from $71 to about $8. At this size, the $63 a month isn't worth the extra code and the new failure modes; in Round 2 it's worth thousands a month, and we pack.
• The redirect tier is the most expensive part, and it's the only part the user feels. That's the right place for the money.

R1.8 Trade-Offs

Stream vs direct database writes

Hourly batch vs a streaming consumer, for this scope

Hourly was what was asked. We'd switch the moment someone needs minutes, and Round 2 does.

Which store for the aggregates

For time-series storage in general (compression, rollups, retention tiers), see the metrics monitoring loop. Our aggregates are much simpler: one number per ad per hour.

R1.9 Failure Modes

A note on the drill: a consumer that fails halfway here is either Firehose (which retries, and sends records it can't convert to an error prefix, our dead-letter queue) or the hourly job (which is safe to retry because it overwrites). Neither loses a click or counts one twice.

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Takes the database off the redirect path and explains why a log, not a background thread.
• Pre-aggregates into one row per ad per hour and keeps raw clicks cheaply for recounts.
• Deduplicates on an ID that every copy of a click shares, and explains where that ID comes from.
• Signs click links, and doesn't take the redirect target from the request.
• Buckets by event time, and can say what goes wrong with arrival time.
• Writes aggregates with overwrite, not increment, and says why.

Follow-up questions

1. "Why not have the redirect service increment a counter in DynamoDB directly? It's only 500 writes a second." Answer: it would work at this size and cost about $0.625 per million clicks, but it puts a database write back on the user's path, and an increment isn't safe to retry: a timeout after a successful ADD followed by a retry counts the click twice, and a double-click adds twice unless we also keep a dedup record per click. The log plus an hourly recount gets exactness for free.

2. "An advertiser says their 10:00 number changed between 11:15 and 12:15. Bug?" Answer: probably not. Every hour is recomputed three times and again the next day, so a late click or a late duplicate can move it. We should say so in the API: Round 2 adds a final flag, and here the report can already show "this hour may still change until 02:10 tomorrow".

3. "How do you rotate the signing key without breaking links already on web pages?" Answer: the key ID in the token. We add the new key to every verifier first, then switch the ad servers to sign with it. Old links carry the old key ID and keep verifying until the old key is retired, 2 hours later: longer than a token's 1-hour life.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "A Billion Clicks a Day, Billed Exactly"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 1B clicks/day: 11,574/s average, 100K/s peak · 100K active ads · dashboards < 3 s fresh (P99) · zero double-billing · redirect 99.95%

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "A small ad network: 10 million clicks a day, 500 a second at peak, 5,000 ads. Ad links carry an HMAC-signed token with a per-impression click_id, the ad, the price and the impression time. A stateless redirect service checks the token, appends the click to a one-shard Kinesis stream, and sends a 302; if Kinesis fails it spools locally and redirects anyway. Firehose writes the stream to S3 as Parquet every 5 minutes. Every hour, an Athena query recounts the last three event hours, keeping only the first copy of each click_id, and a Lambda sets one DynamoDB item per ad per hour. A daily run recounts the whole previous day. Reports read DynamoDB. About $220 a month. Open costs: counts are an hour old, nothing filters bots, and the design has never seen a hot ad or a late phone."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: a log on the click path, a lake of raw clicks, and an hourly exact recount.

Round 1 step summary

Open costs: hour-old numbers; no fraud filtering; one shard; no idea what to do with clicks that arrive long after they happened.

R2.1 The Scope Raise

Interviewer: "We've been bought by a large ad platform. It's now 50 billion impressions a day. Advertisers' bidding tools poll their numbers constantly and want them within 3 seconds. Every valid click must be billed exactly once: finance will not accept a double charge, ever. Our mobile SDK records clicks on the phone, and phones that were offline upload them up to 45 minutes late. Bot farms are clicking our ads. And during the Super Bowl, one advertiser's campaign alone got 40,000 clicks a second."

Scope change

R2.2 What Breaks in the Round 1 Design

We fix them in this order: freshness (2.1), completeness of a window (2.2), late clicks (2.3), crashes and replays (2.4), the hot ad (2.5), bots (2.6).

R2.3 New Requirements and API Additions

Minute-level metrics with valid and invalid counts

httpCopy
GET /v1/ads/ad_1042/metrics?from=2026-09-28T10:10Z&to=2026-09-28T10:15Z&granularity=minute HTTP/1.1
Authorization: Bearer <advertiser token>

jsonCopy
{
  "ad_id": "ad_1042",
  "granularity": "minute",
  "data_complete_through": "2026-09-28T10:14:06Z",
  "rows": [
    { "minute": "2026-09-28T10:12Z", "valid": 14, "invalid": { "duplicate": 1, "velocity": 0, "datacenter": 2 }, "spend_micros": 6300000, "status": "provisional" },
    { "minute": "2026-09-28T10:13Z", "valid": 11, "invalid": { "duplicate": 0, "velocity": 1, "datacenter": 0 }, "spend_micros": 4950000, "status": "provisional" },
    { "minute": "2026-09-28T10:14Z", "valid": 3,  "invalid": { "duplicate": 0, "velocity": 0, "datacenter": 0 }, "spend_micros": 1350000, "status": "in_progress" }
  ]
}

• data_complete_through is the stream's watermark (step 2.2): every minute before it is complete as far as the stream knows. It also tells a caller whether the pipeline is moving: if it stops advancing, the numbers are stale, even when nothing looks wrong.
• status is in_progress (the minute is still open), provisional (closed in the stream, not yet recounted), or final (from the exact recount, step 2.3).

The billing export, with a finalized flag

httpCopy
GET /v1/billing/daily?account=acct_77&date=2026-09-27 HTTP/1.1

jsonCopy
{
  "account": "acct_77",
  "date": "2026-09-27",
  "time_zone": "America/New_York",
  "finalized": true,
  "finalized_at": "2026-09-28T06:20:00Z",
  "lines": [
    { "ad_id": "ad_1042", "valid_clicks": 2210, "invalid_clicks": 131, "spend_micros": 994500000 }
  ]
}

A day in the advertiser's time zone (here UTC 04:00 on the 27th to 04:00 on the 28th) is finalized once its last hour is final: 2 hours 20 minutes after that hour ends (R2.6 shows the timing).

SDK click upload

httpCopy
POST /v1/sdk/clicks HTTP/1.1
Content-Type: application/json

{
  "batch_id": "b-5d1e0c77a2",
  "clicks": [
    { "token": "v1.k7.Y2xr...", "device_time": "2026-09-28T09:31:12.004Z" },
    { "token": "v1.k7.Y2xr...", "device_time": "2026-09-28T09:33:40.519Z" }
  ]
}

jsonCopy
{ "accepted": 2 }

The app resends a batch whose response it never got, so some batches arrive twice. That's safe: every click inside carries its own click_id, and step 2.6 drops the copies.

The event-time rule for device clocks. A phone's clock can be minutes or days off, and a fraudster can set it to anything. The ingest service computes:

event_time = clamp(device_time, impression_time, arrival_time)

The click can't have happened before the signed impression, or after it reached us. A click whose arrival_time − impression_time is more than 2 hours is recorded as invalid (stale), never billed. Forty-five minutes of offline upload plus up to an hour between impression and click fits inside that.

R2.4 Design Evolution: Seconds-Fresh, Exactly Billed

Step 2.1: Counts Must Be 3 Seconds Fresh

The meaning of the window, as streaming SQL:

sqlCopy
CREATE TABLE clicks (
  click_id    STRING,
  ad_id       STRING,
  cpc_micros  BIGINT,
  verdict     STRING,          -- 'valid' or an invalid reason, set by step 2.6
  event_time  TIMESTAMP(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH ( 'connector' = 'kinesis' /* stream name, region, EFO options */ );

SELECT ad_id,
       window_start,
       SUM(CASE WHEN verdict = 'valid' THEN 1 ELSE 0 END)          AS valid_clicks,
       SUM(CASE WHEN verdict <> 'valid' THEN 1 ELSE 0 END)         AS invalid_clicks,
       SUM(CASE WHEN verdict = 'valid' THEN cpc_micros ELSE 0 END) AS spend_micros
FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '1' MINUTE))
GROUP BY ad_id, window_start, window_end;

We use the SQL to show what the window means. The production job is written with Flink's DataStream API, because we need two things SQL windows don't give us: a side output for late records (Flink SQL windows simply drop them) and custom early firings.

Primitive: Message Queues vs Event Streams

Step 2.2: When Is a Minute Complete?

A small trace of a single shard:

Step 2.3: Clicks Arrive 45 Minutes Late

Synthesizing vector architecture diagram...

Both paths read the same stream. The stream answers in seconds; the recount answers exactly, hours later, and writes into its own attributes.

Primitive: Event Sourcing & CQRS (the lake as the log of record; tables rebuilt from it)

Step 2.4: A Worker Crash Replayed a Minute and Double-Billed

Synthesizing vector architecture diagram...

The replay repeats the output, and the write is a SET of an absolute total, so the repeat changes nothing. An ADD would have made it 1,624.

Primitive: Two-Phase Commit & Saga Orchestration

Step 2.5: One Campaign's 40K Clicks a Second

Synthesizing vector architecture diagram...

Each subtask sends two partial sums a second for the hot ad instead of 625 clicks, so its owner receives 128 records a second (up to ~256 at a minute boundary), not 40,000.

Primitive: Database Sharding & Partition Keys

Step 2.6: Bots Are Clicking

Remembering 2 hours of click IDs. Three ways to do step 1:

How the exact state expires. Flink's built-in state TTL counts processing time, and RocksDB physically removes expired entries only when it compacts, so a TTL alone could forget an ID early during a backlog, or keep it long after. Instead, each entry is cleared by an event-time timer at impression_time + 2 h. The timer fires only when the watermark passes that time, so lag can't expire it early. A copy that is still accepted must arrive within 2 hours of its impression, and the watermark can't pass that point before the copy is read (event time is clamped to arrival), so every accepted copy finds its original. A 26-hour processing-time TTL stays on as a guard in case the watermark ever gets stuck; if it ever fires first, the recount still catches the duplicate.

The Bloom filter math, for reference: with $m = -n \ln p / (\ln 2)^2 = 60{,}000{,}000 \times 6.908 / 0.4805 \approx 862.6$ million bits $\approx 107.8$ MB, and $k = (m/n)\ln 2 \approx 14.4 \times 0.693 \approx 10$ hash functions.

Why we don't choose it here:

• False positives drop real, billable clicks. At its design load, 0.1% of new clicks look like duplicates. Because the filter is sized for the peak, it's almost empty at average load (6.9M of 60M entries in 10 minutes), where the false-positive rate is about $(1 - e^{-10 \times 6.9/862.6})^{10} \approx 10^{-11}$, so the loss is concentrated in the busiest minutes.
• Overfilling is silent. If a botnet storm pushed 200M IDs into the 60M-entry filter, it would have 4.3 bits per entry and a false-positive rate of $(1 - e^{-10/4.31})^{10} \approx 35\%$: a third of real clicks thrown away as duplicates, with no error anywhere. A filter must rotate on a count as well as a timer, or be rebuilt larger.
• It only covers 10–20 minutes, and SDK resends come 45 minutes later.
• The memory argument is weak inside Flink. Flink's state lives in RocksDB on each worker's local disk (50 GB per KPU), and RocksDB keeps its own small Bloom filter per file, so a lookup for a new ID rarely touches disk. 5 GB of exact state (43 GB in the worst case of 2 hours at full peak) costs nothing extra.

A Bloom filter is the right tool when an exact set would not fit anywhere affordable, as in a crawler's billions of URLs. Here the exact set fits.

Why not an external Redis for dedup and velocity at all? Besides the memory, every click would cost a network round trip, and, more important, external state isn't part of Flink's checkpoint. After a restore, Flink replays clicks that Redis already recorded, so they'd be marked as duplicates: a crash would under-bill. Keeping this state inside Flink makes it roll back with everything else.

Primitives: Bloom Filters & Counting Filters · Distributed Rate Limiting (the per-IP velocity rule is a rate limit that marks instead of blocking) · Drill: Bloom filter crawler dedup

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Two readers of one stream: Flink for seconds-fresh provisional numbers, Firehose for the lake that the exact hourly recount reads. The dashboard reads the last two hours from Valkey and older history from DynamoDB.

Why these pieces:

• Network Load Balancer in TCP pass-through mode; the ingest tasks terminate TLS themselves. Nearly every click is a new TLS connection, and a Layer 7 load balancer bills per new connection (R2.6 compares them).
• Redirect and SDK ingest service on EC2 (Graviton c7g.xlarge), at least 2 per AZ. It checks tokens, applies the event-time clamp, packs clicks into Kinesis records by bytes (concatenated JSON objects), closing a record before the next click would push it past ~4.8 KB, so no record crosses Firehose's 5 KB billing step; at a 500 B average that's about 9 clicks. It writes with a random partition key. It redirects without waiting for Kinesis, holding at most 100 ms of clicks in memory (the cost of that choice is in R2.7). It keeps Round 1's local disk spool for Kinesis errors, and publishes its counters every minute: requests accepted, records delivered, clicks spooled, and its "delivered-through" time.
• Kinesis Data Streams, 64 provisioned shards (R2.6), reached through an interface VPC endpoint.
• Flink reads with enhanced fan-out (EFO): each consumer gets its own 2 MB/s per shard, pushed to it, instead of sharing 2 MB/s and five GetRecords calls per second with Firehose.
• Valkey holds each ad's per-minute running totals for the last 2 hours: what dashboards poll. Each of its 3 shards has a primary and two replicas, one node in each AZ, and the query service reads from the node in its own AZ, so the 10K reads a second don't pay cross-AZ transfer.
• DynamoDB holds closed minutes (8 days), hourly rows (13 months), the late totals and the final counts.
• The exceptions log is Flink writing, for every click it did not count as valid on time (duplicates, invalid clicks, late clicks), one line to S3: request_id, click_id, reason and job_gen. For a duplicate it records the request_id it dropped, so the recount knows exactly which copy the stream kept. Flink's file sink commits files only when a checkpoint completes, so the log matches the stream's state exactly. It's small: about 5% of clicks, ~60 bytes each.
• Firehose turns the stream into the Parquet lake, as in Round 1.
• The hourly job recounts from the lake and writes final_*; the daily billing export reads final rows only.

The Flink job

Synthesizing vector architecture diagram...

Three shuffles: by click for dedup, by source and ad for velocity, by ad for counting. Pre-aggregation sits just before the last one, where the hot key would otherwise land, and it is where lateness is decided, click by click. Hour rows' fast_* values are the sum of that hour's closed minutes, rolled up by the window operator when the hour's last minute closes.

The data model

ad_metrics (DynamoDB), one table for ads and campaigns:

Each writer sets only its own attributes with UpdateItem ... SET, so the stream and the recount never overwrite each other. The minute rows expire through DynamoDB TTL, which deletes items typically within a few days after expires_at, not at an exact time; the query service ignores minute rows older than 7 days, so a late deletion is invisible.

Valkey keys (cluster mode; the {ad_1042} hash tag keeps an ad's keys in one slot so one MGET reads them):

Flink only rewrites an ad's key when its count changed, so an ad with no new clicks is never touched. The wm:* heartbeats tell the query service the difference between "no clicks" and "the pipeline stopped": data_complete_through is the minimum over all subtasks' wm:* keys, and a missing key means that subtask is not reporting.

Reconciliation: three equations that cover every path

A number we can't explain is a number we can't bill. Every hour, the recount checks three equations. Any non-zero difference holds that hour's billing and pages the on-call.

1. Ingest. For each ingest host and minute: requests accepted (its own counter) = distinct request_ids in the lake + requests still in its spool. This proves the lake has everything the edge took in.

2. Stream. For each minute, from Flink's counters: records read = on-time valid + on-time invalid + duplicates + late + unparseable (sent to a dead-letter prefix). This proves the stream dropped nothing unaccounted.

3. Fast vs final, for each ad and event hour:

   final_valid = fast_valid + late_valid − A + B

   • A = clicks the stream counted valid that the recount rejects: offline fraud verdicts (and duplicates, which should be zero because the dedup state covers the whole acceptance window).
   • B = clicks the recount counts valid that the stream rejected: for example velocity verdicts overturned because the IP is a known carrier gateway.
   • The recount computes A and B click by click, by joining the lake with the exceptions log (keeping the highest job_gen per click). The stream's valid set is every copy in the lake whose request_id is not in the log as duplicate or invalid; late entries stay in the set, because they are counted in late_valid. For duplicates, the recount keeps the same copy the stream kept (the one whose request_id the log didn't drop), not the one with the earliest event_time, so a double-click across an hour boundary lands in the same hour in both counts. The equation must balance exactly. The same equation runs on spend_micros.

Trace 1: a normal web click (times from R2.6's freshness budget)

Synthesizing vector architecture diagram...

The user leaves after the token check. The count reaches Valkey through two buffers (100 ms at ingest, 500 ms of pre-aggregation) and one timer (the 1-second firing).

Trace 2: a click that arrives 45 minutes late

1. 09:31:12 on the phone: the user taps an ad. The app records it with the token and device_time 09:31:12.
2. 10:16:03: the phone uploads. The ingest service computes event_time = clamp(09:31:12, impression 09:30:58, arrival 10:16:03) = 09:31:12. Arrival − impression = 45 min 5 s, under 2 hours: accepted.
3. Flink: dedup says new; velocity and ranges pass. At the pre-aggregation operator, the watermark is ~10:15:58, far past the end of its minute (09:32:00): late.
4. The side output adds it to (ad_1042, 09:00); within 10 seconds, the hour row's late_valid goes from 22 to 23. The dashboard's 09:00 hour now includes it. The minute row for 09:31 does not change.
5. The exceptions log records c-… late.
6. Hour 09:00 (it ends at 10:00) is recounted by the 10:20, 11:20 and 12:20 runs. The 10:20 run may miss this click, since Firehose delivers it by about 10:21; the 11:20 run includes it in final_valid. The 12:20 run is the first that starts at least 2 h 5 min after 10:00, so it marks hour 09:00 final.

Trace 3: a replay after a crash

1. 10:04:30: checkpoint 1,207 completes (every operator's state plus each shard's read position).
2. 10:04:52: a worker dies. Window 10:03 closed at about 10:04:05, before the checkpoint, so its row is written and its state gone; window 10:04 is still open.
3. ~10:05:40: the service has restarted the job from checkpoint 1,207 (restart time of about 50 seconds is an assumption; we measure it). Flink rereads everything after the saved positions: about 70 seconds of clicks, ~810,000 at the average rate.
4. Before emitting anything new, every sink operator does its full flush: it rewrites each key it holds in restored state (open minutes in Valkey, late totals and other pending rows in DynamoDB), about 2 minutes for the ~300K late-total keys at 2,800 WCU. Any value written between 10:04:30 and the crash is replaced, including keys the replay will never touch; keys created after 10:04:30 that restored state doesn't hold are found by their ckpt tag and deleted.
5. Early firings for 10:04 briefly show smaller running totals in Valkey (state rolled back to 10:04:30), then climb back as the replay catches up. data_complete_through stays at about 10:04:25 until then, so tools can tell.
6. With 32 KPUs processing about 64K clicks/s and 11.6K/s arriving, the backlog drains at ~52K/s: about 16 seconds of processing, running alongside the flush.
7. Window 10:04 closes on the watermark and writes SET fast_valid = … once. No row was added to twice.

R2.6 Numbers and Cost

Targets

We don't promise "five nines" (99.999%, about 5 minutes a year), which a common textbook answer claims for click ingestion: one region can't honestly back it. Round 3 adds regions and raises the target to 99.99%.

Traffic, and one planning peak

A common textbook answer calls 100K/s a "5× surge", but 5 × 11,574 is 57,870. We use one peak, 100K/s, everywhere below, and say how we got it.

Bytes and storage

Kinesis: records and bytes, unpacked vs packed

With a random key per record, each shard's share varies only by chance. Records per shard per second behave roughly like a Poisson count with mean 174, so the standard deviation is about √174 ≈ 13, and the busiest of 64 shards in a given second sits about 2.4 standard deviations up: ~205 records/s × ~4.5 KB ≈ 0.92 MB/s, about 18% above the mean and still under the 1 MB/s limit. That is why we keep the 80% headroom rather than sizing to the mean.

Provisioned vs on-demand Kinesis. On-demand needs no shard planning: a new on-demand stream starts at 4 MB/s of writes and adapts to twice its highest write rate of the previous 30 days, up to 10 GB/s in us-east-1 and eu-west-1 and 200 MB/s by default in most other regions. A jump beyond twice the previous peak can throttle for about 15 minutes, though On-demand Advantage lets us set warm throughput ahead of an event. In every mode, one partition key still can't exceed a single shard's limits. At our volume:

Provisioned wins here because our traffic follows a predictable daily curve and our spikes are scheduled events we pre-scale for. On-demand would be the right call for a new, unpredictable workload.

Flink

AWS's starting guidance for Managed Service for Apache Flink is about 1 MB/s per KPU (a KPU is 1 vCPU and 4 GB of memory, with 50 GB of local disk). We plan with that and load-test:

We pre-scale on a schedule before known events, because the service's automatic scaling reacts only after 15 minutes of CPU above 75%, doubles parallelism, and restarts the application to do it. 64 KPUs is also the default per-application limit, so any bigger event needs a quota increase requested well ahead. If traffic beats the plan, Flink applies backpressure and falls behind; the 24-hour stream holds the backlog, and dashboards show a stalled data_complete_through rather than wrong numbers.

Flink state

With incremental checkpoints, each checkpoint uploads only new RocksDB files: at the average rate, about 11,574 × 30 s × 60 B ≈ 21 MB of new dedup entries per 30-second checkpoint, plus compaction output.

DynamoDB writes

Against writing every click at peak (100,000/s), aggregation cuts database writes by 98.3% (100,000 → 1,667 for ad rows); against the average of 11,574/s, it's 86%. The minute rows are written as windows close, which is a burst at the top of each minute; the sink spreads each minute's writes over the following ~50 seconds with a rate limiter, so a steady 2,800 WCU is enough.

Freshness, step by step. These happen one after another, so the worst cases add:

We do not cache API responses: each ad is polled once per 10 seconds, so a cache would almost never hit, and a 1-second response cache would eat a third of the budget.

Rough monthly cost (us-east-1 list prices, on-demand, 730 hours and 30.4 days a month; check the AWS Pricing Calculator before quoting)

Synthesizing vector architecture diagram...

No single line dominates. The stream processor is the largest; the lake, which people expect to be expensive, is under a tenth of the bill.

That's about $416 per billion clicks ($12,640 ÷ 30.4B). Four lessons:

• Packing paid for itself many times over. One click per record would bill Firehose for 5 KB × 1B = 5 TB a day: 152 TB × $0.047 ≈ $7,140 a month instead of $794, and need 128 shards (+$1,400 of shard-hours and EFO, +$380 of PUT units). About $8,100 a month saved, for a few lines in the ingest service, and possible only because the partition key no longer carries the ad (step 2.5).
• The request-priced services would have been the biggest bills. API Gateway for 10K dashboard queries/s: 26.3B requests a month at about $0.90 per million ≈ $23,700. AWS WAF in front of the click path: 30.4B requests × $0.60 per million ≈ $18,200. An ALB instead of the NLB on the click path: ~463 LCU for 11,574 new connections/s ≈ $2,700. Each is more than most of the table above.
• The aggregation is the saving. Writing every click to DynamoDB on demand: 30.4B writes × $0.625 per million ≈ $19,000 a month, and the hot ad's item would throttle at 1,000 writes/s per partition.
• Networking choices show up in dollars. The ingest hosts reach Kinesis through an interface endpoint at $0.01/GB; through a NAT gateway it would be $0.045/GB, ≈ $680 a month for the same bytes. Same-AZ Valkey reads save more than the replicas they need.

Two options worth checking before building the lake path, not assumed above: Firehose's pricing for a Kinesis Data Streams source without the 5 KB rounding (a higher per-GB rate on actual bytes), and Kinesis Data Streams' own delivery to S3, which AWS offers only for on-demand streams. Either could remove the need to pack records.

R2.7 Trade-Offs

Window size

Watermark allowance: delay vs completeness

Past a few seconds, a bigger allowance buys almost nothing: the late clicks are the ones delayed by minutes, and no reasonable allowance catches them. They need the late path either way.

Lambda vs kappa

We keep two paths because the bill needs a check that doesn't share the stream's code. The exceptions log keeps the two paths comparable click by click. Repairs come from the lake, the stored source of truth, never by replaying raw stream records, which would re-publish clicks that were later found invalid.

Exactly-once vs idempotent at-least-once

Fast redirect vs zero crash loss

We accept the second: it can only under-bill, never double-bill, and a host crash is rare. A hard requirement of "zero loss" would put the log append back on the user's path.

R2.8 Failure Modes

R2.9 Production Gotchas

1. Writing every click to a database

• Symptom: throttling on one item, a large bill, and totals that are too high after every retry.
• Cause: one UpdateItem ... ADD per click: 100K writes/s at peak, 40K/s on the hot ad's item, and an increment that retries can't undo.
• Fix: aggregate in the stream and write absolute totals per window (steps 2.1, 2.4).

2. Processing time for billing

• Symptom: clicks from 23:59 billed on the next day; a backlog shifts an hour's clicks into the following hour.
• Cause: windows by arrival time or server clock.
• Fix: event time, clamped, with watermarks and a late path (steps 2.2, 2.3).

3. Unbounded click history in Redis

• Symptom: memory grows by ~64 GB a day until the cluster runs out; after a Flink restore, replayed clicks are marked as duplicates.
• Cause: every click_id in a Redis set with no expiry, outside Flink's checkpoints.
• Fix: exact keyed state inside Flink, cleared by an event-time timer at the end of the acceptance window (step 2.6).

4. Blocking the redirect on fraud analysis

• Symptom: click latency rises from ~15 ms to hundreds of milliseconds; users give up before the page loads.
• Cause: a model or a remote lookup on the click path.
• Fix: only the token check on the path; everything else in the stream and offline (step 2.6).

R2.10 Pillar Check

Security in detail. Ingest hosts may only put records to the one stream and read the signing keys. The Flink application's role may read the stream, write to its checkpoint and exceptions prefixes, write ad_metrics and reach Valkey; it can't read the lake. Only the recount job may write final_* attributes (a separate role, with an IAM condition limiting it to those attribute names). The query service is read-only. The lake and checkpoints are encrypted with SSE-KMS.

Operations in detail.

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer adds over L5

• Chooses stream processing for freshness and keeps an exact recount for billing, and says which number is which.
• Defines a watermark precisely, clamps device time, and handles idle shards and late data without reopening windows.
• Says exactly what Flink's exactly-once covers (state) and makes the sink idempotent; knows why a transactional sink would break the 3 s target.
• Does the per-shard math, not just the per-key math, and moves the hot key into Flink with local pre-aggregation.
• Chooses dedup state that rolls back with the job, and can size both the Bloom filter and the exact alternative.
• Reconciles every path with equations that must balance.

Follow-up questions

1. "Why not just run everything in Flink and skip the batch recount?" Answer: we could (kappa), and many teams do for dashboards. For billing we want a second computation that doesn't share the stream's code: if a bad deployment of the job miscounts for an hour, the recount still bills correctly and the equation flags the gap. The price is keeping the rules consistent in two places, which the exceptions log lets us check click by click.

2. "The dedup state is 5 GB on average. What happens during a 2-hour Super Bowl peak?" Answer: up to 720M IDs × ~60 B ≈ 43 GB, spread over 64 KPUs with 50 GB of local disk each (3.2 TB), so it fits with room. What grows is RocksDB compaction work and incremental checkpoint size, which is why we pre-scale and watch checkpoint duration. Event-time timers and RocksDB compaction keep it near two hours' worth.

3. "A publisher says we filtered 30% of their clicks as invalid yesterday. How do you answer?" Answer: from the exceptions log and the final rows: invalid counts by reason, by hour, and the rule that fired. If it's velocity on a carrier gateway range, that's a false positive: we add the range to the reviewed list, and the recount's B term restores those clicks for every hour not yet final. For final hours, it becomes a credit, not a rewrite.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Global, Budget Pacing, Privacy, Attribution"

~45 min · Principal (L7) · 3 regions (us-east-1, eu-west-1, ap-northeast-1), 3 AZs each · 5B clicks/day · 500K active ads, 200K campaigns with daily budgets · 100M conversions/day · redirect 99.99%

R3.0 Where We Left Off

Round 2 in 60 seconds. "A billion clicks a day, 100K/s at peak. Ingest hosts check signed tokens, clamp device time into [impression, arrival], redirect immediately, and pack clicks into records of up to ~4.8 KB (about nine) with a random key: 64 shards. Flink on Managed Service for Apache Flink, 32 KPUs, reads with enhanced fan-out: exact dedup on click_id in keyed state until 2 hours after the impression, a velocity rule per IP or device and ad, data-center ranges, local pre-aggregation that decides lateness click by click, then 1-minute event-time windows per ad with a 5-second watermark. Running totals go to Valkey every second, closed minutes and late totals to DynamoDB, all as absolute SETs, and every sink rewrites all its keys after a restore, so replays are harmless. Firehose writes the Parquet lake; an hourly Athena recount writes separate final attributes, and an hour is final 2 hours 20 minutes after it ends, once every host has delivered. Three equations reconcile ingest, stream and final counts before billing. About $12,600 a month, $416 per billion clicks. Open costs: one region; the counts never influence which ads we serve; raw IPs sit in the lake; and conversions don't exist yet."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: one stream, a fast provisional path and an exact recount that writes its own attributes.

Round 2 step summary

Open costs: one region; nothing feeds back into ad serving; raw IPs and user agents kept indefinitely; no conversions; one region's totals are the whole truth.

R3.1 The Scope Raise

Interviewer: "We're global now: 5 billion clicks a day across the Americas, Europe and Asia. Advertisers set daily budgets, and when a budget runs out their ads must stop, without us overspending by much. Privacy laws in several markets limit what user-level data we keep and where. Advertisers want to know which clicks led to purchases. And finance needs one bill per advertiser that matches across regions."

Scope change

R3.2 What Breaks in the Round 2 Design

We fix them in this order: stopping on budget (3.1), regions (3.2), conversions (3.3), privacy (3.4), billing disputes (3.5), and build-or-buy (3.6).

R3.3 New Requirements and API Additions

Budget and pacing settings

httpCopy
PUT /v1/campaigns/cmp_88/budget HTTP/1.1
Content-Type: application/json

{
  "daily_budget_micros": 2000000000,
  "time_zone": "America/New_York",
  "pacing": "accelerated",
  "regions": ["us", "eu"]
}

pacing: even spreads spend over the day; accelerated spends as fast as traffic allows, subject to the rate cap in step 3.1. cmp_88 runs accelerated for an evening sale, which R3.5's trace follows.

The click reference. At redirect time, the ingest host appends our reference to the advertiser's URL (https://shoes.example/spring-sale?adref=eu.c-7a0c…). The advertiser's site stores the latest one in a first-party cookie and sends it back with the conversion. The eu. prefix names the region that recorded the click, so a conversion can be routed to the one region that holds it.

Conversion ingestion

httpCopy
POST /v1/conversions HTTP/1.1
Authorization: Bearer <advertiser server key>
Content-Type: application/json

{
  "conversion_id": "ord-55120",
  "adref": "eu.c-7a0c5e91b24d4f08a1c3d7e60b95f214",
  "conversion_time": "2026-09-29T09:15:40Z",
  "value_micros": 89990000,
  "consent": { "measurement": true }
}

jsonCopy
{ "status": "accepted", "conversion_id": "ord-55120" }

Advertisers retry, sometimes days later. conversion_id is the idempotency key, and it's enforced twice: the stream remembers IDs for 7 days, and the batch attribution deduplicates on (advertiser, conversion_id) across the whole 13 months the lake keeps. A retry after 8 days finds no stream memory, but the batch still counts it once. A short idempotency window alone isn't a guarantee.

Privacy-safe breakdowns

httpCopy
GET /v1/ads/ad_1042/breakdown?dimension=country&date=2026-09-28 HTTP/1.1

jsonCopy
{
  "rows": [
    { "country": "US", "valid_clicks": 18204 },
    { "country": "CA", "valid_clicks": 1240 },
    { "country": "other", "valid_clicks": 131 }
  ],
  "suppressed_below": 50
}

Any cell with fewer than 50 clicks is folded into other. Only fixed dimensions (country, device type, hour) are offered, so a caller can't subtract two overlapping custom filters to isolate a small group.

Billing statements

httpCopy
GET /v1/billing/statements/2026-09?account=acct_77 HTTP/1.1

jsonCopy
{
  "account": "acct_77",
  "month": "2026-09",
  "days": [
    { "date": "2026-09-28", "valid_spend_micros": 2000250000, "over_budget_free_micros": 250000,
      "billed_micros": 2000000000, "invalid_clicks": 6310, "status": "final" }
  ],
  "adjustments": [
    { "id": "adj-3391", "date": "2026-09-14", "reason": "invalid_traffic_found_later", "credit_micros": -351000000 }
  ],
  "total_billed_micros": 58122640000
}

R3.4 Design Evolution: Feeding Counts Back, Across Regions, Under Rules

Step 3.1: Stop Serving Ads When the Budget Is Spent

The timing chain. Every delay between a click and an ad server acting on it, in order:

Then two terms that are not pipeline delays:

• Clicks on ads already shown. Impressions served before the stop can be clicked later. If the mean delay from impression to click is 20 seconds (an assumption to measure), the clicks still to come equal about 20 seconds of spend.
• Late uploads. About 2% of clicks come from phones that upload them late (Round 2's assumption), and we assume, as a new assumption to measure, that they arrive 15 minutes late on average: 0.02 × 900 s = 18 seconds of spend we haven't seen yet.

Total: 4 + 20 + 18 = 42 seconds. Expected overspend without a margin ≈ spend_rate × 42 s.

With the margin, the expected overspend is about zero; the cap bounds how far a bad minute (more late uploads than usual) can push it. Whatever lands past the budget is served free and shows as over_budget_free on the statement.

How budget state reaches the ad servers

• One writer at a time. Each region runs an active pacer and a standby. The active one holds a lease: an item in a regional DynamoDB table with an epoch number and a heartbeat counter it increments every second. If the standby sees the counter unchanged for 3 seconds (on its own clock), it takes over with a conditional update, SET epoch = :seen + 1 on the condition epoch = :seen, using numbers it read, never DynamoDB's clock. Every sequence number is the pair (epoch, n), so a paused old pacer that wakes up writes with a lower epoch, and everyone ignores it; it also stops itself when it next reads the lease.
• The pacer writes each second's changes as one hash, {pacing}:delta:<epoch>:<n> (campaign → probability and status), with a 5-minute expiry, then sets {pacing}:head = <epoch>:<n>. Every 60 seconds it writes a full snapshot, {pacing}:snap:<epoch>:<n>, with a 5-minute expiry, so only the latest few exist (without an expiry, 12.8 MB a minute would pile up to ~18 GB a day). The shared {pacing} hash tag keeps every pacing key in one shard, so replicas apply them in the order they were written.
• A delta is written every second even when nothing changed. It then holds only a sentinel field _seq, because Valkey deletes a hash that has no fields. The steadily advancing head is the keep-alive: an ad server can tell "no changes" from "the pacer is dead".
• Each ad server polls {pacing}:head, fetches the deltas it's missing in order, and loads the latest snapshot when it starts, when it finds a gap (a delta already expired), or when the epoch rises; it ignores anything with a lower epoch than the one it has.
• If head hasn't moved for 5 seconds, the ad server treats its state as stale and slows down: campaigns with under 10% of their budget left stop, and the rest serve at half their last probability, until fresh state arrives. Stale budget state must fail toward spending less.
• The pacer's budget settings come from the campaigns table's own change stream (DynamoDB Streams), not from the API service writing to two places. On every start, a pacer first notes its position in the change stream (or starts from the stream's oldest record), then scans the whole table (200K items is cheap), then applies stream records from the noted position, keeping only items with a newer version than the scan saw. Streams keep changes for only 24 hours, so a pacer never relies on replaying them alone.
• The budget store is one Valkey primary with two replicas, one node per AZ. Ad servers in the primary's AZ read the primary; the other two AZs read their own replica.

Synthesizing vector architecture diagram...

Ad requests never leave the ad server's memory. Only ~2 small reads a second per ad server cross the network, and each stays inside its AZ, so they don't pay cross-AZ transfer ($0.01/GB each way).

Step 3.2: Clicks Happen in Every Region

Why S3 and not a DynamoDB global table for the 5-second snapshots? A snapshot of 200K campaigns × 16 B is 3.2 MB. DynamoDB bills writes per KB, so one snapshot is ~3,200 write units locally plus 3,200 replicated units in each other region: 640 units/s written and 1,280/s replicated, per region, ≈ 15 billion units a month across three regions, about $9,500 a month at us-east-1 rates. The same data through S3 costs a few PUT requests and about $440 of inter-Region transfer (R3.6).

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active

Step 3.3: A Purchase Happened 3 Days After the Click

Step 3.4: Privacy Rules Limit User-Level Data

Step 3.5: The Advertiser Disputes the Bill

Step 3.6: Build or Buy?

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each region is Round 2's complete stack plus a pacer. Only two things cross regions: 5-second spend snapshots between pacers, and final aggregates to billing. Raw clicks never do.

Why these pieces:

• Route 53 latency records with health checks send each click to the nearest region; each latency record points to a failover pair, the home region first and its standby cell second (below). Failover is decided by Route 53's health checks, which keep working when a region is down; nothing needs an API call to the failed region, or a DNS record change during the outage. We don't alarm on the health checks' own status metric for this; the canary clicks in R3.9 tell us whether clicks actually land.
• Signing keys in every region and cell. Tokens minted anywhere verify anywhere: keys are replicated to each region's Secrets Manager (replicated secrets), so a click rerouted during a failover still verifies.
• Standby ingest cells, one per region, in the same market area: us-west-2 for us-east-1, eu-central-1 (Frankfurt) for eu-west-1, and ap-northeast-3 (Osaka) for ap-northeast-1. A cell is ingest only: load balancer, ingest hosts, a small Kinesis stream, Firehose to a lake bucket in the cell's own region, and a small function that sums spend per campaign every 5 seconds and publishes it like any region's snapshot.

Regional failover

Capacity choice: throttle and spool, count later. A cell is sized for its region's average, not its peak. Above that, Kinesis throttles and the hosts spool to local disk and still redirect, while auto scaling adds hosts over the next ~5–10 minutes; if the region fails at its busiest, some redirects are slow or fail until then, and an alarm on the cell's spool depth and error rate pages. Full N+1 capacity for the largest region's peak (225K/s: ~63 hosts and 144 shards) would cost roughly $8,200 a month for the US cell alone, against ~$1,190 for average capacity (R3.6). Clicks recorded in a cell aren't in any region's Flink: they reach dashboards and bills through the home region's recount once it recovers (the recount reads the cell's lake too), and the cell's spend summaries count every click, even duplicates and invalid ones, so the pacers err toward spending less.

What a failover costs in availability. Health checks every 10 seconds with a failure threshold of 3 detect a dead region in about 30 seconds, and resolvers may keep the old answer for the record's 60-second TTL: about 90 seconds of failed redirects for that region's users. Our 99.99% target allows 4.4 minutes a month (43,800 min × 0.0001), so one regional failover spends about a third of a month's budget.

• Per-region stacks as in Round 2, sized in R3.6. Conversions are ingested in every region and forwarded to the region in their adref.
• The billing region (us-east-1) receives only final hourly aggregates per (ad, hour, region), a few hundred MB a day, and builds daily statements.

Trace 1: a budget running out (cmp_88, US share $1,180, spend rate $0.25/s, margin 0.25 × 42 = $10.50)

Synthesizing vector architecture diagram...

The stop takes effect about 2 seconds after the counter crosses the margin (one pacer cycle, one poll). The margin covers the clicks still to come; the last $0.25 is served free.

Step by step: at $0.25/s, the remaining $20.00 falls to the $10.50 margin in (20.00 − 10.50) ÷ 0.25 = 38 s, at 21:40:38. In-flight clicks bring the US to $1,181.25 (the expected $10.50 plus a little variance). Europe finishes at $819.00. Total $2,000.25: the statement bills $2,000.00 and shows $0.25 as over_budget_free.

Trace 2: an attributed conversion

1. Monday 18:03:12 UTC (20:03 in Paris): a click on ad_1042 is recorded in eu-west-1 with click_id c-7a0c…; the ingest host appends adref=eu.c-7a0c… to the landing URL. Flink stores the click's ad, time, price and valid verdict in its keyed state until 24 hours after the impression.
2. Tuesday 09:15:40 UTC: the user buys. The advertiser's server posts the conversion with that adref to our nearest endpoint, in us-east-1, which forwards it to eu-west-1 (the prefix).
3. Flink looks up c-7a0c…: found (15 h 12 min old, inside the 24-hour preview), valid, consent allows measurement. ad_1042's provisional conversions for Monday go up by one, value $89.99.
4. Wednesday 03:00: the daily batch in eu-west-1 joins Tuesday's conversions with the last 7 days of clicks and confirms it. Monday's attribution becomes final 9 days after Monday.

Trace 3: a billing dispute (the numbers from step 3.5)

1. The advertiser opens a dispute for ad_1042, 2026-09-14, 10:00–11:00.
2. We pull the hour row from each region (all of it was us-east-1): fast_valid 47,500, late_valid 1,505, final_valid 48,240.
3. The recount's saved terms for that hour: A = 780 (offline checks: one publisher's traffic from a device farm), B = 15 (velocity verdicts overturned by the reviewed carrier-gateway list). 47,500 + 1,505 − 780 + 15 = 48,240: the equation balances.
4. We export the 780 clicks' IDs, times and reasons. If the advertiser's logs show real purchases from some of them, the fraud team reviews those clicks; any change becomes an overrides row (for the recount of hours not yet billed) or an adjustments credit for this billed day.

R3.6 Numbers and Cost

Per region

The us-east-1 application needs more than the default 64 KPUs even on a normal day, and 144 for events, so its quota increase must be requested and confirmed before launch; the other two need it for events.

Budget-state reads. "How many budget reads a second?" has two answers:

Attribution state

Lake growth

The rewrite on day 30 replaces each day's files instead of copying them, so no day is ever stored twice.

Rough monthly cost (list prices per region, on-demand; check the AWS Pricing Calculator before quoting)

Checked regional prices: Kinesis shard-hour $0.015 / $0.017 / $0.0195 (US / Ireland / Tokyo), EFO consumer-shard-hour the same, EFO data $0.013 / $0.0147 / $0.0169 per GB, PUT units $0.014 / $0.0165 / $0.0215 per million; Flink KPU-hour $0.11 / $0.12 / $0.142, running storage $0.10 / $0.11 / $0.12 per GB-month; Firehose ingest $0.029 / $0.031 / $0.036 and Parquet conversion $0.018 / $0.019 / $0.022 per GB; S3 Standard $0.023 / $0.023 / $0.025 per GB-month (first 50 TB). For EC2-based lines outside us-east-1 we apply an estimated +10% (Ireland) and +20% (Tokyo).

Synthesizing vector architecture diagram...

The cross-region line is small because only snapshots and aggregates cross between regions. Tokyo carries 25% of the clicks and about 27% of the cost, from its higher list prices.

That's about $494 per billion clicks ($75,100 ÷ 152B clicks a month). Lessons:

• Regional prices move the bill: Tokyo's Flink KPU-hour is 29% above us-east-1's, its Firehose ingest 24% above.
• What crosses regions decides the transfer bill, and so does how it crosses: DynamoDB global tables for the snapshots would cost ~$9,500 in write units alone.
• The stream processor is the largest single service in every region, which is why step 3.6 revisits it first.
• Standby capacity is a choice with a price: ~$3,000 a month buys average-rate cells in every market area; peak-rate cells would cost several times that.

R3.7 Trade-Offs

Pacing lag vs overspend

Most of the margin isn't our pipeline at all; it's how users behave (click delay, late uploads). Polling faster than once a second buys little.

Attribution models

Privacy vs granularity

Closing the loop. Round 1 asked "how do we count clicks without slowing the click?" and answered with a log, a lake and an hourly exact recount. Round 2 asked "how do we count a billion a day within seconds, and still bill exactly?" and answered with event-time windows, watermarks, exactly-once state with idempotent sinks, local pre-aggregation and three reconciliation equations. Round 3 asked "how do the counts change what we serve, across regions, under privacy rules?" and answered with a pacer and a margin from the full timing chain, regional shares, a keyed attribution join, pseudonymization and an append-only adjustments record. Round 1's rule never went away: every total is set, not added, and the lake is the source of truth.

R3.8 Failure Modes

R3.9 Runbook

Signals, per region OPS 8 · REL 6

Procedure: a hot key

1. Confirm: Flink's backpressure metrics show one subtask of the window operator busy while the rest are idle, and consumer lag climbing.
2. Check the pre-aggregation operator is enabled in the running version (a deployment may have removed it) and that its interval is 500 ms.
3. If the key is a campaign-level sum (not an ad), confirm the campaign aggregation also pre-aggregates.
4. If lag keeps growing, scale the application (command below); the stream holds the backlog meanwhile, and dashboards show a stalled data_complete_through.

Procedure: checkpoint stalls

1. Is backpressure the cause (a hot key or too few KPUs)? Fix that first; checkpoints recover by themselves.
2. Confirm the application hasn't overridden the service's default incremental checkpoints, and that unaligned checkpoints are enabled in the job's code for backpressured periods.
3. Check state size growth: a stuck watermark stops the event-time timers that clear dedup and attribution entries, and then only the guard TTL (and RocksDB compaction) removes them.
4. While checkpoints fail, the job keeps processing, but a crash would replay further back: raise the priority accordingly.

Commands (AWS CLI; names and IDs are placeholders; check them before running)

textCopy
aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/Kinesis --metric-name SubscribeToShardEvent.MillisBehindLatest --dimensions Name=StreamName,Value=clicks-use1 Name=ConsumerName,Value=click-agg-use1 --start-time 2026-09-28T21:00:00Z --end-time 2026-09-28T21:30:00Z --period 60 --statistics Maximum

aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/KinesisAnalytics --metric-name lastCheckpointDuration --dimensions Name=Application,Value=click-agg-use1 --start-time 2026-09-28T21:00:00Z --end-time 2026-09-28T21:30:00Z --period 60 --statistics Maximum

aws kinesisanalyticsv2 describe-application --region us-east-1 --application-name click-agg-use1

aws kinesisanalyticsv2 update-application --region us-east-1 --application-name click-agg-use1 --current-application-version-id 57 --application-configuration-update '{"FlinkApplicationConfigurationUpdate":{"ParallelismConfigurationUpdate":{"ConfigurationTypeUpdate":"CUSTOM","ParallelismUpdate":144,"ParallelismPerKPUUpdate":1,"AutoScalingEnabledUpdate":false}}}'

aws kinesis update-shard-count --region us-east-1 --stream-name clicks-use1 --target-shard-count 288 --scaling-type UNIFORM_SCALING

The first two show Flink's lag behind the stream and its checkpoint duration. The fourth scales the application to 144 KPUs (it restarts from a snapshot, so expect a short pause and a replay); it needs the version ID that describe-application returns. The last doubles the stream, the most one call can do.

Incident flow OPS 10

Synthesizing vector architecture diagram...

The first question is whether money can be wrong. Freshness problems are visible and temporary; money problems are held before they reach a statement.

Game days. Monthly: kill a Flink worker at peak and time the catch-up; stop a pacer and confirm ad servers enter safe mode within 5 s. Quarterly: fail a region in staging and check that shares freeze, clicks move, and the reconciliation equations still balance after recovery. REL 12 · OPS 11

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Closes the loop from counts to serving, and derives the overspend margin from every delay in the chain, including user behavior.
• Bounds overspend by construction (a rate cap), and makes stale budget state fail toward spending less.
• Splits a global budget into regional shares with no cross-region call on the serving path, and freezes a stale region's share instead of reallocating it.
• Attributes conversions with a key it controls rather than a cross-site identity.
• Turns privacy rules into mechanisms: deleted keys, two retention tiers, regional data, thresholds, consent flags, and expiry of every copy.
• Keeps billing auditable: finals written once, adjustments and overrides as separate layers, cost per billion clicks tracked.

Follow-up questions

1. "Why not reserve the whole global budget in each region and stop when any region sees it spent?" Answer: each region would see only its own spend, so all three could spend nearly the full budget: up to 3× overspend. Shares make each region's limit meaningful locally; rebalancing moves budget toward where it's being spent, and the floors let a quiet region serve a sudden burst.

2. "An advertiser in Germany asks where their click data is and how long we keep it." Answer: clicks recorded in eu-west-1 stay there, and during a regional failover they go to our Frankfurt cell, still in the EU: full detail with pseudonymized keys for 30 days, then without them for 13 months, then deleted, including old object versions. Only per-ad hourly aggregates go to the billing region. The monthly keys that pseudonymize IPs and device IDs are deleted once that month's last day has been rewritten, so after that nothing can be linked back to an IP. For the legal specifics, we point to what the legal team has set per market; the pipeline enforces it.

3. "Your margin assumes a 20-second click delay. What if it's really 2 minutes?" Answer: then T becomes 4 + 120 + 18 = 142 s, and without a change we'd overspend about 3.4× more than planned. We measure it: the recount knows each click's impression time, so the pacer's T is recomputed daily from the observed distribution, per region and per device type, and the rate cap follows (budget ÷ (100 × T)).

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design, let me ask: what's counted, how fresh must it be, which number is billed, and how late and how duplicated can clicks be?"
• When the scope is raised: "Here's what breaks in the current design, and here's the order I'll fix it in."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels