Design Real-Time Ad Click Event Aggregation

1. Problem Statement & Scope

System Mission

Design a mission-critical, low-latency, real-time ad click event ingestion and aggregation pipeline capable of processing hundreds of thousands of ad clicks per second, computing multi-granularity tumbling/sliding window metrics (1-minute, 1-hour, 24-hour CTR and spend), filtering click fraud and botnet attacks in real time, and maintaining strict exactly-once processing guarantees for advertiser billing.

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Functional Requirements

1. Ad Click Ingestion: Securely collect, authenticate, and validate ad click tokens globally with sub-$20\text{ ms}$ response times.
2. Multi-Window Real-Time Aggregation: Aggregate click counts, conversion counts, and total monetary spend partitioned by ad_id, campaign_id, and publisher_id across 1-minute, 1-hour, and 24-hour sliding/tumbling windows.
3. Real-Time Click Fraud & Botnet Filtering: Discard duplicate clicks, bot click-farms, and rapid replay clicks before aggregating billing figures.
4. Advertiser Analytics Query API: Provide sub-$50\text{ ms}$ query responses for advertiser campaign dashboards tracking Click-Through Rate (CTR) and budget pacing.
5. Historical Replay & Data Lake Audit: Persist raw click streams in a Snappy-compressed Parquet data lake for regulatory reconciliation and ML model retraining.

Non-Functional Requirements (SLAs/SLOs)

• High Throughput: Baseline $50,000\text{ clicks/sec}$; peak capacity up to $\mathbf{200,000\text{ clicks/sec}}$.
• Aggregation Freshness / Pipeline Latency: Aggregated metrics reflected in advertiser dashboards within $< 3\text{ seconds}$ ($P99$).
• Data Accuracy & Billing Consistency: Strict Exactly-Once processing semantics. Zero duplicate billing for advertiser clicks.
• High Availability: $99.999\%$ uptime for click ingestion endpoints.
• Fault Tolerance: Lossless stream recovery with stateful checkpoints in $< 60\text{ seconds}$ during worker failure.

Out-of-Scope

• Real-Time Bidding (RTB) auctions and ad serving exchange matchmaking (sub-$50\text{ ms}$ DSP/SSP ad bidding covered in dedicated ad serving blueprints).
• Conversion attribution attribution modeling (e.g. multi-touch Shapley value attribution algorithms).

2. Capacity & Scale Estimation

Traffic & Event Volume Calculations

• Daily Ad Impressions: $50\text{ Billion}$ impressions/day.
• Average Click-Through Rate (CTR): $2\% \implies 1\text{ Billion clicks/day}$. $\text{Average Ingestion QPS} = \frac{10^9 \text{ clicks}}{86,400 \text{ s}} \approx 11,574 \text{ clicks/sec}$ $\text{Peak Ingestion QPS (5}\times\text{ Super Bowl / Black Friday)} = \mathbf{100,000\text{ clicks/sec}}$

Storage & Data Sizing Math

• Raw Click Event Payload Size: $\approx 500\text{ Bytes}$ (JSON/Protobuf: click_id, ad_id, campaign_id, user_token, ip_hash, cpc_cents, timestamp). $\text{Peak Ingestion Network Bandwidth} = 100,000 \times 500\text{ Bytes} = 50\text{ MB/s} = \mathbf{400\text{ Mbps}}$ $\text{Daily Raw Click Storage} = 10^9 \times 500\text{ Bytes} = 500\text{ GB/day}$ $\text{Snappy-Parquet Compressed Storage (4}\times\text{ compression)} = \mathbf{125\text{ GB/day}} \implies 45.6\text{ TB/year}$
• Aggregated Metric Record Volume:
  • 100,000 active campaigns aggregated per minute $\implies 100,000 \text{ rows/minute} = 1,667 \text{ writes/sec}$ to DynamoDB (vs. 100,000 raw writes/sec, a $98.3\%$ write load reduction).

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Component Responsibility Breakdown

4. API Interface Design & Wire Protocols

1. Ingest Ad Click Event

httpCopy
POST /v1/events/adclick HTTP/1.1
Host: click.ads.aws.internal
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X)...
X-Forwarded-For: 203.0.113.195
Content-Type: application/json

{
  "click_id": "clk_88a91c74f0b21a",
  "ad_id": "ad_nike_airmax_2026",
  "campaign_id": "camp_nike_spring_global",
  "publisher_id": "pub_techcrunch_01",
  "user_id": "usr_998124a87",
  "cost_per_click_cents": 45,
  "impression_timestamp": 1718000000100,
  "click_timestamp": 1718000002340,
  "click_token_sig": "HMAC_SHA256(ad_id + timestamp + salt)"
}

Response: 202 Accepted

jsonCopy
{
  "status": "ACCEPTED",
  "redirect_url": "https://www.nike.com/air-max-2026?utm_source=ad_campaign"
}

2. Query Real-Time Campaign Performance

httpCopy
GET /v1/analytics/campaigns/camp_nike_spring_global/metrics?window=1h&start_time=1718000000&end_time=1718003600 HTTP/1.1
Host: dashboard.ads.aws.internal
Authorization: Bearer <jwt_token>

Response: 200 OK

jsonCopy
{
  "campaign_id": "camp_nike_spring_global",
  "window": "1_HOUR",
  "total_clicks": 142850,
  "total_conversions": 4285,
  "total_spend_cents": 6428250,
  "ctr_percentage": 2.85,
  "cvr_percentage": 3.00,
  "fraud_clicks_blocked": 1820
}

5. Data Models & Storage Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

DynamoDB Schema (AdAggregatesTable)