Design a Distributed Metrics Monitoring & Alerting System

1. Problem Statement & Scope Clarification

System Mission

Design an ultra-high-throughput, planetary-scale metrics monitoring, time-series storage, and alerting platform (similar to Prometheus/Thanos, Datadog, and Amazon CloudWatch) capable of ingesting tens of millions of metric points per second, executing real-time alert rules with sub-minute evaluation cycles, and delivering sub-second analytical dashboard queries over multi-year time horizons.

Functional Requirements

1. Metric Ingestion (PushMetric / BatchIngest): Ingest high-throughput, structured time-series metrics (metric_name, timestamp_epoch_sec, value_float64, tags/labels).
2. Flexible Tag-Based Query Engine: Support multi-dimensional label filtering (e.g., cpu.util{env="prod", region="us-east-1", service="payment"}) and mathematical rollups (avg, sum, rate, p99).
3. Real-Time Alerting Engine: Continuously evaluate user-defined threshold and anomaly rules over sliding windows, dispatching deduplicated alerts to PagerDuty, Slack, and webhooks.
4. Automated Downsampling & Tiered Retention: Progressively aggregate high-resolution 1-second metrics into 1-minute and 1-hour rollups to optimize storage costs over multi-year retentions.

Non-Functional Requirements (SLAs & SLOs)

• High Ingestion Throughput: Ingest $> 10,000,000\text{ metric points/sec}$ ($10\text{M points/sec}$ sustained, Peak: $30\text{M points/sec}$).
• Query Latency: Dashboard queries for recent data ($< 2\text{ hours}$): $P95 < 50\text{ ms}$; historical range queries ($30\text{ days}$): $P95 < 500\text{ ms}$.
• High Availability: $99.999\%$ uptime SLA for ingestion and alerting.
• Compression Efficiency: Achieve $\ge 90\%$ storage reduction using Gorilla XOR Floating-Point Compression.

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Ingestion Scale & Network Bandwidth

• Sustained Ingest Rate: $10,000,000\text{ data points/sec}$ ($10\text{M points/sec}$).
• Peak Ingest Rate ($3\times$ multiplier): $30,000,000\text{ points/sec}$.
• Raw Metric Point Size: $64\text{ bytes}$ (metric_name pointer, 8B timestamp, 8B float value, label pointers).
• Peak Raw Ingress Bandwidth: $\text{Bandwidth} = 30 \times 10^6\text{ points/sec} \times 64\text{ bytes} = 1.92\text{ GB/sec} \approx \mathbf{15.36\text{ Gbps}}$

Gorilla Compression & Storage Derivations

Using Facebook's Gorilla compression algorithm (Delta-of-delta timestamps + XOR float64 values):

• Average compressed size per point: $\approx 1.37\text{ bytes}$ ($> 97\%$ reduction).
• Compressed Daily Ingestion Volume: $\text{Daily Compressed Storage} = 10 \times 10^6\text{ points/sec} \times 86,400\text{ sec} \times 1.37\text{ bytes} \approx \mathbf{1.18\text{ TB/day}}$

Tiered Storage Footprint (1-Year Accumulation)

1. Hot Tier (In-Memory / NVMe SSD - 2 Hours): $10\text{M pts/sec} \times 7,200\text{ sec} \times 1.37\text{ bytes} \approx \mathbf{98.6\text{ GB RAM}}$
2. Warm Tier (ClickHouse / Timestream - 30 Days): $1.18\text{ TB/day} \times 30\text{ days} \approx \mathbf{35.4\text{ TB}}$
3. Cold Tier (Downsampled S3 Parquet Lakehouse - 1 Year): Downsampling raw 1s metrics to 1-minute rollups ($60\times$ reduction) + 1-hour rollups ($3600\times$ reduction): $\text{Annual Cold Storage} \approx \frac{1.18\text{ TB/day} \times 365\text{ days}}{60} \approx \mathbf{7.18\text{ TB/year}}$

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

4. Storage Engine Internals: The Gorilla Compression Algorithm

1. Delta-of-Delta Timestamp Encoding

Timestamps typically arrive at fixed intervals (e.g., every 10 seconds: $t_{-1} = 10, t_0 = 20, t_1 = 30$).

• First Delta: $D = t_1 - t_0 = 10$
• Delta-of-Delta: $D' = (t_1 - t_0) - (t_0 - t_{-1}) = 10 - 10 = 0$
• If $D' == 0$, encode as a single bit 0.
• If $-63 \le D' \le 64$, encode as bits '10' + 7 bits of $D'$.
• If $-255 \le D' \le 256$, encode as bits '110' + 9 bits of $D'$.
• Otherwise, store bits '1110' or '1111' with full integer value. Over $95\%$ of points encode into a single 0 bit.

2. XOR Floating-Point Value Compression

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

5. Time-Series Inverted Index & Label Posting Lists

To enable ultra-fast label queries like node.cpu{region="us-east-1", env="prod"}, the TSDB maintains an In-Memory Inverted Index utilizing Roaring Bitmaps:

textCopy
Label Index:
"__name__=node.cpu"  -> RoaringBitmap: [Series 1, Series 4, Series 9, Series 12]
"region=us-east-1"    -> RoaringBitmap: [Series 1, Series 2, Series 4, Series 15]
"env=prod"            -> RoaringBitmap: [Series 1, Series 4, Series 8]

Query: node.cpu AND region=us-east-1 AND env=prod
Result: Bitwise AND Intersection = Series 1, Series 4 (Executed in < 5 microseconds)