Design a Metrics Monitoring and Alerting System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is Metrics Monitoring?

You Already Know One: a Car's Dashboard

A car's dashboard has two kinds of things on it. Gauges show numbers that change over time: speed, engine temperature, fuel. Warning lights come on when something is wrong: low oil pressure, a brake fault. You glance at the gauges now and then. The warning lights must come on when, and only when, something really needs you.

A metrics monitoring system does the same for software. Every server and service reports numbers every few seconds: CPU use, requests per second, error counts, request latency. The system stores those numbers, draws them as graphs (the gauges), and runs alert rules that notify a person when something is wrong (the warning lights).

A series is named by a metric name plus labels (key-value pairs): http_requests_total{service="checkout", code="500", instance="web-17"} is one series. Change any label value and it is a different series.

What Makes It Hard

• The numbers never stop. Millions of samples arrive every second, forever. Almost all are written once and never changed.
• Queries span seconds to years. The on-call engineer wants the last five minutes, in detail. The capacity planner wants two years, in outline. One storage layout can't serve both cheaply.
• The number of series explodes. One careless label, such as a user ID, turns one series into millions. Storage cost and memory depend on how many distinct series exist, not only on how many samples arrive.
• False alarms are as bad as none. A team paged ten times a night for nothing stops reading pages, and then misses the real one.
• The monitor must outlive what it watches. An outage is exactly when people need their graphs and alerts. If our system fails with everything else, it fails when it matters most.

The Question the Whole Loop Answers

How do we store and query an endless stream of numbers cheaply, and raise an alarm people trust?

The answer gets sharper every round:

• Round 1: a time-series layout instead of a table of rows, a label index, scraping, alert rules that wait before firing, grouped notifications, and something outside the system that notices when the system itself dies.
• Round 2: scale: bit-level compression, a sharded and replicated in-memory tier fed by a stream, downsampled tiers for long ranges, defenses against series explosions and late data, and alert storms reduced to one page.
• Round 3: monitoring as a product: per-tenant limits and shuffle sharding, alerting that survives a region outage, alerts based on error budgets, controlled query costs, and a bill for every tenant.

---

Round 1 · Mid-level · "Monitor One Company's 500 Servers"

~35 min · SDE II (L5) · 1 region, 3 AZs · 500 servers, ~50K series · 5,000 samples/s · 30 days · alerting 99.9%

R1.1 Establish Design Scope

The interviewer says: "Our ops team runs about 500 servers and a handful of services. Today they find out about problems from customers. Design a system that collects metrics, shows them on dashboards, and pages the on-call engineer when something breaks." Before drawing anything, we ask questions and say what each answer changes.

Out of scope for this round:

• Years of history. 30 days is enough; long-range tiers come in Round 2.
• Defending against series explosions. 500 servers under one team's control won't invent millions of series by accident. Round 2 brings that back.
• Many teams and many tenants. One ops team owns everything.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into something the system does:

Not yet: multi-year retention, cardinality defenses, alert inhibition across teams.

R1.3 Non-Functional Requirements: the Questions

We state each quality in words first, in the order we'd defend it. The numbers come in R1.7.

• Alerting is the most important path. Dashboards can be slow for a minute; a missed page can't be undone. Alerting must survive one server or one AZ failing.
• Alerts must be fresh. Once a rule's condition has held long enough, the page should arrive within about a minute.
• The workload is write-heavy and append-only. 5,000 samples a second arrive all day, and nobody edits them. Reads are rare in comparison, and almost always "one series, a time range".
• Reads are by time range and labels. "CPU for every host in service=checkout over the last 6 hours" is the typical query.
• Monitoring must not share fate with what it watches. If the monitored servers or their network fail, we must still see it and page someone.

R1.4 The API

Everything in this round is internal. There are three contracts.

1. A sample. The logical data model every other part of the system speaks (shown as JSON for readability):

jsonCopy
{
  "name": "node_cpu_seconds_total",
  "labels": { "instance": "web-17:9100", "job": "node", "mode": "user", "cpu": "3" },
  "timestamp_ms": 1790510410000,
  "value": 81234.57
}

A counter like this one only goes up (seconds of CPU time used since boot), so a graph shows its rate per second, not its raw value. A gauge (memory in use, queue depth) is shown as it is.

With pull collection (step 1.3), this is what a host's agent serves when we ask it, in the plain-text format Prometheus-style agents use:

textCopy
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="3",mode="user"} 81234.57
node_cpu_seconds_total{cpu="3",mode="idle"} 1204521.02
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 3.1254528e+09

The collector adds instance and job labels and the timestamp when it reads the page.

2. The query endpoint. A range query takes an expression, a start, an end and a step (the gap between the points the graph gets back):

httpCopy
GET /api/v1/query_range?query=sum+by+(instance)(rate(node_cpu_seconds_total{job="node",mode!="idle"}[5m]))&start=1790488800&end=1790510400&step=60 HTTP/1.1
Host: metrics.internal.example.com

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      { "metric": { "instance": "web-17:9100" },
        "values": [ [1790488800, "2.91"], [1790488860, "3.04"], [1790488920, "2.87"] ] }
    ]
  }
}

The query is written in PromQL, the Prometheus query language (a query language, not program code). rate(x[5m]) means "the per-second increase of counter x, averaged over the last 5 minutes". Six hours at a 60-second step returns 360 points per series, which is about what a graph a few hundred pixels wide can show.

3. An alert rule (YAML, in the format Prometheus-style rule evaluators read):

yamlCopy
groups:
  - name: checkout
    interval: 15s
    rules:
      - alert: CheckoutHighErrorRatio
        expr: >
          sum(rate(http_requests_total{service="checkout", code=~"5.."}[5m]))
          / sum(rate(http_requests_total{service="checkout"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout 5xx ratio above 5% for 5 minutes"
          runbook: "https://wiki.example.com/runbooks/checkout-errors"

interval is how often the rule is evaluated. for is how long the condition must stay true before the alert fires (step 1.4). labels decide where it goes (step 1.5).

Recap

• Three contracts: a sample (name, labels, timestamp, value), a range query with a step, and an alert rule with a for duration.
• About 50K series every 10 seconds, kept 30 days, in one region.
• Alerting first, fresh within a minute, and independent of what it watches.

Let's build it, starting with the simplest version that works.

R1.5 Design Evolution: From a Table of Rows to Alerts People Trust

Every step follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

A relational database with one table. Every sample is one row: (metric_name, labels, timestamp, value), with an index on (metric_name, timestamp). A script on each server inserts its numbers every 10 seconds. Another script runs alert queries every minute and emails someone if a query returns rows.

Synthesizing vector architecture diagram...

Every sample is a row, and every question is a query against one big table.

What's good about it: it works on day one, and everyone knows SQL. What's wrong: 5,000 rows a second is 432 million rows a day. Each row repeats its name and labels (tens of bytes), and each index entry adds more. A row per sample costs roughly 50 to 100 bytes on disk once overheads and indexes are counted, against 16 bytes for the timestamp and value alone. And a graph of one host's CPU over a day has to find 8,640 rows scattered among 432 million.

Step 1.1: Storage and Range Queries Are Slow

Primitive: Write-Ahead Log & LSM-Trees

Synthesizing vector architecture diagram...

Writes are appends to the WAL and to open chunks in memory; every two hours the head becomes an immutable block, and old data leaves by dropping whole blocks.

Step 1.2: Find Every Series With service=checkout

Primitive: Trie Data Structure & Inverted Index

Step 1.3: Pull or Push?

Step 1.4: Alerts Fire and Clear Every Minute

Synthesizing vector architecture diagram...

An alert has to earn its way to Firing, and has to fall well below the fire threshold to leave it.

Step 1.5: Five Alerts for One Outage Wake the On-Call Five Times

Synthesizing vector architecture diagram...

Five alerts from two evaluators become one notification.

Step 1.6: The Monitoring Server Died and Nobody Noticed

Synthesizing vector architecture diagram...

The alarm fires on silence, so it catches every way the pipeline can die: the evaluator, Alertmanager, the network, or the notifier.

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get names.

Synthesizing vector architecture diagram...

Two identical Prometheus servers scrape everything and send every alert to all three Alertmanagers, which share state by gossip and send one notification. The watchdog path sits outside the stack.

Why these pieces:

• Two identical Prometheus servers in different AZs, each scraping every target and evaluating every rule. This is the standard way to make Prometheus highly available: there is no replication between them, just two full copies. If one dies, the other keeps collecting and alerting. Each copy may have small gaps the other doesn't, which is fine for monitoring.
• Three Alertmanagers, one per AZ, forming a cluster. Each Prometheus sends every alert to all of them (not through a load balancer). They share silences and a notification log (which notifications were sent) by gossip. Before sending, each instance waits according to its position in the cluster (position × --cluster.peer-timeout, 15 s by default), so the first one sends and the others see it in the shared log and stay quiet. If gossip breaks, they may each send: a duplicate page, never a missing one.
• Grafana draws dashboards from Prometheus 1, falling back to Prometheus 2. Dashboards are kept as files in version control, so the Grafana server itself holds nothing we can't rebuild.
• EC2 service discovery gives Prometheus the list of instances to scrape (every instance tagged monitoring=on), so new servers are picked up without editing a file, and a missing one shows as up == 0.
• The watchdog (step 1.6) runs in Lambda, CloudWatch and SNS: AWS services that share no servers, disks or configuration with our stack.

Tracing the three flows

1. A sample from scrape to disk

1. At 10:00:10, Prometheus 1 fetches http://web-17:9100/metrics (about 100 series from this host, a few KB of text).
2. It parses each line, adds job="node" and instance="web-17:9100", and stamps every sample with the scrape time.
3. Each label set is looked up in the head's index. A known one gives its series ID; a new one gets a new ID and new posting-list entries.
4. The samples are appended to the WAL, then to each series' open chunk in memory. Within milliseconds they can be queried.
5. At the next 2-hour boundary, the head is cut into a block on disk and the old WAL segments are removed.

2. A dashboard query

1. Grafana asks for rate(node_cpu_seconds_total{instance="web-17:9100", mode!="idle"}[5m]) over 24 hours at a 60 s step.
2. The index intersects the posting lists: 7 non-idle modes × the host's CPU cores, since node_cpu_seconds_total has a cpu label: 14 series on a 2-vCPU host (summed per host later).
3. Prometheus reads those series' chunks from 12 two-hour blocks and the head: 14 × 8,640 ≈ 121,000 samples, decoded in well under a second.
4. It computes the rate at each of the 1,440 steps and returns the points.

3. An alert from breach to page

Synthesizing vector architecture diagram...

The page arrives about 7 minutes 40 seconds after the error started: about 2 minutes for the 5-minute rate window to carry the ratio past 5% (5 ÷ 12 × 5 min ≈ 2 min 5 s), the 5-minute for, and the 30-second group wait.

R1.7 Numbers

Targets

Series and samples

The 80 series per host is an assumption: a host agent with every collector turned on emits several hundred series per host (CPU time alone is cores × 8 modes). Choosing what to collect is the first cardinality decision, even at this size.

Bytes

Alert freshness, step by step. These steps happen one after another, so the worst cases add up:

A group that is already firing adds new alerts at the next group_interval (up to 5 minutes), which is why each team gets its own group.

This budget starts when the rule's expression becomes true, which is not the moment the problem starts. An expression over rate(...[5m]) lags the event: a jump from ~0% to 12% errors lifts the 5-minute ratio past 5% only after about (5 ÷ 12) × 5 min ≈ 2 min (the trace in R1.6). Shorter rate windows react faster but are noisier.

Rough monthly cost, three options (us-east-1 list prices; check the AWS Pricing Calculator before quoting)

Three lessons:

• CloudWatch prices per metric, and to CloudWatch every series is a metric. It's excellent for AWS services' own metrics and a few hundred custom ones, and very expensive for 50,000 series. The price of a monitoring system follows its series count.
• Managed Prometheus costs about $600 a month more here, in return for no servers to patch, no disks to fill (the cause of step 1.6's outage) and no upgrades. For a small ops team, that's a few engineer-hours a month, and it can easily be worth it. Its minimum rule evaluation interval is 30 s (a service quota), which adds 15 s to our freshness budget.
• We choose self-run for this round because it's cheap, it teaches every moving part, and the team already runs servers. We'd revisit it the first time the monitoring stack causes an incident.

R1.8 Trade-Offs

Pull vs push, for this scope

Self-run vs managed

Static thresholds vs rates and ratios

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Explains why a table of rows is the wrong shape, and what a time-series layout does instead (series IDs, time-ordered compressed chunks, WAL, blocks).
• Finds series with an inverted index and intersects posting lists.
• Compares pull and push on down detection, not just on "scale".
• Uses for, hysteresis and ratios to stop flapping, and knows the cost is slower alerts.
• Groups and deduplicates alerts before they reach a person.
• Asks "who watches the watcher?" without being prompted.

Follow-up questions

1. "Your two Prometheus copies disagree slightly. Is that a problem?" Answer: no. They scrape at slightly different moments, so values differ in the last digits, and one may have a gap the other doesn't. Dashboards read one copy (with fallback); alerts come from both and are deduplicated by labels in Alertmanager. If exact agreement mattered (billing, say), metrics would be the wrong tool.

2. "A batch job runs for 20 seconds every night. How do you know if it failed?" Answer: it pushes its result (last_success_timestamp, duration, records processed) to the push gateway before it exits, and we alert on time() - last_success_timestamp > 26 * 3600: "no success in 26 hours". That catches both a failed run and a run that never started, which an "errors > 0" rule would miss.

3. "Why does the watchdog go through CloudWatch? Couldn't the second Prometheus watch the first?" Answer: it can, and we do that too. But both copies share the same Alertmanager cluster, the same pager integration and often the same configuration mistake. The dead man's switch tests the whole path end to end, from somewhere that shares none of it.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "10M Points a Second and 100M Series"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 100M active series · 10M samples/s sustained, 30M/s peak · 5-year history · ingest and alerting 99.95%, survives an AZ loss

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We monitor 500 servers and ten services: about 50,000 series sampled every 10 seconds, 5,000 samples a second, kept 30 days. Two identical Prometheus servers in different AZs scrape every target, found through EC2 service discovery, and store samples in a time-series layout: each series gets an ID, its samples go into compressed, time-ordered chunks, new data lands in a WAL and an in-memory head, and every two hours the head becomes an immutable block. An inverted index of posting lists finds series by label. Rules run every 15 seconds with for durations and hysteresis, on ratios rather than counts. Both servers send alerts to a three-node Alertmanager cluster that deduplicates, groups by team and service, waits 30 seconds, and routes pages to the pager and email. A Watchdog alert feeds a CloudWatch dead man's switch outside the stack. About $200 a month. Open costs: everything lives on one server's disk and memory, there's nothing older than 30 days, and nothing stops a team from creating a million series."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: two full copies of a single-node time-series database, one alert router cluster, and a watcher outside.

Round 1 step summary

Open costs: one server's memory and disk hold everything; no long-range history; no defense against series explosions or late data; an outage still produces many alerts across teams.

R2.1 The Scope Raise

Interviewer: "It worked, and now every team wants it. It becomes the company's monitoring platform: every service, container and batch job. We expect 10 million samples a second, bursting to 30 million. Capacity planners want five years of history, and a five-year dashboard must still load in about a second. Last month a team added user_id as a label on one metric and our test cluster fell over. Mobile apps and batch jobs send data late. In our last outage, 2,000 alerts fired. And the platform must survive losing an AZ."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

R2.2 What Breaks in the Round 1 Design

We fix them in this order: bytes per sample (2.1), sharding and replication (2.2), long ranges (2.3), cardinality (2.4), late data (2.5) and alert storms (2.6).

R2.3 New Requirements and API Additions

1. Batch ingest. Agents on every host (and in every Kubernetes pod network) scrape their local targets and push batches to the platform, using the Prometheus remote write protocol: a protobuf message, compressed with Snappy, sent by HTTP POST. It carries many series, each with its labels once and many samples.

httpCopy
POST /api/v1/push HTTP/1.1
Host: ingest.metrics.internal.example.com
Authorization: Bearer <team token>
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Prometheus-Remote-Write-Version: 0.1.0

httpCopy
HTTP/1.1 200 OK

A 200 means durably accepted: the batch is in the stream (step 2.2). A 429 means the team is over its rate limit (retry later); a 400 means the batch was rejected (bad labels, timestamps too far in the future), and the body says why.

2. A query with a resolution hint. Same PromQL API as Round 1, plus a hint:

httpCopy
GET /api/v1/query_range?query=sum(rate(http_requests_total{service="checkout"}[1h]))&start=1632744000&end=1790510400&step=86400&resolution=auto HTTP/1.1

resolution is auto (the default: pick the coarsest tier that still gives at least one point per step), or raw, 5m or 1h to force a tier. The response says which tier answered, in a resolution_used field, so a graph can label itself "hourly data".

3. Per-team limits, set by the platform team and readable by each team:

jsonCopy
{
  "team": "payments",
  "max_active_series": 2000000,
  "max_series_per_metric": 200000,
  "ingestion_rate_samples_per_s": 200000,
  "ingestion_burst_samples": 2000000,
  "max_label_names_per_series": 30,
  "max_label_value_length": 1024,
  "drop_labels": ["user_id", "session_id", "request_id", "email"],
  "out_of_order_window_s": 600
}

4. Silences and inhibition. A silence mutes matching alerts for a fixed time (planned maintenance). It is created through the Alertmanager API:

jsonCopy
{
  "matchers": [ { "name": "cluster", "value": "payments-db-2", "isRegex": false, "isEqual": true } ],
  "startsAt": "2026-09-27T22:00:00Z",
  "endsAt": "2026-09-28T00:00:00Z",
  "createdBy": "maria@example.com",
  "comment": "Planned failover drill, CHG-4821"
}

An inhibition rule mutes alerts while another alert is firing (step 2.6):

yamlCopy
inhibit_rules:
  - source_matchers: ['alertname="AvailabilityZoneImpaired"']
    target_matchers: ['severity="cause"']
    equal: ['availability_zone']

R2.4 Design Evolution: Scaling the Store, Taming the Alerts

Step 2.1: Raw Storage Is 16 Bytes Per Sample, 10 Million Times a Second

Step 2.2: One Node Can't Take 10 Million Samples a Second

Synthesizing vector architecture diagram...

Two consumer groups read one replicated stream, so the head exists twice and the stream holds every acknowledged sample for 24 hours.

Primitive: Message Queues vs Event Streams

Synthesizing vector architecture diagram...

A lost ingester costs a replica, never data: its twin answers while the replacement replays the stream from the last checkpoint.

Step 2.3: A Five-Year Graph Scans Billions of Samples

Synthesizing vector architecture diagram...

Every range reads the coarsest tier that still gives one point per step.

Step 2.4: Someone Put user_id in a Label

Synthesizing vector architecture diagram...

Cheap checks reject bad batches before the stream; the exact series limit is enforced where the series live.

Step 2.5: Late and Out-of-Order Samples Are Rejected

Step 2.6: One Outage Produced 2,000 Pages

Synthesizing vector architecture diagram...

Symptoms page, causes explain, and the shared cause pages the one team that can fix it.

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Writes flow left to right into a replicated stream; two ingester groups hold the recent hours; blocks in S3 hold everything older; one read path serves dashboards and rule evaluation alike.

Why these pieces:

• Agents at the edge. Each host (or Kubernetes node) runs an agent that scrapes its local targets every 10 s and pushes batches to us. This keeps what pull was good at (the agent knows its targets and records up) while no central scraper has to reach millions of endpoints. The agent keeps a local buffer on disk and retries with backoff, so a short outage on our side delays data rather than losing it.
• An internal Network Load Balancer spreads long-lived HTTPS connections from agents over the gateways. Teams in other accounts reach it through PrivateLink.
• Ingest gateways are stateless: authenticate the team, apply label rules and rate limits, hash, route by age, and write to Kinesis. They scale with CPU.
• Kinesis Data Streams is the durable buffer and write-ahead log (step 2.2). Provisioned with 600 shards (R2.6).
• Ingesters, 25 per group, two groups (A in AZ a, B in AZ b), each holding every series' last few hours, plus local disk for building blocks.
• S3 holds all blocks: raw, 5-minute and 1-hour. S3 is designed for 99.999999999% (11 nines) object durability; that's a property of the stored blocks, not a promise about our whole pipeline.
• The compactor merges the two replicas' blocks into one, merges 2-hour blocks into 24-hour blocks (fewer, larger objects), builds rollups, merges backfilled blocks, and deletes data past each tier's retention.
• Store gateways serve queries on blocks in S3. They keep a small index header of each block on local disk and fetch only the chunks a query needs, with a chunk cache.
• The query frontend splits long queries into one-day pieces run in parallel, caches the results of past days (keyed with a cache generation that backfill bumps), and picks the tier (step 2.3).
• Rule evaluators run every team's rules through the same queriers, sharded so each rule group is evaluated by exactly one evaluator at a time, and send alerts to a three-node Alertmanager cluster (one per AZ).
• ElastiCache for Valkey holds query results and chunks.

The data model

A block is a directory of objects in S3 under blocks/<team>/<block-id>/:

Keeping blocks per team makes retention, deletion and cost attribution a matter of prefixes. Round 3 turns teams into paying tenants.

The ingester's lease and checkpoint table (DynamoDB, managed by the Kinesis Client Library):

The lease is decided by a counter the owner keeps incrementing, not by comparing clocks, which fits DynamoDB: its condition expressions have no server-side "now", so the library compares counters it has seen, on its own schedule.

Trace 1: ingest

Synthesizing vector architecture diagram...

The agent's 200 means the stream has the samples; each ingester group builds its own copy of the head from the same records.

Trace 2: a 30-day query

1. Grafana asks for sum by (service) (rate(http_requests_total{team="payments"}[1h])) over 30 days at a 15-minute step.
2. The query frontend picks the 5-minute tier (a step of at least 5 minutes but under an hour), splits the range into 30 one-day pieces, and finds 29 of them in the results cache. A cached day can go stale when the backfill path (step 2.5) adds late samples to it, so every backfilled block bumps a per-team cache generation for the days it covers; the generation is part of the cache key, and affected days are recomputed.
3. Only today's piece runs: a querier asks the store gateways for today's older hours and both ingester groups for the last few hours, merges the replicas, and computes.
4. The frontend stitches 29 cached days and 1 fresh one and returns 2,880 points per service.

Trace 3: an alert storm reduced to pages

1. 14:02: packet loss in AZ b. Latency and error rules start going pending across 150 services.
2. 14:04: AvailabilityZoneImpaired{availability_zone="b"} (for: 2m) fires and pages the network on-call.
3. 14:05 onward: ~1,970 severity="cause" alerts fire and are inhibited: recorded, not sent.
4. ~30 symptom alerts fire; Alertmanager groups them by alertname and availability_zone, and each affected team gets one page naming AZ b.
5. 14:24: the network recovers; the source alert resolves, the inhibition lifts, and everything still firing is sent normally. Nothing is.

R2.6 Numbers and Cost

Targets

How we measure the first line matters. Kinesis Data Streams' own SLA is 99.9% a month; an SLA is a credit commitment, not a measurement, but it's a warning not to claim more for a path that runs through one stream. Agents buffer and retry, so a short stream problem delays samples instead of losing them; we count ingest as available when samples land within 5 minutes of being sent.

We don't promise "five nines" (99.999%, which a common textbook design claims) for this round: one region can't honestly offer it, and we haven't built anything that survives a region. Round 3 raises the target with the design to back it.

Rates and bytes

Memory: the data floor vs a real engine. The two lines above say the recent data is about 119 GB per replica. That's the floor: compressed bytes plus the index. A real engine spends far more per series: an open chunk per series, hash maps, per-series bookkeeping, label strings, query buffers. Grafana Mimir's capacity guide says to plan 2.5 GB of memory and 1 CPU core per 300,000 in-memory series (minimums, with 50% extra recommended):

A common textbook design sizes the hot tier from the data floor (three r6i.4xlarge, unreplicated). Planned from the engine's real per-series cost, the head needs about seven times the memory the floor suggests. This is the single most important number in the round, and it scales with series, not samples.

Stream

The replay estimate is why we keep two replicas: 33 minutes without recent data for 4% of all series would break the alerting target.

Gateways

We don't keep 1,200 cores idle for a burst that comes from agents catching up: agents buffer, so a few minutes of delay is the cheaper answer.

Query latency budgets. A recent query fans out to all 50 ingesters (both groups) and waits for all of them. The steps run in order, so they add up; the fan-out is parallel, so it costs the slowest of the 50 calls:

The catch is the fan-out's tail. For the slowest of 50 calls to be under 25 ms 95% of the time, each call must be under 25 ms with probability 0.95^(1/50) ≈ 0.999: each ingester's P99.9, not its P95, must fit. That's a real requirement on ingester garbage collection and noisy neighbors, and a reason Round 3 shrinks each tenant's fan-out.

A 30-day query runs its one-day pieces in parallel too: with 29 of 30 days cached (trace 2), usually only one piece runs, taking about 20 ms for index lookups plus chunk fetches (a few ms from the cache; tens to a couple of hundred ms from S3 on a miss) and decoding. We budget 400 ms for the slowest piece and 50 ms for merging: under 500 ms, and cold-cache queries across all 30 days are what we load-test.

Storage tiers, recomputed (rollup values assumed at ~2 B each, since they compress worse than raw)

(Rounding 1.18 TB × 30 gives the 35.4 TB often quoted; unrounded it's 35.5.) The 5-minute tier is the biggest: half a year of four aggregates. Storage here is written per day and kept for a fixed number of days, so each tier's size is simply writes per day × days kept. Before the compactor merges them, both replicas' raw blocks exist for a day or so, adding about 1.2 TB.

Rough monthly cost (us-east-1 list prices, on-demand; check the AWS Pricing Calculator before quoting)

Synthesizing vector architecture diagram...

Memory for 100M series is the biggest line; storage, which people worry about first, is among the smallest.

That's about $760 per million active series per month. Four lessons:

• Series drive the bill. The ingesters are a third of it, and they're sized by series count. Every limit in step 2.4 is also a cost control.
• Storage is cheap once it's compressed and tiered: 84 TB for about $2,400 a month.
• The network path matters. Interface endpoints cost $11,600 a month here; putting the same traffic through a NAT gateway would cost $0.045/GB, about $57,000. The 16-bytes-per-sample assumption drives both lines, so we measure it first.
• Managed would cost far more at this size. Amazon Managed Service for Prometheus at 26.28 trillion samples a month: the first 2B at $0.90 per 10M, the next 250B at $0.35 per 10M and the rest at $0.16 per 10M (the published tiers we checked) come to about $425,000 a month before queries, and its default quotas (1.67M samples/s per workspace) would need raising. At Round 1's size it was the easy choice; here building saves several times its cost in engineers.

R2.7 Trade-Offs

How to replicate the hot tier

We pay 2× memory because 30 minutes of blind alerting on 4% of series is not acceptable, and we skip the third copy because the stream already guarantees durability.

How many aggregates to keep in rollups

Kinesis vs a Kafka cluster (Amazon MSK) for the stream

Build on open source vs managed

R2.8 Failure Modes

Go deeper: why an LSM engine stalls writes when compaction falls behind, and why we don't. In an LSM-tree engine (RocksDB, Cassandra), writes go to a WAL and an in-memory table; full tables are flushed to small sorted files at level 0, and compaction merges them into larger levels. Every level-0 file must be checked on reads, so if compaction can't keep up with the flush rate, level-0 files pile up. To keep reads bounded and memory from running out, the engine deliberately slows, then stops, foreground writes when level-0 files pass a threshold: a write stall. Compaction itself rewrites data many times over (write amplification), so under a burst it competes with flushes for disk bandwidth. Our design avoids the stall by not making flushes wait for compaction: an ingester's 2-hour block goes straight to S3 as a finished, immutable object, and the compactor works on S3 later. The price is paid on reads instead, which is why we alarm on compaction lag. Why not a B-tree engine instead? A B-tree gives predictable reads with no background merging, but it updates pages in place: 10M samples a second spread over 100M series would be random writes all over the tree, far beyond what disks sustain, where an append-only layout writes sequentially. Time-series engines accept background merging to get sequential writes (step 1.1).

R2.9 Production Gotchas

1. One central scraper for everything

• Symptom: scrapes run late, timestamps drift, targets flap to up == 0 because a scrape timed out, not because the target died.
• Cause: one process connecting to millions of endpoints every 10 seconds.
• Fix: agents scrape locally and push (R2.5); the central platform never connects to targets.

2. Querying raw data for months

• Symptom: a 90-day dashboard times out and slows every other query.
• Cause: 777,600 raw samples per series for 90 days, decoded for a graph 800 pixels wide.
• Fix: tiers chosen by step (step 2.3); a per-query limit on samples read.

3. No hysteresis or for

• Symptom: pages that resolve a minute later, dozens a night.
• Cause: firing on every evaluation where the condition is true.
• Fix: for, separate clear thresholds, symptom-based paging (steps 1.4, 2.6).

4. Unbounded label values

• Symptom: memory grows steadily; one day every ingester runs out.
• Cause: IDs, emails, URLs with query strings, or error messages as label values.
• Fix: platform-wide forbidden labels, value-length limits, per-team series limits, cardinality alerts (step 2.4). Put per-request detail in logs or traces, not metrics.

5. Averaging percentiles

• Symptom: the "p99" on the yearly graph is far lower than any incident review remembers.
• Cause: averaging per-host or per-hour p99 values.
• Fix: record histograms, sum their buckets, compute the percentile last (step 2.3).

R2.10 Pillar Check

Security in detail. Each team's token maps to a team ID at the gateway; the team ID is never taken from a request header. Gateways may only put records to the stream and write to the late-data prefix. Ingesters may read the stream, write to the lease table and write blocks under blocks/; they can't delete. Only the compactor may delete blocks. Store gateways and queriers are read-only on S3. Blocks are encrypted with SSE-KMS, and all traffic uses TLS.

Operations in detail.

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer adds over L5

• Encodes a few samples with Gorilla by hand, and cites 1.37 bytes as one paper's workload, not a law.
• Knows that series, not samples, drive memory and cost, and sizes memory from the engine's per-series cost, not from compressed bytes.
• Keeps the hot tier durable and available through node and AZ loss, and explains what a node loss does to the last two hours.
• Recomputes rollup savings with the real interval and the number of aggregates.
• Defends against cardinality at the gateway and where the series live.
• Handles late data without corrupting chunks, and turns an alert storm into a few pages.

Follow-up questions

1. "Why not just add a third ingester group instead of the stream?" Answer: three groups written synchronously (a quorum of two) is a valid design and removes the stream bill, but then a burst must be absorbed by the ingesters themselves, and a slow ingester slows writes. With the stream, the gateway's 200 depends only on Kinesis, and ingesters can lag, restart or be replaced without affecting agents. We'd reconsider if the stream's cost (about $18,500 a month with its endpoints) grew faster than its value.

2. "A team says its alerts didn't fire during last night's incident. Where do you look?" Answer: in order: was the data there (discarded-samples counter for their team, consumer lag); did the rule evaluate (missed iterations, evaluation errors, and whether a label change made its selector match nothing, which looks like "all fine"); did it fire but get muted (inhibition or a silence); was it sent (notification failures). "No data" and "no problem" look the same to a threshold rule, which is why paging rules on important services also carry an absent() companion.

3. "The 16 bytes per sample on the stream turns out to be 40. What changes?" Answer: stream bytes grow 2.5×: 1,200 MB/s at peak needs 1,500 shards, which adds about $9,900 of shard-hours, and the endpoint traffic grows to about 3.15 PB (about $22,900 instead of $11,600, since volume beyond the first PB is cheaper). Ingesters, S3 and memory don't change: they depend on series and on the compressed format. We'd first try larger batches (labels sent once per series per batch) before paying for it.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Monitoring as a Product: Tenants, Regions, SLOs"

~45 min · Principal (L7) · 3 home regions of cells + 3 alerting standby regions · ~1B active series · ~100M samples/s · 99.99% alerting per region, through a region outage · one tenant can't slow another

R3.0 Where We Left Off

What the candidate says in the first 60 seconds of Round 3, and everything you need if you start here.

Round 2 in 60 seconds. "We run the company's monitoring platform: 100 million active series, 10 million samples a second, bursting to 30 million, in one region. Agents scrape locally and push with remote write. Stateless gateways enforce each team's label rules and rate limits, hash each series, and write to a 600-shard Kinesis stream, which is our replicated write-ahead log; a 200 means the stream has the data. Two groups of 25 ingesters, in two AZs, each read every shard and hold the last few hours in Gorilla-compressed memory, about 1.37 bytes a sample on the paper's workload but about 8 KB of memory per series in a real engine, which makes memory the biggest line of the bill. Every two hours they upload blocks to S3 and only then checkpoint. A compactor merges replicas and builds 5-minute and 1-hour rollups of min, max, sum and count; queries pick a tier by step. Per-team series limits are enforced at the ingesters; late data within 10 minutes goes into out-of-order chunks, older data through a backfill path. Symptom-based paging, inhibition and grouping turned 2,000 alerts into a handful of pages. About $76K a month, $760 per million active series. Open costs: one region, limits built for friendly teams, no way to bill anyone, and alerts based on thresholds that page too much or too late."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: a stream in the middle, two copies of the recent hours, blocks in S3, one read path for dashboards and rules.

Rounds 1–2 step summary

Open costs: a single region, so a region outage silences every alert; limits and query capacity shared by everyone; no idea what any team costs; threshold alerts that page on noise and miss slow burns.

R3.1 The Scope Raise

Interviewer: "The platform is good enough that we're going to sell it. Other companies will send us their metrics and rely on our alerts. Tenants range from a startup with a few hundred series to an enterprise with 10 million. One customer's giant query must never slow another's dashboard. Customers told us plainly: the moment they need alerts most is when a cloud region is having a bad day, so our alerting has to keep working through a region outage, including one of ours. They also want alerts based on SLOs, not just thresholds. And finance wants to know what each tenant costs us, so we can price it."

We ask back, and say what each answer changes.

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

1. Tenant identity on every call. Every write and query carries an API key. The gateway and the query frontend map it to a tenant ID; a tenant ID sent by the client in a header is ignored. Everything after that point (stream records, blocks in S3, cache keys, usage records) carries the tenant ID.

httpCopy
POST /api/v1/push HTTP/1.1
Host: ingest.us-east-1.metrics.example.com
Authorization: Bearer mk_live_7Hq...
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Prometheus-Remote-Write-Version: 0.1.0

2. Per-tenant limits, set by the tenant's plan and overridable by us:

jsonCopy
{
  "tenant_id": "t-4821",
  "plan": "business",
  "home_region": "us-east-1",
  "cell": "use1-cell-3",
  "max_active_series": 5000000,
  "ingestion_rate_samples_per_s": 500000,
  "ingestion_burst_samples": 5000000,
  "ingester_shard_size": 5,
  "querier_shard_size": 4,
  "max_samples_per_query": 50000000,
  "max_series_per_query": 100000,
  "max_query_length_days_raw": 32,
  "query_timeout_s": 120,
  "max_outstanding_queries": 100,
  "retention_days": { "raw": 30, "5m": 180, "1h": 1825 }
}

3. An SLO definition (YAML), from which we generate recording and alerting rules (step 3.3):

yamlCopy
slo:
  name: checkout-availability
  tenant_id: t-4821
  objective: 0.999
  window: 30d
  sli:
    total: 'http_requests_total{service="checkout"}'
    errors: 'http_requests_total{service="checkout", code=~"5.."}'
  alerting:
    page:
      - { long: 1h, short: 5m, burn_rate: 14.4 }
      - { long: 6h, short: 30m, burn_rate: 6 }
    ticket:
      - { long: 3d, short: 6h, burn_rate: 1 }
  notify: { page: oncall-pager, ticket: team-tickets }

4. Usage export, one record per tenant, meter and hour:

jsonCopy
{
  "tenant_id": "t-4821",
  "hour": "2026-09-27T14:00:00Z",
  "active_series_avg": 3814220,
  "samples_ingested": 1373119200,
  "samples_discarded": 0,
  "bytes_stored_gb": 2118.4,
  "query_samples_processed": 918442000123,
  "source_version": "meter-v7"
}

R3.4 Design Evolution: Running Monitoring for Others

Step 3.1: One Tenant's Cardinality and Queries Hurt Everyone

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance Patterns · Drill: One customer, one shard, one outage (the cross-tenant report question is answered in step 3.5)

Synthesizing vector architecture diagram...

t-9 can exhaust q07, q12, q30 and q41. t-4821 shares only q12, so three of its four queriers are untouched.

Step 3.2: Alerting Must Survive Our Own Region Outage

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active

Synthesizing vector architecture diagram...

Two independent paths feed one Alertmanager cluster that spans the pair; either region alone can still page.

Step 3.3: Threshold Alerts Page Too Often, or Too Late

Step 3.4: Some Queries Scan Months Across Millions of Series

Synthesizing vector architecture diagram...

Most of a long dashboard comes from cache; what's left waits in the tenant's own queue and runs only on its own queriers.

Step 3.5: What Does Each Tenant Cost Us?

Synthesizing vector architecture diagram...

Each meter is written once per tenant per hour, so replays overwrite rather than double-count, and two independent meters check each other.

Step 3.6: Should We Build This at All?

Further reading: the trade-off analysis playbook for how to present a build-versus-buy decision.

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each geography has a home region of cells and a standby region for alerting; configuration is written once and read locally everywhere; usage flows to one warehouse. The Europe and Asia-Pacific pairs have the same Alertmanager layout as North America (not drawn).

Why these pieces:

• Cells are Round 2's stack, unchanged, sized to 100M series. Growing means adding cells, never making one bigger, so our largest failure stays the size we have tested.
• Cell routers are thin: they read the tenant → cell table and forward. They hold no state that can't be reloaded.
• Standby cells hold only alert-referenced series for 8 hours, with no dashboards and no long-term storage.
• DynamoDB global tables carry tenants, limits, rules and SLOs, written in one control region (us-east-1) and read locally everywhere.
• Route 53 health checks with failover records send the customer-facing alert API (silences, acknowledgments) to the standby region when the home region is unhealthy. The records and health checks are set up in advance, because Route 53's data plane (answering queries and running health checks) keeps working when changing records might not.
• The outside watcher runs in a region that isn't part of the pair it watches, on CloudWatch and SNS.

Trace 1: a tenant query throttled

Synthesizing vector architecture diagram...

The noisy tenant waits in its own queue and on its own queriers; the other tenant's single query is taken on the next round-robin turn.

Trace 2: a region outage with alerts still firing

Synthesizing vector architecture diagram...

The standby never needed a failover: it was receiving and evaluating all along. Dashboards for us-east-1 tenants are down until 15:30; pages are not.

Trace 3: an SLO burn alert

1. 09:00: tenant t-4821 deploys a bad config; 3% of checkout requests fail (burn rate 30).
2. Recording rules update the 5-minute and 1-hour error ratios every minute.
3. 09:29: the 1-hour ratio reaches 0.03 × 29 ÷ 60 ≈ 1.45%, above 1.44%, and the 5-minute ratio is 3%. The fast-burn page fires; about 2% of the month's budget is spent.
4. 09:40: the deploy is rolled back. The 5-minute ratio falls below 1.44% by 09:45, and the alert resolves, although the 1-hour ratio is still high.
5. The page's annotation shows the budget left for the month: 40 minutes at 3% errors spent 0.03 × 40 ÷ 43.2 ≈ 2.8% of it, so 97.2% remains.

R3.6 Numbers and Cost

Targets

Why two paths: each path depends on one region's stream, and Kinesis's SLA is 99.9% a month. Two paths that fail independently are unavailable together far more rarely than either alone. They aren't perfectly independent (the same code, the same agent, the same customer), which is why we still test region evacuation (R3.9) instead of trusting the multiplication.

Per region

The per-cell numbers are Round 2's. Per-tenant retention plans change the storage column: a tenant on 13 months of 5-minute data costs more than one on 180 days, and the compactor's per-tenant bytes meter bills it. Blocks are stored per tenant, so a plan change is a retention setting on a prefix.

Standby sizing. The subset is 10% of samples (10M/s) and 10% of series (100M), about one Round 2 cell's worth spread over three regions: 50 ingesters (two groups), gateways for 10M samples/s, 600 shards in total, rule evaluators and Alertmanagers.

Forwarding vs dual sending (the alternative in step 3.2)

Asia-Pacific carries a fifth of the data and more than half of the transfer bill. With dual sending, none of this is charged to us.

Rough monthly cost (us-east-1 list prices used everywhere, which understates the Asia-Pacific and Frankfurt regions, where instances cost more; check the calculator)

That's about $830 per million active series per month, against $760 in Round 2. The extra $70 buys a standby for alerting, per-tenant isolation and billing. The home cells are nine-tenths of the bill, and in each cell the ingesters (memory for series) are the biggest line, so the lever that matters most is still cardinality: every tenant limit, forbidden label and cardinality alert is also a cost control. Steady fleets like ingesters are also the right place for Savings Plans, which lower their price in return for a commitment.

R3.7 Trade-Offs

Shuffle-shard size

Alerting in the same region vs a standby region

Burn-rate windows

Build vs buy (COST 11): step 3.6. At 1B series, building saves about $3.4M a month in list prices against managed ingestion alone, and it's the product we sell; at 50K series, managed would have been the sensible default.

Closing the loop. Round 1 asked "how do we store numbers and raise an alarm?" and answered with a time-series layout, for and hysteresis, grouping, and a dead man's switch. Round 2 asked "how do we do it for 100M series?" and answered with compression, a replicated stream-fed head, tiers and cardinality limits. Round 3 asked "how do we do it for strangers, through our own outages, and get paid?" and answered with cells, shuffle sharding, an alerting standby fed straight from the agents, burn-rate alerts and metering. The dead man's switch from Round 1 never went away: it became a canary tenant per cell and a heartbeat per tenant, watched from outside.

R3.8 Failure Modes

R3.9 Runbook

Signals for the monitor itself, per cell OPS 8 · REL 6

Procedure: a tenant's cardinality spike

1. Confirm in the tenant's cardinality report which metric and label grew (top-10 labels by new series in the last hour).
2. If the tenant's own limits are containing it (new series dropped, other tenants' ingesters normal), notify the tenant and stop there.
3. If its ingesters are above 80% memory, add a drop rule for the label to the tenant's limits (command below). Gateways pick up the change within a minute; its series stop growing.
4. Watch the tenant's ingesters' memory fall as those series go stale and the next head cut drops them.
5. Tell the tenant what was dropped and how to add the label back safely (as a log field or trace attribute).

Procedure: stream backlog

1. Check iterator age per shard group: all shards (ingesters too slow) or a few (hot shards).
2. All shards: are ingesters at their memory stop? Add ingesters to the group; each takes over whole shard ranges and replays them.
3. A few shards: a tenant sending far above its rate through few series? Lower its rate limit. Otherwise raise the shard count (below); one call can at most double it, and the number of calls per day is limited, so raise it well above need.

Commands (AWS CLI; IDs and names are placeholders; check them before running)

textCopy
aws cloudwatch get-metric-statistics --region us-east-1 --namespace AWS/Kinesis --metric-name GetRecords.IteratorAgeMilliseconds --dimensions Name=StreamName,Value=metrics-use1-cell3 --start-time 2026-09-27T10:00:00Z --end-time 2026-09-27T10:30:00Z --period 60 --statistics Maximum

aws kinesis describe-stream-summary --region us-east-1 --stream-name metrics-use1-cell3

aws kinesis update-shard-count --region us-east-1 --stream-name metrics-use1-cell3 --target-shard-count 900 --scaling-type UNIFORM_SCALING

aws dynamodb update-item --region us-east-1 --table-name tenant-limits --key '{"tenant_id":{"S":"t-9"}}' --update-expression "SET drop_labels = list_append(drop_labels, :l)" --expression-attribute-values '{":l":{"L":[{"S":"user_id"}]}}'

aws route53 get-health-check-status --health-check-id 1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d

aws appconfig start-deployment --application-id a1b2c3d --environment-id e4f5g6h --configuration-profile-id p7q8r9s --configuration-version 41 --deployment-strategy-id AppConfig.AllAtOnce

The first shows how far behind the slowest consumer of a cell's stream is. The fourth adds a drop rule in the control region, from where the global table carries it to every region. The last is the rule-pack rollback: it redeploys the last good version (41) to one cohort's environment at once, using AppConfig's predefined all-at-once strategy; we run it for each cohort's environment.

Incident flow OPS 10

Synthesizing vector architecture diagram...

The first question is the blast radius; cells and shuffle sharding exist so the answer is usually "one tenant" or "one cell".

Game days. Monthly: stop a cell's ingester group A and time the replay; send a canary tenant's cardinality to 3× and watch its limits hold. Quarterly: cut a home region off from its standby and from agents in a staging pair, and measure the canary alerts' delay. REL 12 · OPS 11

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Isolates tenants with limits, cells and shuffle sharding, and can do the overlap math.
• Designs alerting to survive the provider's own region outage, with warm history and no failover step, and prices the alternatives, including the higher Asia-Pacific transfer rates.
• Turns alerts into a product feature: SLOs, error budgets and multi-window burn rates, with the numbers.
• Protects shared query capacity with fair queues, limits, correct cache keys and recording rules.
• Meters usage idempotently and reconciles it, and knows cost per million series and what drives it.
• Answers build-versus-buy differently at each scale, with numbers.

Follow-up questions

1. "The standby fires an alert, but the home region says everything is fine. Who's right?" Answer: usually the one that sees the data sooner. If the home region is healthy, both evaluate the same rules on the same series, and small timing differences produce the same alert a few seconds apart, deduplicated by Alertmanager. A real disagreement means the subsets differ: a rule changed and the agent config hasn't refreshed, or the agent's second destination is failing. We alarm on "standby and home alert sets differ for more than 5 minutes" per tenant, and treat it as a bug in the subset, not as a vote.

2. "A customer disputes last month's bill. How do you explain it?" Answer: from the usage table: active series per hour, samples ingested and discarded, bytes stored, query samples, each with its source. We show the hours where usage jumped and, from the cardinality report, the metric and label that caused it. Because every hour's value was set once and reconciled against a second meter, we can show the same numbers they see on their usage dashboard.

3. "A tenant wants 13 months of raw data for an audit. What does it cost, and what changes?" Answer: storage only, mostly: raw is about 11.8 KB per series per day, so a 5M-series tenant adds 5M × 11.8 KB × 365 ≈ 21.5 TB a year at about $0.022/GB-month in S3 Standard, about $470 a month, or less if blocks older than 30 days move to a cheaper storage class they rarely read. Queries over raw data beyond 32 days still hit the query-length limit; for audits they'd use exports, not dashboards.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design, let me ask a few scoping questions: how many series, at what interval, kept how long, and who gets paged?"
• When the scope is raised: "Here's what breaks in the current design, and here's the order I'll fix it in."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels