Time-Series Storage & Roll-ups

Part 0. Start here

The problem: two questions about one metric

Basketly, the grocery shop of pages 10, 12 and 14, runs its checkout service on four pods. Three of them take 20 requests a second each. The fourth, p4, is a canary: it runs the next build and takes 2 requests a second, 1 request in 31. Every pod's latency is sampled every 10 seconds, and the store rolls the samples up to 1-minute and 1-hour resolution.

On Tuesday at 10:20:00 the canary gets a slow build. For 90 seconds, every request it serves takes 2 seconds, until the rollback at 10:21:30. Two questions follow.

Question 1, the incident review on Wednesday: "What were checkout's mean latency and p99 during those 90 seconds?" The team's dashboard averages each pod's mean and each pod's p99.

Averaging broke both numbers, in opposite directions. The mean went up because the canary's 2 requests a second counted as much as another pod's 20. The p99 went down because three healthy pods outvoted the one that was slow.

Question 2, the capacity review a month later: "Show the yearly p99, with that incident on it." A yearly panel reads the 1-hour tier (Part 8), so what it shows for that morning is the 1-hour roll-up of 10:00 to 11:00.

Two facts frame the page. The Gorilla paper, whose compression Prometheus and many other time-series stores use, starts from "16 bytes per point" (an 8-byte timestamp and an 8-byte value) and reports that it can "compress time series to an average of 1.37 bytes per point, a 12x reduction in size". And Amazon CloudWatch keeps sub-minute points for 3 hours, 1-minute points for 15 days, 5-minute points for 63 days and 1-hour points for 455 days: "Data points that are initially published with a shorter period are aggregated together for long-term storage". Every long-lived metrics store rolls up. The question is what it keeps when it does.

The big picture

Synthesizing vector architecture diagram...

What to notice: samples enter at the head and are kept raw, in the head and then in blocks, for 15 days. The 1-minute tier is built from raw samples and the 1-hour tier from 60 one-minute rows, so both keep only values that merge. Queries pick a tier by their step (the three dotted arrows: raw, 1-minute or 1-hour); alert rules read raw data, because coarse tiers see an incident late or not at all.

What you'll be able to do after this page

1. Count the series a metric creates, and tell a counter, a gauge and a histogram apart (Part 1).
2. Explain why series, not samples, size memory and cost, and where to stop a label that multiplies them (Part 2).
3. Trace a sample from the write-ahead log into a chunk, a block, and out by retention (Part 3).
4. Compress timestamps with delta of deltas and values with XOR, by hand, and say when compression gets worse (Part 4).
5. Choose each roll-up's aggregates by metric type, handle counter resets, and write roll-ups that are safe to run twice (Part 5).
6. Compute a percentile over any range from merged buckets or a sketch, and state its error (Part 6).
7. Size each tier by the values it keeps, and see through the "60× smaller" claim (Part 7).
8. Pick the tier for a query, align its steps, take rates across restarts, and handle stale series and gaps (Part 8).
9. Decide what happens to a sample that arrives after its minute was rolled up (Part 9).
10. Work out when an alert can fire on raw data and on each tier (Part 10).
11. Keep the roll-up pipeline correct when a job is retried, a worker falls behind or an ingester crashes (Part 11).
12. Follow one sample through every layer, from the client library to a year-long graph (Part 12).
13. Map all of it to AWS, and name the look-alikes (Part 13).

You may have arrived from a step that relies on this: steps 1.1, 1.2 and 2.1 to 2.5 of the metrics and alerting loop (chunks, the label index, Gorilla compression, roll-up tiers, cardinality, late samples) and its step 3.4 (tiers and the results cache); R1.8 of the ad-click aggregation loop, which sends readers to the metrics loop "for time-series storage in general (compression, rollups, retention tiers)"; step 3.5 of the URL shortener (unique counts that merge); or step 3.2 of the gaming leaderboard (a histogram whose error is its bucket). In all, 9 of the 38 interview loops depend on a time-series or roll-up mechanism in at least one step.

---

Part 1. A series is a name and labels

Four pods, one latency. How many things does the store keep? It isn't one ("checkout latency"), and it isn't four (one per pod). This Part names what the store actually keeps, and sets up the example.

Name plus labels

A series is a metric name plus one full set of label values. checkout_inflight{pod="p1"} and checkout_inflight{pod="p2"} are two series; change any label value and you have a different series. A sample is one (timestamp, value) pair of one series: a timestamp in milliseconds and a 64-bit floating-point value.

Counter, gauge, histogram

A fourth type, the summary, computes quantiles inside the client and exports them finished. Part 6 shows why two summaries can't be combined.

Here is what p4 exposes when it is scraped at 10:00:00, in the text format agents read:

textCopy
checkout_duration_seconds_bucket{pod="p4",le="0.1"}   7200
checkout_duration_seconds_bucket{pod="p4",le="0.5"}   7200
checkout_duration_seconds_bucket{pod="p4",le="1"}     7200
checkout_duration_seconds_bucket{pod="p4",le="2.5"}   7200
checkout_duration_seconds_bucket{pod="p4",le="+Inf"}  7200
checkout_duration_seconds_sum{pod="p4"}               287.9999999999997
checkout_duration_seconds_count{pod="p4"}             7200
checkout_inflight{pod="p4"}                           0

Eight lines, eight series: seven counters from one histogram, and one gauge. (_sum should be 7,200 × 0.04 = 288; the client adds 0.04 in 64-bit floating point, 7,200 times, and ends a hair below it. Part 4 shows what that costs.)

The example we follow

A real metrics platform holds a hundred million series, too many to watch. So one story runs through the whole page: one metric at 10 s, 1 minute and 1 hour resolution. It is Basketly again (pages 10, 12 and 14), now through its checkout service. Every trace on this page comes from running a private reference simulation of this setup (the requests, the client counters, the scrapes, the store, the encoders, the roll-up jobs and the queries, with every rule as a setting), not from working it out by hand. The example's numbers are small so that every value fits on a line; the last column says what changes at real scale.

One simplification is labelled: our client library counts a request when it starts (a real one counts it when it completes), so every scrape's increase is an exact multiple. Side row G in Part 14 shows what counting at completion changes.

We follow Tuesday from 10:00:00 to 12:00:00, one 2-hour block, and then let the tiers age over days and months with arithmetic. Four things happen on that main line, spaced so they never overlap: the canary incident (10:20:00 to 10:21:30), a restart of pod p2 (10:40:03), a network partition that makes p3's samples seven minutes late (11:05:00 to 11:12:05), and the canary's removal (11:30:00). Everything else (a user_id label, a bucket layout that ends too low, samples resent newest first, a retried roll-up job, a stalled roll-up worker, a crashed ingester) runs on a copy of the main line, one per side row, so no event swamps another.

The story in twelve beats: 32 series from two metrics (Part 1); the label that made 280,000 (Part 2); a sample's trip to disk (Part 3); 16 bytes into one or two (Part 4); the minute the canary broke (Part 5); the p99 that averaging hid (Part 6); fifteen days, ninety days, two years (Part 7); the graph, the restart and the pod that left (Part 8); seven minutes late (Part 9); the alert the hourly tier never sent (Part 10); the job that ran twice (Part 11); one sample, end to end (Part 12). Each Part shows only its own events; the full table is in Part 14.

Events 1 and 2

Why 2 and not 1.3? At each scrape instant, p1's two slow requests from the second before (0.18 s and 0.4 s, started 75 ms and 25 ms before the instant) are still running; its fast ones are not. A sample 10 seconds later sees exactly the same picture. The gauge can't tell you what happened between those two instants, however well you store it.

Snapshot T1, 10:00:10: p4's eight series in the head

Each series holds its two samples in its own open chunk, which started at 10:00:00 and will close after 120 samples. IDs are given in the order the store first saw each series.

---

Part 2. Series, not samples, cost memory

At 10:50:00 on a copy of the main line (side row C), a developer adds a user_id label to checkout's histogram to debug one customer's slow orders. Checkout still serves 62 requests a second. Nothing about the traffic changes. Here is what does.

8.3 KB is a planning figure, from Grafana Mimir's capacity guide: "Memory: 2.5GB for every 300,000 series in memory". The $0.30 and $0.10 are CloudWatch's prices per metric a month for the first 10,000 and the next 240,000 metrics (US East). The same traffic now costs almost 9,000 times the memory and 5,000 times the price.

Finding series: the label index

A query names series by their labels: checkout_duration_seconds_bucket{pod="p4", le="2.5"}. The store doesn't scan all its series to find them. It keeps an inverted index: for every label=value pair, a sorted list of the IDs of the series that carry it (a posting list).

Event 3. The query intersects the three lists: only 28 is in all of them, so the store reads one series' chunks and nothing else. A regular expression (pod=~"p1|p2") unions the lists of the matching values first. The index costs memory per series, not per sample: about 200 B a series in the metrics loop's estimate, so 6.4 KB for our 32. The same idea, posting lists intersected, is how a search engine finds documents (Trie & Inverted Index).

What a series costs

A new sample for an existing series adds a byte or two to an open chunk (Part 4). A new series needs an ID, its label strings, entries in several posting lists, an open chunk and bookkeeping in memory: kilobytes, whatever its sample rate. That is why a series sampled once a minute costs about as much head memory as one sampled every second. "Samples tell you the bandwidth; series tell you the memory" (step 2.4 of the metrics loop).

A series stays in the head until the head is next cut into a block after its last sample (Part 3), so every user in side row C stays in memory for hours after their last request, not seconds. Managed Service for Prometheus counts the same way for its quota: "A series is active if a sample has been reported in the past 2 hours".

Where to stop a label

The metrics loop's step 2.4 stops cardinality in three layers; here they are in one paragraph each.

1. At the gateway, by rule. Drop labels known to be unbounded (user_id, request_id, raw URLs), and cap the number and length of labels on a series. The write is refused with 400, and the team sees why.
2. Where the series live, by count. Each tenant has an active-series limit, split into a share per ingester (the servers that hold the head). A sample for an existing series is still accepted; a sample that would create a new series over the limit is dropped and counted. The limit sits where the memory is.
3. On growth. Alert when a tenant's series double. To count distinct series cheaply, use a HyperLogLog sketch, which estimates "how many different?", not a Bloom filter, which answers "have I seen this one?" (Part 6).

An ingester's memory is shared by every tenant whose series hash to it, so one team's user_id label hurts every other team on those ingesters. That is why the limit is per tenant, not just per ingester. A limit on samples a second is a different limit, for bandwidth: it is usually a token bucket per tenant (Rate Limiting Algorithms). Which ingester holds which series is Sharding, Hot Keys & Rebalancing.

---

Part 3. Writing a sample: head, log, chunks, blocks

At 10:00:01 the 32 samples of the 10:00:00 scrape land at the store (event 4). Checkout sends 3.2 samples a second, 720 per series in every two hours, and the store must accept each in microseconds, never lose one it acknowledged, and later read any series' last two hours in one sweep. This Part follows a sample from its arrival until it is deleted.

Log first, then the chunk

The store's in-memory part is called the head. A sample goes through it in two steps:

1. Append it to the write-ahead log (WAL), a file written only at its end, so the sample survives a crash. Prometheus writes its WAL "in 128MB segments". What "survives" means, and what fsync buys, is Write-Ahead Log, fsync & Group Commit.
2. Append it to its series' open chunk in memory, compressed as it goes (Part 4). The sample is queryable at once.

A chunk holds one series' samples in time order. It is closed at 120 samples (20 minutes at our interval), and a new one is opened. Short chunks matter because a compressed chunk can only be read from its start.

Synthesizing vector architecture diagram...

What to notice: every arrow points forward. A sample is appended to the log and to a chunk, the head is cut into an immutable 2-hour block, blocks move to object storage and are merged, and retention deletes whole blocks. No arrow ever returns to edit a block in place.

Blocks, written once

Every so often the store takes the oldest two hours of the head and writes them as a block: every series' chunks for those two hours plus the label index (Part 2), in files that are never edited again. The WAL segments older than the block are then dropped: the block holds that data now.

When is the head cut? Prometheus waits until the head spans more than three hours (MaxTime − MinTime > chunkRange/2*3 in its source), then writes the oldest two-hour window. The extra hour is on purpose: the newest hour stays open for samples that arrive a little late (Part 9).

Events 4 to 6

How old data leaves

Retention is simple because blocks are: when a block is older than the retention period, the whole block is deleted. Nothing is rewritten. Prometheus notes that "Expired block cleanup happens in the background. It may take up to two hours", so a dashboard can still show data a little past its retention.

In the background, compaction merges small adjacent blocks into larger ones (Prometheus: "up to 10% of the retention time, or 31 days, whichever is smaller"), which cuts the number of files a long query opens. It merges; it never edits a sample. This is a time-window policy: data sorted by time and expiring by age leaves "as whole files", which is the row for "Time series and caches that expire" in Part 5 of LSM-Trees & Compaction. That page also owns what happens when compaction falls behind and writes stall; its drill, The Time-Series Database That Froze on Flush, is answered there.

Why not a table of rows

The first design in step 1.0 of the metrics loop stores one row per sample in a relational table. Each row repeats the metric's name and labels and has index entries of its own: "roughly 50 to 100 bytes on disk", against 16 bytes of timestamp and value, and about 2 bytes after Part 4's compression. And a B-tree index keyed by time scatters one series' samples across pages, so a two-hour graph of one series reads hundreds of pages instead of six chunks (step 1.1 there).

---

Part 4. About a byte a sample

A sample is an 8-byte timestamp and an 8-byte value: 16 bytes. The metrics loop's Round 2 writes 10 million of them a second, 13.8 TB a day. But consecutive samples of one series are nearly identical: the timestamps are 10,000 ms apart, and a counter grows by about the same step each time. Gorilla, the compression in Facebook's paper and in Prometheus, stores only what changed. Here it is on p4's _count, which reads 7,200, 7,220 and 7,240 at 10:00:00, 10:00:10 and 10:00:20.

Timestamps: delta of deltas

Store the first timestamp in full (the paper: as a 14-bit offset from the block's start). After that, don't store the time, or even the gap since the previous sample: store how the gap changed, the delta of deltas D = (tₙ − tₙ₋₁) − (tₙ₋₁ − tₙ₋₂). With scrapes every 10 s, the gap is always 10 s, so D is almost always 0, and 0 costs one bit.

The paper found that "about 96% of all time stamps can be compressed to a single bit". Prometheus stores milliseconds, so it uses wider ranges (10 + 14 bits, 110 + 17, 1110 + 20, 1111 + 64): "Gorilla has a max resolution of seconds, Prometheus milliseconds. Thus we use higher value range steps".

Values: XOR with the previous value

Two nearby floating-point numbers share their sign, their exponent and their leading bits. XOR the new value with the previous one: the result is all zeros except in a few "meaningful" bits in the middle.

Event 7: three samples of p4's _count, bit by bit (the paper's encoder, in seconds; the block starts at 10:00:00)

From the third sample on, the timestamp costs 1 bit, and the value costs 10 to 20 bits: this counter grows by 20 each time, so its XOR is never zero. (In milliseconds with the paper's ranges, sample 2's D would be 10,000 and cost 1111 + 32 = 36 bits; Prometheus stores the first gap as a variable-length integer instead, 2 bytes.)

What the paper measured, and what our series cost

On Facebook's workload, the paper reports: "Roughly 51% of all values are compressed to a single bit" (unchanged); about 30% take the 10 form, "with an average compressed size of 26.6 bits"; the remaining 19% take 11, "with an average size of 36.9 bits". In all, "1.37 bytes per data point". Prometheus documents "an average of only 1-2 bytes per sample". Both are averages over a workload. Ours is different.

Event 8: bytes per sample over the 10:00 to 12:00 block

Synthesizing vector architecture diagram...

What to notice: our gauges cost a quarter of a byte, because their value almost never changes. Our integer counters cost 2 bytes and the float _sum series cost 7, so all 32 series together average 2.41 bytes: worse than the paper's 1.37 (the second bar), and far better than 16 raw (the last bar).

This workload does worse than the paper for two reasons, and both are common:

• Counters that change on every sample never take the 1-bit 0 path. Of p1's 719 value XORs for _count, 704 take the 10 path and 15 the 11 path; none is zero.
• A float that accumulates fractions is the worst case. p1's _sum should read 4,680 at 10:00:00, but after 72,000 additions of 0.04, 0.18 and 0.4 in 64-bit floating point it holds 4679.999999997908. Each new value differs from the last in dozens of low bits, so it costs about 7 bytes.

The gauges show the other end: p1's gauge reads 2 at all 720 scrapes, so 719 of its value XORs are zero, 1 bit each, plus 1 bit of timestamp.

Aligned scrapes keep timestamps at one bit

Side row J (a copy): the agent's timestamps drift by −1, 0 and +1 ms in turn. The gaps become 10,001, 10,001 and 9,998 ms, so D runs 0, −3, +3. In Prometheus's encoding, two timestamps in every three now cost 10 + 14 = 16 bits instead of 1: about 11 bits a timestamp on average (10.9 in the run over the delta-of-delta timestamps; each chunk's first two timestamps are stored whole). p1's _count grows from 2.03 to 3.25 bytes a sample. The paper's encoder in seconds would not see the jitter if timestamps are rounded to the nearest second, but it would if they are truncated: 10:00:09.999 becomes second 9, and p1's _count grows from 1.97 to 2.97 bytes a sample. Prometheus's scraper, for one, moves a scrape's timestamp by up to 2 ms onto its intended schedule, "to enable better compression at the TSDB level" (a comment in its source).

---

Part 5. Roll-ups keep what can be merged

At 10:21:30 the 1-minute roll-up of minute 10:20 runs: the minute in which the canary broke. From now on, a query for "last year" should never have to decode this minute's raw samples; in 15 days they are deleted anyway. So the roll-up must store, for this one minute, everything a query a year from now could ask about it: its mean latency, its slowest moment, its p99, its request rate. What exactly do you store?

Minute 10:20, rolled up

A roll-up of a counter uses the gaps between consecutive samples: minute 10:20's increase is taken over the six gaps from 10:20:00 to 10:21:00. A gauge's minute holds the six samples 10:20:00, :10, …, :50. The roll-up runs 30 seconds after the minute ends (at M + 1 min 30 s), so the 10:21:00 sample has landed.

Synthesizing vector architecture diagram...

What to notice: the panel "Raw, minute 10:20" holds 7 samples per counter and 6 per gauge; the panel "1-minute rows" holds one number per counter and four for the gauge. Both give the same answers, mean 0.127 s and a p99 estimate of 2.035 s, because every stored number can be added to other minutes' and other pods' numbers.

Events 9 to 11

What merges, and what doesn't

A value merges if the value for a longer window, or for more pods, can be computed from the values of the parts. Only those belong in a roll-up.

A stock chart's one-minute bars (open, high, low, close, volume) are a roll-up of this kind: first, max, min, last and sum, all mergeable, so a 5-minute bar is built from five 1-minute bars (the mobile stock-trading loop, R1.4). The payment loop's revenue account is summed "into per-minute totals" by a roll-up job in the same way (step 3.1 of the payment loop).

The type decides

A counter needs one value per window, its increase: the mean, the rate and every percentile come from increases. A gauge needs four: count, sum, min and max. A store that doesn't know the type must keep all five for every series: Thanos's downsampled chunks hold count, sum, min, max and counter for each one. (A series in delta form, where each sample is already the increase since the last, has no resets to handle, but a lost sample is a lost increase.)

Counters and restarts

p2's process restarts at 10:40:03 (event 13). Its counters start again from 0, and the 60 requests it counted between its 10:40:00 scrape and the restart are lost with the old process.

Synthesizing vector architecture diagram...

What to notice: the counter climbs to 120,000 at 10:40:00, then drops to 140 at 10:40:10 because the new process counts from 0. Taken naively, minute 10:40's last value minus its first is 940 − 120,000 = −119,060. The reset-adjusted increase treats the drop as a restart: 140 + 5 × 200 = 1,140.

The roll-up walks the gaps and treats any drop as a restart from 0:

textCopy
INCREASE(one counter's samples, window (M, M + w])
  total = 0 ; gaps = 0
  for each pair of consecutive samples (t0, v0), (t1, v1) with t1 in (M, M + w]:
      if v1 >= v0: total = total + (v1 - v0)
      else:        total = total + v1          the counter restarted from 0
      gaps = gaps + 1
  if gaps == 0: write no row                   no data is not zero
  return total

Snapshot T3, 10:41:30: p2's _count across the restart (event 13)

Only a counter can lose these increments, and only a few seconds' worth per restart. The query side of the same rule is Part 8's rate().

Keeping the last point loses the spike

Side row W replays the two hours with a cheaper roll-up, decimation: keep the last sample of each window, which is what VictoriaMetrics' downsampling does ("leaving the last sample per each interval"). It is compared with D4, the mergeable roll-up, on the same data, from both sides.

The last value of a cumulative counter still carries every increase before it, so decimated histograms give the same mean and p99 as D4. What decimation loses is a gauge's spikes between the kept samples, and a counter's reset inside a window, where the kept samples no longer show the drop. Side row G in Part 14 goes one step further: a pod that freezes for 3 seconds between two scrapes shows nothing at all in its gauge, however it is rolled up, while its histogram counts every slow request. A gauge can't see between its samples; a counter can.

The hour from 60 minutes

Event 12. At 11:05:00 hour 10 is rolled up from its 60 minute rows: sums of increases, sums of counts and sums, the min of the mins and the max of the maxes. Checkout's _count for the hour is 223,140 (62 × 3,600 = 223,200, less p2's 60 lost requests). p4's gauge for the hour is count 360, sum 36, min 0, max 4: a mean of 36 ÷ 360 = 0.1 in flight, and a maximum that still shows the incident. The reference run also rolled hour 10 straight from the raw samples: for all 32 series, the two agree exactly. That is the test of a mergeable roll-up.

Set, never add

A roll-up job crashes and is retried; a late sample makes a window run again (Part 9). Every run of a window must leave the same row. So each row is keyed by (series, tier, window start), the whole window's value is computed, and the row is set to it: a second run writes the same value again. If the job instead adds its value to a stored total, a second run doubles it. The ad-click loop learned this in a failure: "Overwrite, never add, when work can be retried" (R1.9 of the ad-click loop); the job scheduler's meter records are keyed so that a re-sent one "overwrites itself in the daily rollup instead of adding to it" (step 3.5 of the job scheduler loop). Part 11 runs the retry both ways.

Where an add can't be avoided, it must be applied exactly once. A stream processor that adds each closed minute into its hour relies on its checkpoints for that ("The roll-up keeps the sum; a closed minute is added once", Part 3 of Event Time, Watermarks & Checkpoints); a store that applies adds guards each one with the last window applied, as the URL shortener's last_window condition does (step 2.2 of the URL shortener). Absolute values in general are Idempotency & Effectively-Once Processing, Part 5 there.

---

Part 6. Percentiles don't roll up

The hook's dashboard said the incident's p99 was 0.8 s by averaging the four pods' p99s; the truth was 2.0 s. A per-minute recording rule (D3) did better: it computed each minute's p99 from all pods' buckets. But averaging its 60 values for hour 10 gives 0.503 s, for an hour whose p99 was 0.4 s. Both mistakes come from treating a percentile as if it merged.

Why a percentile can't be merged

A p99 is the latency that 99% of requests beat: a rank in a sorted list. It depends on every value in the list, so two p99s can't make a third. In minute 10:20, three pods' p99 is 0.4 s and the canary's is 2.0 s. The fleet's p99 depends on how many requests each pod served, and on where the rest of each pod's latencies lie: 120 of the minute's 3,720 requests (3.2%) took 2 s, so the fleet's p99 is 2.0 s. No average of 0.4, 0.4, 0.4 and 2.0 gives that, and no weighting either: weighted by requests it is (3 × 1,200 × 0.4 + 120 × 2.0) ÷ 3,720 = 0.45 s. The p99 values alone don't carry what the merge needs.

Sum the buckets, then estimate

A histogram's bucket counts do merge: the number of requests at or below 1 s across four pods and 60 minutes is the sum of the counts. So the rule is: sum each bucket across pods and windows, and compute the percentile last. Prometheus's histogram_quantile does the last step: find the bucket where the rank falls, and interpolate linearly inside it (the lowest bucket starts at 0).

Event 14, minute 10:20. Summed across the four pods, the buckets' increases are: ≤ 0.1 s: 3,240; ≤ 0.5 s: 3,600; ≤ 1 s: 3,600; ≤ 2.5 s: 3,720; +Inf: 3,720. The rank for p99 is 0.99 × 3,720 = 3,682.8. That is above the 3,600 at or below 1 s, so it falls in the bucket (1, 2.5], which holds 120 requests; it is (3,682.8 − 3,600) ÷ 120 = 0.69 of the way in. The estimate is 1 + 0.69 × 1.5 = 2.035 s: above the slowest real latency, 2.0 s, because interpolation assumes the 120 requests are spread evenly across the bucket.

Event 15, hour 10. From the 1-hour rows: ≤ 0.1 s: 201,366; ≤ 0.5 s: 222,960; ≤ 1 s: 222,960; ≤ 2.5 s: 223,140; +Inf: 223,140. The rank is 0.99 × 223,140 = 220,908.6, which falls in (0.1, 0.5], holding 21,594 requests: (220,908.6 − 201,366) ÷ 21,594 = 0.905 of the way in, so 0.1 + 0.905 × 0.4 = 0.462 s. The true p99 is 0.4 s: only 180 of 223,200 requests were slow (0.08%). The incident survives in the hour only as the (1, 2.5] bucket's 180 and the gauge's max of 4. The p99 of a day, a month or a year is computed the same way from the 1-hour tier's buckets, never from stored p99s.

Synthesizing vector architecture diagram...

What to notice: in the panel "Right path", only bucket counts move from box to box, and each step is an addition; the percentile appears once, at the end. In the panel "Wrong path", percentiles are computed first and then averaged, which gives 0.8 s for an incident whose p99 was 2.0 s, and 0.503 s for an hour whose p99 was 0.4 s.

The bucket decides the error

A bucket estimate is only known to lie inside its bucket. 2.035 s could have been anything from 1 s to 2.5 s; 0.462 s anything from 0.1 s to 0.5 s. Two rules follow:

• Put bounds near the targets you alert on. If the SLO is "p99 under 300 ms", a bound at 0.3 turns a 0.1-to-0.5 guess into a yes or no.
• Put the top finite bound above your worst case. In the +Inf bucket, Prometheus returns "the upper bound of the second highest bucket".

Side row B (a copy): the bounds end at 1 s (0.1, 0.5, 1, +Inf). Minute 10:20's rank now falls in the +Inf bucket, and the estimate is 1.0 s, the highest finite bound. Part 10's alert "p99 above 1 s" can never fire: the answer is capped at exactly 1.

The gaming leaderboard loop (step 3.2) applies the same rule on purpose: 10,000 buckets at last season's quantiles, rank by the counts above plus interpolation inside the player's bucket, and "The error is at most the population of the player's bucket".

Sketches with a guarantee

A fixed bucket's error is absolute and can be huge. A log-bucketed histogram makes the error relative instead: its bucket bounds grow by a fixed ratio, so every bucket is, say, 2% wide wherever it sits. DDSketch is the clearest version. With a relative accuracy α = 1%, bucket i holds the values in (γ^(i−1), γ^i], where γ = (1 + α) ÷ (1 − α) ≈ 1.0202. A quantile is answered with the bucket's representative value, 2γ^i ÷ (γ + 1), which lies within 1% of every value in the bucket. Sketches merge by adding bucket counts, exactly like a histogram. Its paper calls it "the first fully-mergeable, relative-error quantile sketching algorithm with formal guarantees".

Event 16 (design D5). Each pod exports its latencies in DDSketch buckets instead of five fixed ones. The merged sketch answers 1.994 s for minutes 10:20 and 10:21 (true 2.0 s, 0.32% off) and 0.402 s for hour 10 (true 0.4 s, 0.62% off). Each pod-minute occupies 3 buckets on p1 to p3 (0.04, 0.18 and 0.4 s), 1 on p4 (2 in minute 10:21, when both 0.04 and 2.0 appear), and 4 in the merged fleet sketch. Prometheus's native histograms and OpenTelemetry's exponential histograms belong to this family: in Prometheus, "each bucket boundary is the previous boundary times 2^(2^-n)".

Two other sketches come up in interviews. t-digest "can be merged" and keeps its accuracy at the tails ("part per million accuracy for extreme quantiles"), with its error defined by rank rather than value. HDR histograms fix their precision as "the number of significant digits" across a stated range.

Histogram, sketch or summary, on equal terms

A P99 alarm across many hosts must therefore be computed from merged buckets or sketches, never from the hosts' own p99s. The same goes for per-install summaries in the mobile paging loop (step 3.6: "counts and percentiles" per install per day): a fleet-wide p95 needs each install's histogram, with the same bounds everywhere.

Distinct counts

"Unique users per day" doesn't add up into a week either: a user active on Monday and Tuesday would be counted twice. A HyperLogLog sketch per day does merge, by taking the maximum of each register: step 3.5 of the URL shortener keeps one per link per day, about 12 KB, with a standard error of 0.81%, and says it plainly: "sketches merge". HyperLogLog's internals are outside this page; Bloom Filters & Counting Filters covers the family of sketches it belongs to.

---

Part 7. Tiers: how long each resolution lives

Kept raw at 16 bytes a sample, the metrics loop's Round 2 platform (100 million series, 10 million samples a second) would write 10 million × 86,400 × 16 B = 13.8 TB a day. Even compressed to the paper's 1.37 B, five years of raw 10-second data is 1.18 TB × 1,826 days ≈ 2.2 PB (R2.2 of the metrics loop), and a five-year graph of one series would decode 5 × 365 × 8,640 ≈ 15.8 million points. Tiers fix both problems. Here is what each one costs, counted honestly.

What each tier keeps

Counting values, not windows

A roll-up doesn't store "one row per window". It stores values: one per counter per window, four per gauge per window (Part 5). So count values.

Event 18: values kept a day

A tier's size is then one product:

$$\text{bytes of a tier} = \text{values a window} \times \text{windows a day} \times \text{days kept} \times \text{bytes a value}$$

For bytes a value we plan with 2 bytes for every tier: the metrics loop's figure, and Managed Prometheus's own pricing example, which estimates storage as "2 bytes * number of metric samples". The reference run also measured our data: raw samples at 2.41 bytes (Part 4), and roll-up values at about 0.36 bytes. That second figure is an artifact of perfectly regular traffic, where every window's increase is the same number and XORs to zero; real traffic won't give it, so we don't plan with it.

Event 19: stored at retention, 2 bytes a value

The 1-minute tier is the biggest: it is only 4.4 times smaller a day than raw, and it is kept 6 times longer. Roll-ups exist to answer long ranges fast and to outlive the raw data, not mainly to save space. Thanos's documentation says so plainly: keeping every resolution "doesn't save you any space", and "The goal of downsampling is to provide an opportunity to get fast results for range queries of big time intervals".

Synthesizing vector architecture diagram...

What to notice: the raw line climbs for 15 days and levels off at 4.15 million values; the 1-minute line keeps climbing until day 90 and levels off higher, at 5.70 million; the 1-hour line is still rising slowly at two years, at 0.77 million. The middle tier holds the most. (The axis is not evenly spaced.)

The "60× smaller" claim

"Rolling up to 1 minute cuts storage 60×" is true only for 1-second data kept as one value a window.

Event 20

This is why the metrics loop skips a 1-minute tier and keeps 5-minute and 1-hour tiers instead: from 10-second data, a four-aggregate 1-minute tier saves little and is kept for months.

Hot, warm and cold

Tiers of resolution are one axis; tiers of storage are another. The newest hours live in the head, in memory, because alerts and dashboards read them constantly. Recent blocks sit on the ingesters' local disks; older blocks move to object storage, which is cheap and slower to read the first time (R2.5 of the metrics loop). Managed stores draw the same line: Timestream for LiveAnalytics has "a memory store for recent data and a magnetic store for historical data", and OpenSearch Service moves indexes from hot storage to UltraWarm and then to cold. Part 13 has the details.

Six designs on equal terms

Side row X replays the same two hours under six designs and asks each the same questions. Each design gets its due: decimation needs no job logic and keeps one value a window; finished statistics are the smallest and ready to draw; raw forever keeps every question exact. The last column, "an hour from its minute rows", stands for re-rolling after late data: every design can be recomputed from raw while raw exists (15 days), but after that only a design whose rows merge can rebuild a coarser window.

D5 comes out smaller than D4 here only because our latencies take four distinct values, so each pod-minute occupies 1 to 3 log buckets against D4's five fixed ones; real latencies spread over more buckets.

Synthesizing vector architecture diagram...

What to notice: "D0" and "D1" answer everything but cost the most or grow forever. "D2" fails only on the gauge's spike and a reset inside a window. "D3" is the smallest and gets the p99 wrong; its mean is right only because every minute had about the same traffic, and it can't be split by pod. "D4" and "D5" answer every question from a small, mergeable roll-up.

---

Part 8. Reading it back: tiers, steps, rates and gaps

At 12:10:00 someone opens checkout's dashboard, with three panels over three ranges. For the longer ones, imagine the tiers have aged for a year (the arithmetic doesn't change). Each panel is a range query: a start, an end and a step, evaluated at start, start + step, …, end, both ends included. Which tier answers each, and how many values does it read?

Three panels, three tiers

Event 21

The year panel's p99 is written histogram_quantile(0.99, sum by (le) (increase(checkout_duration_seconds_bucket[1d]))): at each daily point, the increase of each bucket over the day comes from 24 hourly rows, the four pods are summed per bucket, and the quantile comes last (Part 6).

A query reads the coarsest tier whose resolution is no coarser than its step, whose retention covers the range, and whose rows have been written. It checks that last condition per point:

textCopy
CHOOSE_TIER(step, the point's range)
  for tier in [1 hour, 1 minute, raw]:                  coarsest first
      if tier.resolution > step: skip
      if the range is older than tier.retention: skip
      if any row of the tier for the range isn't written yet: skip
      return tier                                        raw qualifies within its 15 days
  at a step coarser than the tier, aggregate its rows over each step:
      sum the increases, max of the maxes, sum the buckets and then take the quantile
  report the tier used                                   the loop's resolution_used field

So at 12:10:00 the 7-day panel's newest point, which covers minutes 12:05 to 12:09, is read from raw: minute 12:09 isn't rolled up until 12:10:30. Every other point comes from 1-minute rows. The response names the tier it used, so a graph can label itself "1-minute data" (the loop's resolution_used, R2.3).

Snapshot T5, 12:10:00: the three queries

Steps and alignment

Event 22. A panel opened at 12:00:07 asks for 10:00:07 to 12:00:07 at a 60 s step: 121 points, each at :07 past a minute. The next person to open it, at 12:00:31, asks a different question with different points, and neither matches anything cached. Aligned to the step, both ask for 10:00:00 to 12:00:00: the same 121 points, the same answer. So the query frontend aligns start and end to the step, splits long ranges into days, and caches the answers for past days. The cache key includes everything that changes the answer: tenant, query, step, range, tier, and a per-day generation that a re-roll or a backfill bumps (Part 9). The newest 10 minutes, still open to out-of-order samples, are never cached (step 3.4 of the metrics loop; caching in general is Caching & Invalidation).

Rates across a restart

Event 23. p2's counter restarted at 10:40:03. Prometheus's rate() adjusts resets per series ("Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for") and extrapolates to the ends of its window.

Per series, the rate dips just under 20 because of the 60 requests lost with the old process. The rate of the summed counter is absurd: the sum drops by 119,440 at 10:40:10 (from 372,000 to 252,560), rate() reads the drop as a reset of the whole sum, and adds the old value back, reporting about 5,100 requests a second for one window. Prometheus's documentation gives the rule: "always take a rate() first, then aggregate. Otherwise rate() cannot detect counter resets". The roll-up follows the same order: each series' increase is reset-adjusted first (Part 5), and only then summed.

A rate needs at least two samples in its window, so windows must be several scrape intervals long. rate(…[20s]) is fine while every scrape arrives, and returns nothing the moment one is missed (event 25 below).

Staleness and gaps

At 11:30:00 the canary pod is removed, and its target leaves service discovery (event 24). A query at an instant takes each series' newest sample within the last 5 minutes (the "lookback period is 5 minutes by default"). Without further help, p4 would be drawn at its last value for another 5 minutes. A pull system knows better: when a target disappears, the scraper writes a staleness marker into each of its series, and every evaluation at or after 11:30:00 finds no p4 at all.

Synthesizing vector architecture diagram...

What to notice: the store that receives the staleness marker drops p4 at 11:30:00, the moment its target leaves. The push store never hears that p4 left, so it keeps returning the 11:29:50 sample to every query less than 5 minutes after it, and p4 vanishes only at 11:34:50.

Events 24 and 25, and side row S

A scraper that remote-writes (including Prometheus in agent mode, which keeps "the same scraping APIs, semantics") sends its markers along: the remote-write specification says senders "MUST send stale markers when a time series will no longer be appended to". A push pipeline without them draws ghosts for up to the lookback period.

No data is not zero. A gap must be drawn as a gap. Some tools fill it: CloudWatch's metric math AVG over an array of series says "Missing values are treated as 0", which drags an average down. And a rule that checks "p99 above 1 s" is silent both when things are fine and when there is no data at all, so pair it with an explicit rule for absence (absent(), R1.9 of the metrics loop).

One more reading hazard: a host whose clock runs behind makes its newest samples look late, so a rule over the last minute sees a dip. The metrics loop evaluates rules 30 seconds in the past (query_offset, R2.8) to leave room for it.

---

Part 9. Late data after a roll-up

At 11:05:00 p3's node loses its network. Its agent keeps scraping every 10 seconds and buffers what it can't send. At 11:12:00 the network is back, and at 11:12:05 the agent sends its 42 buffered scrapes (11:05:00 to 11:11:50) in one request, oldest first: 336 samples. By then the roll-ups of minutes 11:04 to 11:10 have already run without them. What does the store do with them, and what does the roll-up do?

Late, or out of order?

Event 26. The store's newest time when the batch lands is 11:12:00 (the other pods' 11:12:00 samples arrived at 11:12:01). p3's own newest sample is 11:04:50. Each buffered sample is newer than its series' newest, so for its series it is in order, and all 336 are accepted and appended to their chunks. They would be accepted even with an out-of-order window of 0.

Late means "arrived long after its timestamp"; out of order means "older than a sample its series already has". A chunk can only be appended to (Part 4), so only the second is a problem for the store. The batch is late but in order.

The out-of-order window

Side row O (a copy): the agent sends the batch newest first. 11:11:50 arrives first and is in order; the other 41 are older than their series' newest sample. Whether they are kept depends on the store's out-of-order window: a sample older than its series' newest is accepted if its timestamp is within the window of the store's newest time ("as long as the timestamp of the sample is >= TSDB.MaxTime-out_of_order_time_window", in Prometheus's words), and goes into a small separate out-of-order chunk for its series, merged with the normal one when the block is written.

The whole rule, as the store applies it to each sample of a request:

textCopy
ACCEPT(sample of series s at time t)                  head_max: the head's newest time when the request arrived
  min_valid = max(head_max - 1 hour, end of the newest written block)
  if t >= min_valid and t > newest(s):   append to s's chunk                      in order
  elif window > 0 and t >= head_max - window:   append to s's out-of-order chunk  merged at block write
  else:                                   reject; route it to the backfill path

The one-hour floor is why Part 3's head keeps three hours before it cuts a block: anything in the newest hour can still be appended in order. A window costs memory for out-of-order chunks, and only series that actually receive late samples have one; Managed Prometheus also caps out-of-order ingestion at 5% of a workspace's ingestion rate. Data older than any window takes a slower path: the backfill path of step 2.5 of the metrics loop, where an hourly job builds normal blocks for the past windows and the compactor merges them with the existing ones (vertical compaction). Side row V in Part 14 runs it: a partition that lasts 95 minutes.

Dirty windows and re-rolls

The samples are in the store now, but the 1-minute rows for 11:04 to 11:10 were written without them. A sample accepted after its minute was rolled up makes that minute dirty: the minute of the gap it ends, and the minute it starts. The next run re-rolls every dirty minute from raw, with set (Part 5), and re-rolls any hour already built from them.

Events 27 and 28

Snapshot T4, 11:12:30: checkout's _count per minute, all pods

Synthesizing vector architecture diagram...

What to notice: the panel "Rolled up 11:05:30 to 11:11:30" shows the minutes as first written, without p3. The batch in "11:12:05: batch lands" marks 7 minutes dirty, and "Re-rolled 11:12:30" rewrites them. In "Hour 11 at 12:05", the hour built after the re-roll counts 219,580; the dotted path, which never re-rolls and has no completeness gate, gives 212,180.

Every answer computed from those minutes before 11:12:30 was wrong and is now different. So a re-roll must also tell the results cache: it bumps the generation of the day it changed (Part 8), and a day's answers aren't cached until it is past the out-of-order window.

Early and often, or late and rarely

There are two ways to schedule roll-ups, and both are in production.

The loops use the first. The S3-like storage loop meters usage into hourly roll-ups, and "A batch that arrives late … marks that hour dirty, and the rollup is recomputed. Hours are final after a day." The ad-click loop recomputes the last three event hours on every run and overwrites them (step 1.5). CloudWatch is honest about its own open periods: "data aggregated between 7:00pm and 8:00pm begins to be visible at 7:00pm, then the values of that aggregated data may change as CloudWatch collects more samples during the period"; and it accepts time stamps "up to two weeks in the past". Whichever you choose, decide it once and say it: which windows are provisional, and when they become final. A streaming job's late path, its watermark and its acceptance horizon are Event Time, Watermarks & Checkpoints, Part 4 there.

---

Part 10. Alerting on coarse data

The rule is "checkout p99 above 1 s", evaluated every 15 seconds. The canary's p99 was 2 seconds for 90 seconds. When can it fire? It depends entirely on which data the rule reads.

Event 30: three tiers, three fire times

Synthesizing vector architecture diagram...

What to notice: the raw rule fires 30 seconds into the 90-second incident. The 1-minute rule fires only when minute 10:20's row is written, at 10:21:30, the moment the incident ends. The hourly rule's bar stretches to 11:05 and ends with no alert: the hour's estimate, 0.462 s, never crosses 1 s.

What sets the delay

On a tier, an alert can't fire before the window closes and its roll-up runs:

$$\text{earliest fire time on a tier} = \text{window end} + \text{roll-up grace} + \text{evaluation interval}$$

For minute 10:20: 10:21:00 + 30 s + 0 (the rule runs as the row is written) = 10:21:30. For an hourly tier, at best 11:00:00 + 5 minutes, and only if the incident moves the hour's p99, which a 90-second incident can't.

On raw data the window slides, so the question is when the bad share inside it crosses the threshold. The p99 is above 1 s once more than 1% of the window's requests are slow: 0.99 × 3,720 = 3,682.8, so at least 38 of a minute's 3,720. The canary adds 2 slow requests a second, so about 19 seconds of them. The scrapes see them in steps of 20: 20 at the 10:20:10 scrape, 40 at 10:20:20, which lands at 10:20:21. The evaluation at 10:20:15 sees 20 slow of 3,100 in its window (0.65%): not yet. The evaluation at 10:20:30 sees 40 of 2,480 (1.6%): it fires. So detection on raw data is the time for the bad share to cross the threshold, plus the scrape interval and the landing delay, plus up to one evaluation interval.

A rule's for duration adds its own delay on top: the metrics loop's example rule waits for: 5m, so this 90-second incident, firing from 10:20:30 to 10:22:00, would never page at all. For a canary, that may be exactly what you want; choose it per rule (step 1.4 of the loop).

The hour dilutes

Event 31. The 1-hour tier doesn't just see the incident late; it barely sees it. Hour 10's mean latency is (3,600 × 3.98 − 180 × 0.04 + 180 × 2.0) ÷ 223,200 = 0.0658 s, against 3.98 ÷ 62 = 0.0642 s in a normal hour (3.98 s is one second's 62 latencies added up). The 180 slow requests are 0.08% of the hour. A coarse tier is for history, not for detection.

So: alert on raw data, or on recording rules computed from raw data every evaluation. Long windows, such as a 30-day SLO, come from recorded short sums added up, not from averaging a tier: the metrics loop builds its 3-day burn-rate ratio "from recorded 5-minute sums rather than raw samples" (step 3.3). The same loop budgets the whole path from scrape to page (R1.7). And a threshold rule is silent when data stops, so give it an absent() companion (Part 8).

On Managed Prometheus the minimum rule evaluation interval is 30 s by default (an adjustable quota), so our 15-second rule would evaluate half as often there.

---

Part 11. When the pipeline breaks

Every Part so far assumed the roll-up jobs run on time, once, and the head never loses its memory. None of that holds for long in production. On copies of the main line, here is what goes wrong, and what keeps the tiers right anyway.

The job that ran twice

Side row R. At 11:21:30 the roll-up of minute 11:20 starts writing its 44 rows. After 22 (all of p1's and p2's: seven counters and four gauge values each), the worker crashes. The job is retried 10 seconds later and writes all 44.

The doubled minute is hard to spot. p1's and p2's buckets, _sum and _count are all doubled together, so the p99 and the mean barely move (the mean goes from 0.0642 s to 0.0645 s). Only the request rate jumps, for one minute, and hour 11 now counts 2,400 requests that never happened.

Synthesizing vector architecture diagram...

What to notice: the retry writes all 44 rows, including the 22 the first attempt already wrote. With add, those 22 are counted twice; with set, the second write of each row leaves the same value, so the minute is right however many times the job runs.

A roll-up that is safe to run any number of times looks like this:

textCopy
ROLL_UP(tier, window W)                          safe to run any number of times
  for each series s:
      if s is a counter: value = INCREASE(s's samples, W)               Part 5
      else:              value = (count, sum, min, max) of s's samples in W
      if there is no data: write nothing          missing, not zero
      else: SET row (s, tier, W) = value          overwrite, never add
  mark W rolled up; clear W's dirty flag

ROLL_UP_HOUR(H)
  if a series that isn't stale lacks a minute row in H: wait, and retry every minute
  else: SET each hour row = the merge of its 60 minute rows

A worker that fell behind

Side row K. The roll-up worker is down from 11:40:00 to 12:10:00. The runs due at 11:40:30 to 12:09:30 don't happen, so 30 minute windows (11:39 to 12:08) have no rows. Two things must not break:

• Queries at a 1-minute step find no rows for that range, so they fall back to raw data for those points (Part 8's rule, applied per point). They are slower, not wrong.
• Hour 11, due at 12:05, would be built from 39 of its 60 minutes. The hour job checks first: for every series that isn't stale, all 60 minute rows must exist. Minutes 11:39 to 11:59 are missing for p1, p2 and p3, so it waits. p4's missing rows after 11:30 don't count: its series have been stale since then, and waiting for them would block the hour forever.

Snapshot T6, copy K

Run anyway at 12:05, hour 11 would have counted 39 minutes: 143,980 requests, missing 21 × 3,600 = 75,600 (60 a second, after the canary left). A coarse tier is complete only when all its inputs are: wait, or write it marked partial so that readers can tell. The worker's capacity is shared by the regular runs, the re-rolls of Part 9 and any catch-up, so a burst of late data or an outage delays every tier at once. The metrics loop's failure table calls this "The compactor falls behind" (R2.8) and alarms on the age of the oldest uncompacted block; the equivalent here is the age of the oldest missing roll-up.

An ingester that crashed

The head lives in memory. When the ingester holding it crashes, the head is rebuilt by replaying the WAL: on a copy (side row I, in Part 14), the ingester crashes at 11:50:00 and is back at 11:50:30, replaying the log's 31,688 samples and markers (and its 32 series entries) rebuilds all 32 series exactly, and the agents resend the 3 scrapes per series they couldn't deliver, in order. At the metrics loop's scale the replay takes much longer: about 33 minutes to rebuild a shard's head (R2.6), which is why that design keeps two replicas of every series so that queries and alerts are answered meanwhile (step 2.2). Durable logs and replay are Write-Ahead Log, fsync & Group Commit; a second copy is Replication, Quorums & Read-Your-Writes.

A corrupt block

Blocks are files, and files get damaged. The storage engine detects it with checksums and recovers by dropping the damaged range and fetching it again from another replica; the mechanics are in Part 10 of LSM-Trees & Compaction. And one failure needs no crash at all: a label that multiplies the series (Part 2).

---

Part 12. End to end through the layers

One sample, many layers. At 10:20:10 p4's bucket le="2.5" reads 9,620: the first sample that shows the canary's slow requests. Here is everywhere it goes, and what each layer keeps.

Synthesizing vector architecture diagram...

What to notice: the path splits at the ingester's head. Alert rules read the head's raw data within seconds; history flows on to blocks in object storage and to the roll-up tiers, which the query frontend reads by step. Each box keeps its data for a different time, from the client's counters (until the process restarts) to the 1-hour tier (2 years).

Event 32: the 10:20:10 sample of p4's le="2.5" bucket

Layer by layer

---

Part 13. On AWS

AWS runs this mechanism in four managed services, and each makes a different choice about the three questions this page keeps asking: what is kept for an old window, how late a sample can be, and what you pay for (samples, series, or instances). Anything else, you run yourself.

Managed services that use it

CloudWatch, Managed Prometheus, Timestream for InfluxDB and your own cluster, on equal terms

Three limits are shared. A Managed Prometheus workspace's 50 million active series and 1,666,666 samples a second are shared by everything that writes to it, and out-of-order samples have their own 5% share. A query's sample budget (Managed Prometheus: 50 million samples per 24-hour interval of a query) is shared by every panel of a dashboard refresh. And CloudWatch's price per metric is multiplied by every dimension combination a service emits (Part 2's user_id: $6,000 a month).

Running it yourself

Look-alikes that are not this mechanism

---

Part 14. What you've learned

Back to the canary

The incident review asked for the 90 seconds' mean and p99. Averaging each pod's numbers said 0.549 s and 0.8 s; the truth was 0.127 s and 2.0 s. The capacity review asked for the yearly p99 with the incident on it. Averaging each minute's p99 said 0.503 s for hour 10; the hour's summed buckets give 0.462 s as an estimate, the truth is 0.4 s, and the incident survives as 180 requests in the (1, 2.5] bucket and a gauge maximum of 4. Here is what each piece did:

• Counting series, not samples (Parts 1 and 2) showed 32 series from two metrics, and why one user_id label turns them into 280,000: 2.3 GB of memory and $6,000 a month, for the same traffic.
• The head, the WAL and immutable blocks (Part 3) made every write an append and every deletion a whole block.
• Delta of deltas and XOR (Part 4) brought 16 bytes down to 2.41 on our series, from a quarter of a byte for a steady gauge to 7 for a float sum.
• Roll-ups that keep only what merges (Part 5): increases for counters, count, sum, min, max for gauges, set by key, so minute 10:20 still answers 0.127 s and max 4 a year later.
• Merging buckets before the percentile (Part 6) turned 0.8 s and 0.503 s into 2.035 s and 0.462 s, and a 1% sketch into 1.994 s and 0.402 s.
• Counting values per tier (Part 7) showed the 1-minute tier is the biggest, 11.4 MB, and that 60× is a myth at 10 s.
• Choosing the tier by step, rating before summing, and treating stale as absent (Part 8) kept the year panel at 8,784 rows, the rate at 19 instead of 5,100, and the removed canary off the graph.
• Re-rolling dirty windows (Part 9) put p3's 7,400 late requests back into hour 11: 219,580, not 212,180.
• Alerting on raw data (Part 10) fired at 10:20:30, while the 1-minute tier fired as the incident ended and the hour never did.
• Set-by-key writes and completeness gates (Part 11) kept a retried job from doubling 22 rows and a stalled worker from publishing an hour 75,600 requests short.
• Following one sample through every layer (Part 12) put each limit where its cost is.

The snapshots, side by side

What it costs

1. Roll-up jobs that must keep up, be safe to run twice, and track which windows late data made dirty.
2. Percentiles only as exact as the buckets, or a sketch format that every layer supports.
3. Detection delay on coarse data: alerts must read raw data or recording rules built from it.
4. Late data that forces re-rolls and cache invalidation, and recent windows that stay provisional until they are declared final.
5. Kilobytes of memory per series, and a limit to enforce on every label.
6. Storage that tiers add: kept together, the tiers are bigger than raw alone, and the middle one is often the biggest.

The whole story, event by event

Side rows run on copies of the main line and are marked "(side)".

The cheat card

Failure checklist

• Is every label bounded, and is there an active-series limit per tenant where the series live?
• Do you size memory from series count, not from compressed bytes?
• Are scrapes aligned, so timestamps compress to a bit?
• Does every roll-up keep increases for counters and count, sum, min, max for gauges, and nothing finished?
• Are counter increases reset-adjusted, in roll-ups and in queries?
• Is every roll-up row set by (series, tier, window), never added to?
• Are percentiles computed only from merged buckets or sketches, never averaged?
• Do bucket bounds sit near the alert thresholds, with the top bound above the worst latency?
• Is each tier sized by the values it keeps, with a real bytes-a-value figure?
• Does every query choose its tier by step, align to the step, and report the tier it used?
• Is every late sample either appended, kept by an out-of-order window, or routed to backfill, and never re-stamped?
• Does every late sample mark its windows dirty, and does the re-roll bump the results cache's generation?
• Do coarse tiers wait for all their non-stale inputs, and do you alarm on the oldest missing roll-up?
• Do alerts read raw data, with an absent() rule for when the data stops?

Words we use

Think-first drills

Interview questions

Where to go next

• Write-Ahead Log, fsync & Group Commit: the head's log, and what a crash can lose (Parts 3 and 11 here).
• LSM-Trees & Compaction: time-window compaction, write stalls and corruption (Part 5 and Part 10 there), and its drill The Time-Series Database That Froze on Flush.
• Event Time, Watermarks & Checkpoints: streaming windows, the hour from closed minutes, allowed lateness and the acceptance horizon (Parts 3 and 4 there).
• Sharding, Hot Keys & Rebalancing: hashing series to ingesters, and mergeable pre-aggregation (Part 5 there).
• Idempotency & Effectively-Once Processing: absolute writes and why set beats add (Part 5 there).
• Caching & Invalidation: the results cache in general.
• Queues & Delivery Semantics: the ingest stream and consumer lag.
• Rate Limiting Algorithms: the samples-a-second limit per tenant.
• Replication, Quorums & Read-Your-Writes and Multi-Region Failover: the head's second copy, and an alerting path that survives a Region.
• Background: Write-Ahead Log & LSM-Trees, Trie & Inverted Index (posting lists), Bloom Filters & Counting Filters (sketches).
• Loops that rely on this page: metrics and alerting (the opener, R1.4, steps 1.0 to 1.2 and 1.4, R1.7, R1.9, steps 2.1 to 2.5, R2.6, R2.8, R2.9, R2.11 and steps 3.1 and 3.3 to 3.6), ad-click aggregation (steps 1.2, 1.5, 2.1 and 2.3, R1.8, R1.9, R2.5, R2.6 and step 3.4), the URL shortener (steps 2.2 and 3.5), the gaming leaderboard (step 3.2), S3-like storage (step 3.5), the job scheduler (step 3.5), payment processing (step 3.1), mobile stock trading (R1.4) and the mobile paging library (step 3.6 and R3.9).