The dashboard recovered by hiding the incident
A hypothetical dashboard averages two minute averages: one minute has one request taking 100 ms; another has 99 requests taking 0 ms in this simplified dataset. It displays 50 ms. The actual combined average is 1 ms.
Compression saved storage by discarding the denominator.
Keep what can be merged
What should each minute retain so an hour's average is correct? Would storing each minute's p99 let you calculate the hour's p99?
For a gauge window, (count, sum, min, max) supports a useful family of later questions. For a counter, retain reset-aware increases or the information necessary to derive them. A last sample alone can hide a brief spike or a counter reset. Compute per-series rates before aggregating series so one reset does not cancel another series' growth.
A histogram merges by adding compatible bucket counts. Estimate a percentile only afterward, and state the bucket-resolution uncertainty. Sketches offer different guarantees; a logarithmic-bucket sketch can target relative value error, while a distinct-count sketch such as HyperLogLog estimates cardinality rather than a latency distribution. Do not treat all āapproximate summariesā as interchangeable.
Labels can cost more than samples
A time series is a metric name plus its complete label set. Add an unbounded request-ID label and every request may create a new series. Memory includes active-series metadata, index entries, open chunks, and recovery stateānot merely compressed sample bytes.
If independent labels have cardinalities 100, 20, and 50, their Cartesian product is up to 100,000 series before other dimensions. Real combinations may be fewer, but a label's potential multiplication deserves a budget. Reject or aggregate unbounded dimensions at instrumentation/ingest boundaries; a downstream roll-up arrives too late to protect head memory.
Storage earns compression from regularity
A typical time-series design logs samples, buffers open chunks, and publishes immutable blocks with indexes. Timestamp delta-of-delta encoding rewards regular intervals: if the time gap repeats, its delta is zero. XOR successive floating-point values and similar bit patterns can compress well. No fixed āone byte per sampleā ratio applies to every metric; irregular timestamps and noisy values compress differently.
Retention often deletes whole blocks. Late samples may require a bounded out-of-order path or backfill; changing an old block means rebuilding affected summaries and invalidating cached queries. Checksums detect corrupt blocks, but restore/replication material supplies the missing data.
Coarser does not mean sixty times smaller
Suppose raw gauge samples arrive every ten seconds: six values per minute. A one-minute summary retaining count, sum, min, and max stores four values. Ignoring encoding/metadata differences, that is a 1.5-fold reduction, not sixty-fold.
Use values_per_window Ć windows_per_day Ć days Ć bytes_per_value, then add indexes, replicas, and temporary compaction/rebuild space. Select a query tier no coarser than the requested step and only if that tier has actually been materialized for the requested range.
No data is not zero. A missing scrape, stale series, and real zero need distinct semantics. Threshold alerts should be paired with missing-data detection where absence itself is dangerous.
A late sample changes several products
A late raw point can dirty its minute, that minute's hour, and cached queries over both. Recompute absolute versioned aggregates; adding the full recomputed minute to the hour repeats old data. Publish coarse windows only when required inputs are complete under policy, and monitor the oldest missing roll-up.
Alert from a resolution that preserves the event you care about. An hourly mean can dilute a one-minute outage; a raw alert includes scrape, evaluation, and threshold-crossing delay. A coarse-window alert additionally waits for window closure and lateness allowance.
Mental model: a roll-up is a promise about which future questions remain answerable. Every discarded statistic removes possible answers.
Transfer: the cheaper SLO report
Storage is expensive, so a proposal keeps only daily p99 and deletes all raw samples and histogram buckets. Can you later compute monthly p99 or investigate a five-minute incident?
Source trail: 07-write-ahead-log-and-lsm-trees, 15-time-series-storage-and-roll-ups. See the source map for consolidation and reference-retained material.