Fast writes, then a completely frozen database

Your hypothetical write service is fast for twenty minutes. Then latency rises, background I/O saturates, and finally the engine stops accepting writes. There is plenty of unused logical space.

The first dashboard says “foreground writes are sequential.” The second says the same bytes have been written many times.

Where the write went after OK

An LSM engine buffers ordered changes in a MemTable, protects them with a WAL according to its durability mode, and flushes immutable sorted tables to disk. This makes incoming writes convenient; it also creates many overlapping versions to search later.

Represent entries as (key ascending, sequence descending, value-or-tombstone). For a read at sequence $s$, choose the newest visible version no later than $s$.

Compaction pays the deferred bill

Reads inspect MemTables, overlapping level-zero files, and candidate files in deeper levels. A block index narrows the range; a filter may rule out a point lookup. A scan still merges ordered iterators and resolves versions—it cannot use a membership filter to skip arbitrary ranges.

Compaction merges sorted runs, discards versions no live view needs, and publishes replacement files through the manifest protocol from lesson 03. With $k$ sorted input runs, a heap-based merge uses roughly $O(M\log k)$ comparisons for $M$ entries; a two-run merge is linear. Disk work includes reading inputs and writing retained outputs.

Now quantify the stall. Assume physical write capacity available to this workload is 200 MB/s, and each logical byte induces ten physical written bytes including the chosen accounting of WAL/flush/compaction. Sustainable ingest is at most about 20 MB/s before headroom. Increasing MemTable size delays the stall but does not make a 30 MB/s steady ingest sustainable.

Separate write amplification, read amplification, and space amplification. The denominators matter: an engine's compaction-only ratio may exclude WAL writes, so do not multiply it into a total-disk estimate without checking the definition.

A policy chooses which bill to pay

With level size ratio $T$ and a first-level scale $M_0$, a rough depth estimate is $\log_T(D/M_0)$ for data size $D$. This is planning intuition, not an exact write-amplification formula. Workload locality, overlap, value sizes, snapshots, and policy determine actual I/O.

Long snapshots keep versions alive; large values make repeated rewriting expensive; tombstone-heavy scans may inspect many entries to return few results. Separating large values from keys can reduce compaction copying but adds value-log reclamation and another read path.

Operate the debt: watch pending compaction bytes, level-zero file count, flush queues, stall time, free temporary space, and useful read latency. Throttle ingestion before backlog exhausts every resource. Reserve capacity for recovery and replica catch-up rather than benchmarking only an empty healthy node.

A B-tree can be a better choice for read-heavy workloads and predictable range access. Its page updates also have logging and write costs, but “random writes are always bad” is not a decision rule. Measure both under realistic cache sizes and durability settings.

Mental model: an LSM converts immediate random-update work into a continuing merge obligation. Sustained throughput is limited by paying that obligation.

Transfer: the larger buffer

---

Source trail: 07-write-ahead-log-and-lsm-trees, 02-lsm-trees-and-compaction. See the source map for consolidation and reference-retained material.