The file exists, but recovery must ignore it
After a simulated power cut, your data directory contains a new sorted table. It has a valid checksum and all the newest values. The database ignores it during restart.
Deleting that file feels like data loss. Keeping it might be worse.
A file is not a committed database version
The storage engine publishes a set of live files through a durable manifest. A flush creates a new SSTable, but the manifest does not yet list it when the crash occurs. The old log still protects those records.
Why not scan every file in the directory and use all valid records? Consider a compaction that replaced two old tables and created only one of its two planned outputs before crashing.
The publish protocol orders durability before visibility:
pseudocodewrite output table(s) sync table contents and required directory entries append one manifest edit: add outputs, remove inputs make that manifest edit durable publish the new live-file set reclaim inputs only when no reader still references them
For a flush, retain its WAL until the replacement is durable and discoverable. For compaction, retain inputs until the replacement is published and old readers are done. A crash before the manifest edit leaves unreferenced outputs; a crash after it may leave obsolete inputs. Both are recoverable because the authoritative file set is explicit.
The last record is only half there
A log record needs framing: length, type, sequence, payload, and checksum. Put a strict upper bound on length before allocating memory. Checksums detect damage; they do not establish whether a record was acknowledged.
In a teaching interrupted-append model, records 1–8 were synced. Record 9 is complete but unsynced; record 10 is torn. Recovery accepts the valid prefix through 9 and stops at the invalid tail. Record 9 may survive even though its caller never received success.
But now change the evidence: record 4 fails its checksum in a previously synced segment. You no longer have evidence of an ordinary interrupted tail. It could be later media corruption affecting acknowledged data. Quarantine or fail recovery according to a documented policy; restore from verified backups or healthy replicas. Silently skipping damaged middle records can produce a database that looks available but violates its history.
The checkpoint is not the slowest reader
A checkpoint may make an old log segment unnecessary for local recovery. A replica, backup shipper, or CDC consumer can still need it. Reclamation requires both conditions:
- Its changes are durable and discoverable elsewhere for recovery.
- Every required downstream reader has passed it, or has an explicit rebuild path replacing that obligation.
If log generation is 20 MB/s and a required consumer stalls for one hour, retention grows by about 72 GB, using decimal units. A cap protects the disk but can create a consumer gap. That is a policy choice requiring alarms and a resnapshot procedure, not free garbage collection.
A B-tree has the same ordering problem in another shape
A page-oriented engine may replay redo records only when the page's log sequence indicates they are newer than its applied history. That makes repeated redo safe. A torn page needs more than a record checksum: full-page images, a doublewrite strategy, or proven atomic page writes can supply repair material. WAL is not exclusively an LSM feature.
A complete restart reads the durable metadata root, reconstructs the committed file/page state, replays the necessary log prefix, and only then accepts new work. Measure recovery duration as a budget involving retained log size and replay speed. A backup is useful only after a restore has been exercised.
Mental model: bytes become part of the database through a durable publication protocol, not by existing on disk.
Transfer: the cleaner's mistake
A cleanup job deletes every WAL segment older than the last checkpoint. Local restart succeeds, but a change-stream consumer cannot resume. Was the checkpoint wrong?
Source trail: 07-write-ahead-log-and-lsm-trees, 01-write-ahead-log-fsync-and-group-commit, 02-lsm-trees-and-compaction. See the source map for consolidation and reference-retained material.