The write that survived the process but not the machine
A hypothetical storage service passes its crash test: kill the process, restart it, and the acknowledged writes return. Then the host loses power. Several successful writes disappear.
The team tested a failure. It did not test the failure its promise implied.
Follow the bytes
A successful write() can leave bytes in the operating system's page cache. Killing the application does not necessarily destroy that cache. Losing power does. A userspace buffer is even earlier in the journey: it may not have reached the kernel at all.
The service appends to a log, updates its in-memory table, and answers OK. At which boundary can it honestly promise survival of an OS crash?
Synthesizing vector architecture diagram...
The solid path protects the local crash promise. The dotted path adds copies, but only its own persistence and commit rules determine what failures those copies survive.
A write-ahead log records enough information to recover a change before the corresponding dirty data is allowed to reach persistent storage. For a committed durability promise, the required log prefix must be durable before success is acknowledged. Some engines update memory before that sync; the essential boundary is persistent ordering and acknowledgement, not a universal rule forbidding earlier in-memory work.
On POSIX/Linux-style systems, fsync includes file data and metadata; fdatasync need not persist metadata irrelevant to retrieving the data. Creating or renaming a file also raises directory-entry durability questions. O_DIRECT bypasses the page cache but is not a synonym for durable writes. Verify the actual filesystem and platform contract. Linux fsync documentation.
RocksDB exposes WAL and sync choices: a non-sync write has a different failure promise from a sync write. Do not infer power-loss durability merely from enabling the WAL. RocksDB WAL documentation.
The safe version is too slow
Assume, for this calculation only, that a serialized log synchronization takes 1 ms and your arrival rate is 100,000 small writes/second. One sync per write completes only about 1,000 writes/second.
Turning sync off changes the contract. Buying advertised IOPS does not automatically shorten a serialized synchronization dependency.
Instead, include several pending records in one sync. At steady arrivals, roughly records arrive during a sync. A batch can cover them together. This is group commit.
A writer arriving during a sync waits for the next batch and that batch's sync. In the simplified model, its wait is roughly one to two sync durations, excluding CPU, queue buildup, and device variability. At an average 1.5 ms, Little's law predicts about 150 concurrent outstanding writes. A single synchronous client cannot generate that concurrency by wishing for more IOPS.
A failed sync is an uncertain history
Use this conceptual write path, not as production code:
pseudocodecollect bounded batch encode records with length, type, sequence, checksum append all bytes, handling short writes synchronize required log prefix if synchronization fails: stop acknowledging writes enter recovery/error handling else: publish the committed state acknowledge the batch
Do not acknowledge a partly synchronized batch. Do not assume retrying fsync repairs an earlier I/O error; the engine must account for what may have been lost or marked clean. A durable record whose response was lost is possible, too. Clients need an idempotent retry protocol because āno OKā does not mean ānot written.ā
Measure sync latency, batch size, queue age, bytes written, and error count separately. Mapping one sync to one device I/O is only a simplified estimate: filesystems, metadata, flush commands, and background writes change physical accounting. Compaction shares that device budget, which will become the next storage bottleneck.
Mental model: OK is a promise against a named failure, made at a specific durable boundary.
Transfer: three copies in memory
The service waits for three replicas to receive a batch but none synchronizes it. Does it now survive power loss? What must a review of the acknowledgement policy ask?
Source trail: 07-write-ahead-log-and-lsm-trees, 01-write-ahead-log-fsync-and-group-commit. See the source map for consolidation and reference-retained material.