37 engineering mysteries
Narrative Curriculum
Two customers booked the last seat. Your transaction succeeded. What went wrong? Start with a failure, make a prediction, then build the mechanism that prevents it.
Bring basic SQL, loops, hash maps, and HTTP. The core path builds from local correctness to regional recovery. Later lessons apply those ideas to search, identity, and abuse prevention.
Core path
- LESSON 01
Two successful transactions, one impossible result
Two buyers both reserve the last seat.
- LESSON 02
The write that survived the process but not the machine
A successful write disappears after power loss.
- LESSON 03
The file exists, but recovery must ignore it
A crash leaves an SSTable outside the durable manifest.
- LESSON 04
Fast writes, then a completely frozen database
Background merging cannot keep up with foreground writes.
- LESSON 05
Can we prove the key is absent without reading it?
Negative lookups spend most of the disk budget.
- LESSON 06
The client gave up; the server kept working
Retries multiply work whose outcomes remain unknown.
- LESSON 07
An optional widget exhausts the checkout pool
A slow dependency consumes resources needed by healthy routes.
- LESSON 08
The limiter that allows twice its advertised limit
A window boundary permits an unexpected burst.
- LESSON 09
Twenty correct limiters enforce the wrong limit
Local enforcement multiplies a tenant budget by fleet size.
- LESSON 10
Deleting the cache resurrected the old price
A slow refill races a committed update.
- LESSON 11
The database cannot survive a successful cache
A cold cache turns nearly every request into origin work.
- LESSON 12
Adding a server moved almost everything
Modulo routing remaps data during expansion.
- LESSON 13
The cluster is idle and one customer is throttled
One logical key exceeds the capacity of its owner.
- LESSON 14
The database acknowledged a write the replica cannot read
Durable, committed, and applied positions differ.
- LESSON 15
Two machines both say they are the leader
An isolated former leader continues to receive requests.
- LESSON 16
The lease expired; the old worker did not
A paused worker resumes after a replacement takes ownership.
- LESSON 17
The copy finished, then the old owner accepted a write
A live migration races stale routers and ongoing writes.
- LESSON 18
The payment succeeded twice after one timeout
An ambiguous external effect is retried as a new operation.
- LESSON 19
The queue is healthy; the customer is still waiting
Redelivery, poison records, and early acknowledgement break progress.
- LESSON 20
The order committed, but nobody heard about it
A database commit and a publish cannot be made atomic by ordering calls.
- LESSON 21
Yesterday changed after the stream restarted
Late events and replay make a plausible counter incorrect.
- LESSON 22
The dashboard recovered by hiding the incident
A roll-up discards the information needed to answer the question.
- LESSON 23
All participants voted yes; no participant can finish
Prepared transactions wait for a durable global decision.
- LESSON 24
The refund failed after the booking failed
Independent commits require recoverable semantic compensation.
- LESSON 25
Traffic failed over before authority did
Routing users to a standby does not make it a safe writer.
- LESSON 26
The current balance cannot explain its own history
A new query needs a replayable model without duplicating effects.
Apply the ideas
- LESSON 27
The unique ID generator repeated itself after restart
Clock rollback and worker reuse violate assumed uniqueness.
- LESSON 28
The failure detector removed a healthy machine
A missing heartbeat cannot distinguish failure from delay.
- LESSON 29
More connections did not fix the slow page
Transport, proxy, and dependency queues create different bottlenecks.
- LESSON 30
The socket reconnected; the messages did not
A live transport supplies neither history nor replay correctness.
- LESSON 31
The token is genuine and the request is forbidden
Authentication, authorization, and revocation have different boundaries.
- LESSON 32
Autocomplete found the prefix and still timed out
Finding a prefix does not find its best completions.
- LESSON 33
The closest driver is invisible to the index
Spatial cells omit nearby points unless coverage is complete.
- LESSON 34
The vector index returns ten excellent unauthorized matches
Approximate retrieval and access filters interact.
- LESSON 35
Every account obeyed the limit; the attack still worked
Many cheap identities evade per-identity controls.
- LESSON 36
The browser passed the challenge twice
A challenge token is accepted without a correct server-side lifecycle.
- LESSON 37
The puzzle was hard once and free thereafter
A reusable proof makes client work irrelevant to repeated abuse.