The database cannot survive a successful cache

In this hypothetical service, a cache hit ratio of 99% lets a database handle 1,000 queries/second behind 100,000 reads/second. The cache restarts. The application treats every cache error as a miss and forwards all 100,000 reads.

The cache was introduced as an optimization. The workload made it a dependency.

Budget misses as real work

For read rate $\lambda$ and hit ratio $h$, a first estimate is origin rate $\lambda(1-h)$. It excludes revalidation, writes, repeated fills, and downstream amplification. Design for the hit ratio you can lose, not just the best one you measured.

Single-flight lets concurrent requests share one in-progress fill per key. Local single-flight on twenty instances can still create twenty fills. A distributed refill lease reduces that duplication but adds coordination and failure handling. Stale-while-revalidate can return a bounded stale answer while one refresh runs; some data cannot tolerate that trade-off.

pseudocodeCopy
if cache unavailable:
    return declared degraded response or fast failure
if cache hit and fresh:
    return cached value
if cached stale value permitted:
    request one bounded refresh
    return stale value
join existing fill or acquire a fill permit
if no permit:
    return declared degraded response or fast failure
try:
    make bounded origin call
    check version and publish atomically
finally:
    release fill permit

Cache errors deserve their own path. Blind fallback to the database can turn partial cache trouble into a full outage. Warm a restarted cache at a rate the origin can afford, prioritizing the working set. A warm replica reduces refill demand but can still contain stale state after failover.

Memory is bought with a workload

Estimate entry size from sampled serialized values, keys, allocator overhead, expiration metadata, and replication/buffer costs. Then measure the working-set curve: how does hit ratio change with memory? Total database size is not a cache-sizing formula.

LRU tracks recency with a hash map plus linked ordering in the exact textbook model, giving expected constant-time lookup/update. LFU tracks frequency but needs aging to adapt. Approximate production policies trade bookkeeping for throughput. A long scan can evict genuinely hot keys under LRU; frequency-aware admission can resist that pollution.

Avoid unbounded key scans or giant objects that monopolize server execution and network buffers. Fragmentation metrics are clues, not universal eviction thresholds. Watch bytes, eviction rate, miss cost, fill queue age, and per-key popularity, not hit ratio alone.

Repeated missing keys can use short-lived negative entries. Random never-repeated keys do not benefit much; a trustworthy membership filter can help when its freshness semantics permit it. Coalesce and limit failed lookups too.

The browser is another cache

A shared edge response must include every representation-changing dimension in its cache key—such as locale or an authorized scope—and exclude irrelevant tracking parameters that destroy hit ratio. Do not accidentally cache one user's private response for everyone.

HTTP max-age concerns ordinary freshness lifetime; s-maxage targets shared caches. private restricts shared storage; no-cache means revalidate before reuse, not “never store”; no-store requests no storage. Age and downstream cache behavior affect accumulated staleness. Revalidation with ETags can avoid retransmitting unchanged content but still costs a request.

Version immutable asset URLs rather than expecting an edge purge to clear a browser's already-cached object. Test the actual CDN policy: configuration can override intended cache behavior.

Mental model: a cache spends freshness and memory to reduce origin work. Cold-start survival is part of that bargain.

Transfer: the perfect hit ratio

---

Source trail: 04-distributed-cache-patterns-and-eviction, 11-caching-and-invalidation. See the source map for consolidation and reference-retained material.