The cluster is idle and one customer is throttled
One tenant times out while a hypothetical cluster averages 15% CPU. Adding three nodes barely changes its latency. The tenant's counter is updated for every event, and every update still lands on one key.
The cluster has spare capacity. The operation cannot reach it.
Separate hot reads from hot writes
Would a dedicated shard increase the maximum write rate of that one counter? What does it actually protect?
Hot reads can use bounded-staleness caching, request coalescing, or additional readable replicas. Coalescing by key lets many waiters share an in-flight read; the benefit depends on overlap. In a simplified Poisson arrival model with rate and fixed read time , one coalescer sends about origin requests/second. That estimate assumes arrivals join a single outstanding fill and no cache lifetime beyond the fill.
Hot writes need a different representation. Split a commutative counter into subkeys, or preaggregate durable events and publish absolute totals with sequence numbers. If another code path still updates the original key per event, it preserves the bottleneck.
Salting creates a read bill
Assume each shard sustains 1,000 writes/second, but your operating target is 80% and other traffic already uses 400. The headroom for this tenant is 400 writes/second per shard. A 6,000-write/second counter needs at least fifteen independently placed subkeys under those assumptions.
Random salts do not prove fifteen distinct physical owners; inspect placement or provision extra headroom. A read now needs fifteen values unless a derived total is cached. The total may be approximate during concurrent changes unless the query has a snapshot/consistency protocol.
Synthesizing vector architecture diagram...
This is a workload sketch. An ordinary INCR is not idempotent when the event is redelivered. Store event identity with the effect, or publish versioned absolute totals from an authoritative aggregation owner.
Some invariants cannot be salted away
A like count can tolerate delayed aggregation. A stock counter deciding whether to sell the last item cannot safely read an arbitrary inconsistent sum and decrement elsewhere. Options include a single authority, preallocated inventory rights whose total is bounded, or a business policy allowing oversell and compensation. Name which one the product accepts.
Partition keys also serve queries. A tenant-plus-time bucket can cap partition growth, but cross-time queries fan out. A time-first range key can concentrate all new writes. A low-cardinality region/country key creates very few large partitions. Secondary indexes can become hot even when base-table keys are well distributed.
Observe top keys and per-shard latency/throughput, including secondary indexes. Use per-tenant admission to protect neighbors while redesigning. A global average conceals the exact imbalance you need to diagnose.
Mental model: a shard is capacity; a key is a serialization choice. More shards help only if work can be divided among them.
Transfer: the audit counter
A worker buffers 1,000 events in memory and writes one counter increment. It crashes just before flushing. A restart replays the events and flushes twice after an uncertain response. What two properties are missing?
Source trail: 01-consistent-hashing, 08-database-sharding-and-partition-keys, 08-sharding-hot-keys-and-rebalancing. See the source map for consolidation and reference-retained material.