The copy finished, then the old owner accepted a write
A hypothetical shard migration copies every key from A to B and reports matching counts. The router flips to B. One client with an old map writes to A; another reads from B and never sees it.
The data copy was complete. The ownership transfer was not.
A snapshot has a position
Take a consistent snapshot at log position , then apply changes after to the target. Copying live rows with no position boundary can miss updates or resurrect deleted rows. Retain the change log long enough for the transfer; compare versions/checksums at a defined position, not unrelated wall-clock instants.
B has caught up at position 900. A accepts position 901 immediately before the directory update. Is ālag zeroā enough to justify switching ownership?
Transfer authority before trusting routers
A simplified single-writer move proceeds as follows:
| Phase | Serving behavior | Recovery obligation |
|---|---|---|
| Copy snapshot | A remains writer | Abandon target safely if copy fails |
| Tail and verify | A remains writer | Preserve source log and target progress |
| Fence A | A rejects/retries writes for the moving unit | Record durable handover state |
| Catch final boundary | No uncontrolled new source writes | B proves it applied the fence/boundary |
| Commit new owner/epoch | B accepts valid new-generation writes | Old maps redirect or fail; A cannot resume writing |
| Cleanup | B serves | Keep evidence needed by readers, rollback, and recovery |
This is a design pattern, not a drop-in protocol for every engine. Every serving owner validates the epoch or equivalent ownership state. Otherwise a stale router bypasses your carefully committed directory.
A timeout during the move may hide a successful source write. Retry the same operation identity at the new owner; do not create a second intent. A post-cutover rollback is another transfer that includes B's new writes, not simply pointing traffic at A's old snapshot.
Movement competes with the work it rescues
Throttle snapshot reads, writes, and log catch-up below spare I/O and network capacity. Track bytes remaining, tail lag, write rejection during fencing, stale-route errors, and target capacity. A rebalance that saturates its donors can trigger further failures and more movement.
Merkle comparisons can locate divergent ranges; counts alone miss values that differ while cardinality matches. Hinted handoff and read repair serve other reconciliation paths but do not replace a migration's ordered publication boundary.
The query that cannot name a shard
A cross-shard query waits for multiple participants. If each shard independently has probability of exceeding a latency threshold, the chance at least one does is over shards. Real failures may be correlated, so the formula is an illustration, not an SLO guarantee.
Use query-local indexes, co-location, a reporting copy, or an explicit partial-result contract when appropriate. Cross-shard joins and transactions need a separate plan; splitting a tenant can improve capacity while moving its invariant across ownership boundaries. Capacity sizing should consider both load and the largest unit you can move or restore in the recovery budget.
Mental model: live migration is a state transfer plus a durable ownership handover. Copy completion proves only the first half.
Transfer: dual writes
During migration the application writes A and B independently. Both usually succeed. What ordering information and failure recovery are still missing?
Source trail: 01-consistent-hashing, 08-database-sharding-and-partition-keys, 08-sharding-hot-keys-and-rebalancing. See the source map for consolidation and reference-retained material.