The lease expired; the old worker did not

Worker A has a ten-second lease to generate invoices. It reads the current invoice state, pauses for thirty seconds, then resumes. Worker B acquired the expired lease and completed the new invoice run while A was paused.

In this hypothetical incident, both workers followed the lock API. A now overwrites B's work. Increasing the lease to a minute only changes how long the pause must be.

Expiration is not cancellation

That is the job of a fencing token: an ordered generation attached to every protected operation. The resource remembers the greatest accepted generation. It never accepts a lower one.

Put the fence where the effect happens

The replacement worker first raises the resource's fence, then reads the state it will modify. A token is not useful merely because it is stored in the lock service.

Here is a SQL transaction sketch. It assumes a permanent job_fence row and that every invoice writer uses this path. Parameters are bound values, not SQL string interpolation.

sqlCopy
BEGIN;
UPDATE job_fence
SET generation = :generation
WHERE job_id = :job AND generation <= :generation
RETURNING generation;
-- Application must ROLLBACK if no row was returned.
-- The row lock remains held until COMMIT.

INSERT INTO invoices(job_id, invoice_id, amount)
VALUES (:job, :invoice, :amount);
COMMIT;

After B commits generation 8, A's generation 7 updates zero rows and must abort. If A already holds the fence row's lock, B waits for that transaction; once B raises the fence, no earlier generation can commit a new protected transaction through this path. There is a precise serialization point, not a promise that every action stops at the wall-clock instant of lease expiration.

Equal generations are allowed because one owner may perform several operations. This also means fencing is not duplicate suppression: repeating an INSERT or a charge within generation 8 still needs a unique operation key or an idempotent effect.

A dangerous variant reads generation, compares it in application code, then writes without a condition. That recreates the original check-then-act race. Another places the fence in one database and the invoices in a different one; the local transaction no longer covers both.

Who issues the numbers?

A safe issuer must not reuse a generation that a resource could still accept. A consensus-backed revision can provide this ordering within its documented lifetime. A counter on an asynchronously replicated cache may move backward after failover. Restoring an old backup can move even a previously durable counter backward.

Deleting the lease record must not delete the permanent generation. A cleanup TTL is not an ownership deadline, and re-creating a counter at one is not a fresh start to a resource that has seen eight. After disaster recovery, use a provably new ordered epoch or fence all old generations through a durable authority. Adding an arbitrary large constant is not a proof unless its bound exceeds every possible issued value.

If the resource cannot enforce a generation, the lease alone cannot provide this safety property. For a payment provider, use its operation idempotency contract and reconciliation. For a conditional object replacement, an ETag compare-and-set can reject a stale version, but it is not a numeric lease epoch or an authorization check.

A renewal response arrives too late

A renewal extends permission at the authority. Network time has already passed before the client receives the response. Counting a full lease from response arrival can let the holder act too long.

Use conservative local deadlines based on the renewal's send time and a monotonic duration clock, with an explicit margin and bounded drift assumptions. Stop issuing protected work when renewal is uncertain; resource fencing still handles old requests. Comparing a timestamp written by one host against another host's wall clock additionally requires a bound on clock offset.

An alternative lease protocol observes a record version, waits a full lease duration, and conditionally claims only if that version remains unchanged. It avoids comparing absolute clock readings, but still needs timing assumptions about duration and must handle paused holders through fencing.

Renew and release only when owner identity and generation match. Otherwise a late release from A can delete B's valid lease. Back off with jitter on acquisition contention. For many queued contenders, a predecessor watch can reduce the herd, but watchers must recheck authority after reconnecting.

Recovery is part of ownership

Suppose B raises the fence and crashes before producing any invoice. The lease eventually permits C, but who notices unfinished work? Record the claim and progress durably; a sweeper can find old unfinished claims and requeue them conditionally. Use operation IDs to avoid repeating completed invoice effects.

Keep the fenced unit small enough to finish or checkpoint. A lease held over a multi-hour workflow combines slow recovery with difficult external effects. Prefer conditional updates for a single record, unique constraints for one-time creation, and optimistic version checks when conflicts are rare. Use a lease when coordinating ownership actually reduces work or establishes a required authority boundary.

Mental model: a lease lets a successor start. A fence stops an obsolete predecessor from damaging the resource. Idempotency stops the current owner from repeating an effect. They solve three different failures.

Transfer: the protected file and the unprotected email

---

Source trail: 06-distributed-locks-and-leases, 03-leases-fencing-tokens-and-distributed-locks. See the source map for consolidation and reference-retained material.