Idempotency & Effectively-Once Processing

Part 0. Start here

Tapped once, charged twice

A buyer taps Pay once for order 42, $30.00. The request reaches the load balancer 0.02 s later and then waits in a busy server's queue. At 10 s the phone gives up, asks "did it go through?", hears "not found", and the buyer taps again. That second request charges the card at 10.28 s. At 12.00 s the first request finally leaves the queue, and at 12.28 s it charges the card too. Tapped once, charged twice: $60.00.

Nobody wrote a bug here. Every part did something reasonable. Something repeated the request, and nothing could tell that the repeat was the same payment.

One tap, eight systems

The tap crosses eight systems before the money is settled, and then a ninth tells the buyer:

phone → load balancer → payment API → card processor → ledger → outbox relay → queue → worker, and the worker hands a push notification to a provider (Apple's or Google's push service).

The card processor is the PSP (payment service provider, for example Stripe or Adyen): the outside company that actually charges the card.

Every hop can lose an answer, and at every hop something may send the same thing again:

The number of unclear outcomes is not small. The payments loop assumes 50,000 payments a day and a 0.1% PSP timeout rate: 50,000 × 0.001 = 50 payments a day whose outcome we don't know, every day.

The big picture

Synthesizing vector architecture diagram...

What to notice: one ID chain runs through the whole path. The phone's key K1 names the payment pay_7; pay_7 names the PSP charge and the event ev_7; ev_7 is what the queue, the worker's inbox and the notification record recognise. The ledger row is the durable guard: it exists for as long as the payment does.

Everyone remembers for a different time

Many systems offer to "remember a repeat" for you. Each one remembers for its own length of time:

None of the first five is wrong. Each one is shorter than the last row.

What you'll be able to do after this page

1. Tell a done, a not-done and an unknown outcome apart, and spot a success disguised as an error (Part 1).
2. Design an idempotency key: who makes it, what it covers, and how it is tied to the request's meaning (Part 2).
3. Claim a key, record the intent before the side effect, and answer every kind of repeat, including one from a crashed or paused owner (Part 3).
4. Size a key's lifetime against the retry horizon, and stop an original that outlives its retry (Part 4).
5. Make writes idempotent by design, and put the condition on the right record (Part 5).
6. Explain what a queue's or a log's dedup covers and what the consumer's inbox must still catch (Part 6).
7. Send a notification once per intent, and make an unavoidable repeat harmless (Part 7).
8. Explain why a stream processor with exactly-once state still needs an idempotent sink, and what a restore must clean up (Part 8).
9. Say what async multi-Region replication does to a claim, and fix it (Part 9).
10. Compare idempotency keys with retry-free designs, locks, two-phase commit, sagas and broker exactly-once, on equal terms (Part 10).
11. List every place along a request's path where a repeat is born, and the ID and guard for each (Part 11).
12. Map all of it to AWS services, and name the look-alikes (Part 12).

You may have arrived from step 1.1 ("The Buyer Was Charged Twice") or 1.2 ("The PSP Call Timed Out. Did It Charge?") of the payments loop, step 1.2 of the mobile stock trading loop ("The Network Dropped After I Tapped Buy, So I Tapped Again"), step 2.4 of the ad-click aggregation loop ("A Worker Crash Replayed a Minute and Double-Billed") or step 1.3 of the notification loop ("A Retry Sent the Confirmation Twice"). This page is the "why" behind all of them.

---

Part 1. The core mental model

Before any mechanism, we need words for what a caller knows after it sends a request, and for what "once" can honestly mean. Everything later on the page follows from these.

Three outcomes of a request

A caller that sent a request ends up in one of three places:

The third row is the dangerous one. A timeout says nothing about the server: the request may never have arrived, may have been done with the reply lost, or may still be running. Treat it as a failure and a buyer is charged with no order. Retry it blindly and the buyer may be charged twice.

The same holds inside a machine: the Write-Ahead Log loop primitive (Part 3) shows that an fsync error leaves the write's fate unknown, so the client's retry must be idempotent. This page is how.

Synthesizing vector architecture diagram...

What to notice: the caller sees the same thing, a timeout, in the second and third cases. Only the server knows which one happened, so the caller must ask in a way that is safe either way.

Success disguised as an error

Sometimes a retry does get a reply, and the reply is an error even though the first attempt worked:

The rule: an error on a retry is a question, not an answer. Read the resource before you decide.

Which HTTP methods are already safe to repeat

RFC 9110 (section 9.2.2) defines PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) as idempotent: sending one twice has the same intended effect as sending it once. POST is not, and "create a payment" is a POST. So a POST that creates something needs a key. Stripe, for example, accepts idempotency keys on every POST and says they have no effect on GET and DELETE, which are idempotent by definition.

Idempotent by definition is still about one request in isolation. Two different PUTs arriving in the wrong order is a different problem (Part 5).

Downstream windows

When we are the caller, the callee's own memory decides how we resolve an unknown. The PSP in our example keeps each key and its saved result for at least 24 hours, and compares the parameters of every repeat with the original:

• Inside the window: repeat the call with the same key and identical parameters. You get the original result back. Stripe saves the first result "regardless of whether it succeeds or fails", including 500 errors. So a saved error keeps coming back for that key, and the way forward is to look the payment up.
• After the window: the key may have been removed, and a repeat is then a new request. Never replay an old key; look the payment up by our own reference.

At-most-once, at-least-once, effectively-once

"Exactly-once" is only true of an effect inside a store that records the key together with the effect, in one transaction. A delivery over a network can't be exactly-once: the sender can never be sure the receiver got it. So we say it hop by hop: "the charge is exactly-once at the PSP, the credit is exactly-once in the merchant database, the push is at-least-once with a replace ID".

Three rules the design never breaks

Not this dedup

The word "dedup" also names mechanisms that have nothing to do with repeated requests. This page doesn't teach them:

• Content-addressed storage dedup: storing identical bytes once (the Drive loop's chunk store).
• URL seen-sets and Bloom filters: skipping pages a crawler has already fetched (the web crawler loop).
• Alert grouping: folding many alerts with the same labels into one page.
• Request collapsing: a cache or CDN turning many concurrent misses for the same object into one origin fetch. That merges reads that happen at the same moment, not a write repeated later.

The example we follow

Real payment systems handle millions of requests, which is too many to watch. So one story runs through the whole page: one payment, retried at every hop. Every trace on this page was produced by running a private reference simulator of this setup (the phone, a load balancer, two API servers, the key store, the ledger, the PSP, the resolver, the webhook, the relay, the queue, two workers, the push provider and a stream job), not worked out by hand.

The story in six beats: the reply is lost (Parts 2 to 4), the queue sees it twice (Part 6), the worker dies after acting (Parts 6 and 7), the stream job restores (Part 8), the buyer comes back two hours later (Part 4), and a Region fails in the middle of replication (Part 9). Each Part shows its own slice; the whole event table is in Part 13.

---

Part 2. Idempotency keys

The server has to recognise a repeat. To do that, every copy of the request must carry the same name, and the name must mean "this intent", not "this attempt" and not "this order". This Part decides who makes the name, what it covers, and how it is tied to the request.

Who makes the key, and when

The key is born before the first send and written to the app's local journal, so an app that is killed mid-send still has it when it starts again.

What the key covers

The key is only unique within a scope: the endpoint, the authenticated caller, and the key. Ours is payments:u1:K1. Two reasons:

• Collisions across callers can't hurt. Two buyers who happen to send the same key never see each other's answers.
• Keys can't be used to probe. A caller can only ever replay answers to its own requests.

Here the caller is the buyer. In a server-to-server API, such as a merchant calling a payments platform, the caller is the merchant.

Binding the key to the request's meaning

A client bug can reuse a key for a different request: the same K1 with $35.00 instead of $30.00. Replaying the $30 answer would be wrong, and so would running the $35 payment. So the server stores a fingerprint with the key: a hash of the request's meaning, not of its bytes.

textCopy
SCOPE(request)       = endpoint + ":" + authenticated caller + ":" + Idempotency-Key header
FINGERPRINT(request) = hash of the canonical meaning fields, in a fixed order:
                         order id, amount in minor units, currency, payment method
                       -- not the raw body: a reordered JSON body, a new client
                       -- timestamp field or an extra header is the same request

on a repeat with the same scope:
  stored fingerprint == new fingerprint  -> the same request: answer it (Part 3)
  stored fingerprint != new fingerprint  -> 422 "key reused with a different request", nothing runs

Side trace 3d, from the simulator: A2 is sent at 10.00 with K1 but an amount of 3,500. Its fingerprint H2 differs from the stored H1, so the server answers 422 and nothing changes: no claim, no PSP call. (Some APIs use 409 here; Stripe errors on mismatched parameters.) The same run confirmed the other direction: a body with the same fields in a different order, plus an extra client timestamp field, gives the same fingerprint H1.

Keys must not contain personal data (Stripe's advice: no email addresses or personal identifiers). They end up in logs, tables and support tools.

Derive what others must re-create; store what you send

The key protects one hop. Every later hop needs its own ID, and the question is how each one is made:

• Derive an ID if someone else may have to re-create it. A resolver replaying a charge, another Region taking over, or a re-run parser must produce the same ID as the first attempt, or the next hop sees a stranger. So pay_7 is computed from the scope, and the PSP key is pay_7:charge. If a second attempt is legitimate (a failover to a second PSP after a definite failure), it gets a new suffix, one per attempt per PSP: pay_7:attempt-2:psp_b. The cost: a derived ID leaks structure, so hash it if it must be opaque.
• Store an ID if it is written down before its first send. An outbox row's event ID (Change Streams & the Transactional Outbox, page 05) is stored in the same transaction as the payment, and every re-send reads it back from the row. So a random ID would do. Ours is derived anyway, as pay_7:SUCCEEDED, because in Part 9 another Region has to re-create the event.
• Use natural keys where they exist: a click_id, a trade_id, a chat app's client_msg_id.

Synthesizing vector architecture diagram...

What to notice: the notification intent's key (ev_7 + channel) and the replace ID (pay_7) are two different IDs with two jobs. The intent key stops us from sending twice; the replace ID tells the phone that a second copy is the same notification.

Standards: there is no standard idempotency header yet. The IETF draft "The Idempotency-Key HTTP Header Field" reached revision 07 and is now an expired draft. The common practice this page describes (a client-made key in an Idempotency-Key header, scoped to the caller, bound to the request) is what payment APIs do in practice.

Trace: the tap and the claim

The request on the wire:

httpCopy
POST /payments HTTP/1.1
Host: api.example.com
Authorization: Bearer <u1's token>
Idempotency-Key: 0192f0c4-5b7e-7c1a-9d3e-2f4b8a6c1e07
Content-Type: application/json

{"order_id": "42", "amount": 3000, "currency": "USD", "payment_method": "pm_1"}

---

Part 3. The key record and the durable guard: claim, record, run, answer

Two copies of the same request can run at the same time. A finished request must answer its repeats the same way. And a crash right after the PSP call must leave something behind that says "a charge may be out there". The naive answers fail: "check whether a payment exists, then create one" races (two copies both see nothing), and "write the payment row once the PSP answers" leaves no trace if we crash in between. The fix is the rule the Write-Ahead Log loop primitive is built on: log first, then act. Here, record the intent before the side effect.

Record the intent before the side effect

textCopy
HANDLE(request)                                            -- on the API server
  1. if now > arrival stamp + deadline: reject; run nothing          (Part 4)
  2. claim idem[scope] only if it is absent (or its expires_at has passed):
       state IN_PROGRESS, fingerprint, owner = this request,
       in_progress_until = arrival stamp + deadline
     claim failed -> ANSWER A REPEAT (below)
  3. payment_id = derive(scope)
     insert the payments row: payment_id, caller, key, fingerprint, state PENDING
       -- UNIQUE (caller_id, client_key); this happens BEFORE any external call
     unique conflict -> read that row and answer from it (a VOID row -> 409; another fingerprint -> 422)
  4. call the PSP with key payment_id + ":charge"
  5. the reply arrives in time:
       one ledger transaction: PENDING -> SUCCEEDED (compare-and-set),
                               journal entries, outbox row
       complete idem, only if owner = me: store the final answer
     the deadline comes first:
       PENDING -> UNKNOWN (compare-and-set)
       complete idem, only if owner = me: store a POINTER to payment_id
  6. completion failed (the owner changed) -> read idem and answer what it holds

The claim, as a DynamoDB conditional put (times in the story's seconds; real items use epoch time):

jsonCopy
{
  "TableName": "idem",
  "Item": {
    "pk": { "S": "payments:u1:K1" },
    "state": { "S": "IN_PROGRESS" },
    "fp": { "S": "H1" },
    "owner": { "S": "A1" },
    "in_progress_until": { "N": "5.02" }
  },
  "ConditionExpression": "attribute_not_exists(pk) OR expires_at < :now",
  "ExpressionAttributeValues": { ":now": { "N": "0.02" } }
}

The durable guard, in the ledger:

sqlCopy
CREATE TABLE payments (
  payment_id   TEXT PRIMARY KEY,     -- derived from the scope: pay_7
  caller_id    TEXT NOT NULL,        -- u1
  client_key   TEXT NOT NULL,        -- K1
  fingerprint  TEXT,                 -- H1; NULL only on a VOID row
  amount       BIGINT,               -- 3000; NULL only on a VOID row
  currency     CHAR(3),
  state        TEXT NOT NULL,        -- PENDING, UNKNOWN, SUCCEEDED, FAILED, VOID
  psp_charge   TEXT,                 -- ch_1 once known
  UNIQUE (caller_id, client_key)     -- the durable guard: lives as long as the row
);

The states of a key record

Synthesizing vector architecture diagram...

What to notice: "absent" includes "expired": a reader treats a record whose expires_at has passed as absent, whether or not it has been deleted yet. When the key record expires, the ledger row (including a VOID row) is still there, and it is what answers from then on (Part 4).

What gets stored: final answers and pointers

A pointer matters because the state behind it changes. If we stored 202 PROCESSING byte for byte, a retry an hour later would still hear "processing" about a payment that finished long ago. When the payment does become final, the resolver also rewrites the stored answer to the final one.

Trace: a concurrent duplicate, a lost reply and a retry

Snapshot P1, right after event 5:

Synthesizing vector architecture diagram...

After event 5: the card has been charged, the ledger knows a charge may be out there (PENDING), and the phone knows nothing.

Snapshot P2, after event 9 (t = 10.00):

Every answer to a repeat

A crashed or paused owner

A request that claimed a key can crash, or freeze (a long garbage-collection pause, a paused VM), and leave the record IN_PROGRESS. A later repeat may take over, with three safeguards:

1. One clock decides expiry. Either a single SQL database's now() for rows in that database (the payments loop does this), or the claimant's own clock plus a margin larger than the worst clock offset between machines. We use 1 s. Leases, Fencing Tokens & Distributed Locks (Part 5) explains why the margin must exceed the offset.
2. The takeover is conditional on the owner still being the one the claimant read, and it checks the durable record first, because the old owner may have got further than the key record says.
3. Completion is conditional on owner = me. A paused owner that wakes up can't overwrite the new owner's answer. It reads the record instead.

From the simulator, three variants of the main line:

In 3c, a paused owner is a zombie: it still believes it owns the request. Its PSP call, had it made one after waking, would have been harmless only because the PSP key is derived (pay_7:charge), so the PSP replays instead of charging. Stopping a zombie's writes to your own stores is the fencing-token problem; the Leases page owns it.

In our simulator the resolver only picks up UNKNOWN payments, so in 3b the PENDING row waits for the phone's retry. A production resolver also sweeps PENDING rows whose deadline plus margin has passed, so a crashed owner's payment is settled even if the phone never comes back; its compare-and-set on the state makes it safe to race with a takeover.

One store or two, and what if the key store is down

Finish what the first attempt started

A repeat that is recognised as a duplicate must not stop at "already done". The first attempt may have died between two steps, so the duplicate finishes every step the first attempt may not have finished. The chat loop's retry re-sends the acknowledgement and re-publishes the message; in our story, A3 at 7,200 rewrites the expired key record, and in Part 7 the second worker sends the notification the first may never have sent.

---

Part 4. Time: how long to remember, and the late original

The key record is a window: in our example it forgets K1 an hour after completing. Phones retry for longer than that. And there is a second, sneakier timing problem: an original request that runs after its client has given up on it.

When a late original can hurt

A late original can hurt only when the client stops using a key and makes a new one, as when the buyer taps Pay again. Retries with the same key are already safe: whichever copy comes first claims the key, and the rest are answered. The danger is a first request still sitting in a queue somewhere when the second key starts its own payment. Two tools close that gap: a deadline on the server, and a "did it happen?" call that voids the old key.

The retry horizon

The retry horizon is the longest time after the first send at which a retry can still arrive:

Formula 1:

$$\text{how long the guard remembers a key} \;\ge\; \text{retry horizon}$$

Our key record remembers K1 until 3,605.02. The buyer comes back at 7,200: a horizon of at least 7,200 s against a window of 3,600 s, so the window fails. The durable guard remembers K1 as long as the payment row exists, so the guard passes. Every window on the Part 0 table (1 minute to 24 hours) is shorter than a phone's horizon. That is fine, as long as each window is treated as an accelerator and correctness sits on the durable guard.

Expired means expires_at < now, checked in the read path. DynamoDB's TTL deletes expired items "within a few days of their expiration time", so an expired item can still be there. TTL is cleanup, never the window.

Trace: the buyer comes back

Snapshot P6:

Synthesizing vector architecture diagram...

What to notice: the key store forgot K1 an hour ago and it didn't matter: the ledger row answered, and nothing reached the PSP.

Branch X, the same event without the durable guard. Take away the unique key and mint the payment ID at random, and the simulator gives: the expired key looks new, the server creates pay_9, and its PSP key pay_9:charge is new to the PSP, which charges ch_2 at 7,200.28. The buyer has paid $60.00 for one order, and was told "succeeded" for both.

The deadline

In the opening story, A1 reached the load balancer at 0.02 and waited in api-1's queue until 12.00, while the phone gave up at 10.00 and the buyer tapped again with a new key. The in-progress lease can't help: A1 hadn't claimed anything yet. What stops it is a deadline measured from when the request arrived:

• The load balancer stamps the arrival time on the request.
• The server refuses to start a request whose stamp is more than 5 s old, and when a running request reaches 5 s it starts no new side effect and commits nothing new. A call already sent becomes UNKNOWN, never "abandoned".
• The server checks the stamp on its own clock, so it needs a margin for the offset between the load balancer's clock and its own (the Leases page, Part 5).

Formula 2:

$$\text{server deadline} + \text{longest wait before the arrival stamp} \;<\; \text{client timeout}$$

With the stamp at the load balancer, the only wait before it is the network trip: 5 + 0.02 = 5.02 s < 10 s. By the time the phone's timer fires at 10 s, any copy of A1 has either claimed K1 already (so a "did it happen?" call sees it) or will never run. If the stamp is taken only when a server thread picks the request up, the wait before the stamp includes the load balancer's, the connection and the thread-pool queues. In our story that wait was 12.00 − 0.02 = 11.98 s, and 5 + 11.98 = 16.98 s > 10 s. Then you must bound those queues too: shedding requests that waited more than 2 s gives 2 + 5 = 7 s < 10 s.

On AWS: an Application Load Balancer adds an X-Amzn-Trace-Id header whose ID fields carry the epoch time in seconds (8 hex digits). If the client already sent a Root field, the load balancer inserts a Self field and keeps the client's Root; read the load balancer's own field, not one a client could send. The time is in whole seconds, so it can read up to 1 s early, which only makes the deadline stricter. Also note that the load balancer won't end a slow request when the phone gives up: its idle timeout defaults to 60 s. The server's own deadline is what stops the work.

Synthesizing vector architecture diagram...

What to notice: in the second half, two separate rules each stop A1. The deadline rejects it before it claims anything; and even without the deadline, A1's claim would find K1 as VOID and get 409. The simulator ran both: with only the void, A1 gets 409 at 12.00; and after the key record's hour has passed, A1's insert hits the VOID row under UNIQUE (u1, K1) and still gets 409, with no PSP call.

Resolve, don't just look

The app's "did it go through?" call must change the state, not just read it:

textCopy
RESOLVE(caller, key)                          -- the app's "what happened to my request?"
  1. read idem[scope] and the payments row for (caller, key)
  2. found:  return its state ("still confirming" for PENDING or UNKNOWN; else the final answer)
  3. absent: insert-if-absent in BOTH stores:
               idem[scope] = VOID
               a payments row (caller, key, state VOID) under UNIQUE (caller_id, client_key)
             return "not found, and it never will be": the app may now make a new key
  4. a late original that arrives later finds VOID and gets 409; nothing runs

The insert-if-absent races fairly with a late original's claim: whichever gets the unique row first wins. If the original wins, the resolve call reports "in progress", and the app keeps waiting instead of making a new key.

One more hole: an app reinstall loses the journal and its keys. The trading and news-feed loops keep, per device, the last accepted action ID, so a reinstalled app can ask "what was my last order?" before it lets the user act again.

When a compensation times out

The same rules apply to the steps that undo a payment. Suppose an order fails after the charge and we refund. The refund call to the PSP times out. That is an unknown, not a failure: record it as such, and retry it with its own derived key, pay_7:cancel:refund, so every retry is the same refund. Keep retrying with backoff; after the retry budget, also alert a human, but keep the state UNKNOWN rather than guessing. The nightly reconciliation against the PSP's records has the last word. Nothing is lost permanently because the intent to refund was recorded before the first call and every attempt carries the same key. This answers the first question of the drill The Flight Booking That Charged Without a Seat; Part 10 answers the second.

---

Part 5. Idempotent by design: absolute writes and conditions on the durable record

A key table in front of every write is heavy. Many writes can be made harmless to repeat by their shape alone, with no key table at all. And where a condition is needed, it matters which record carries it.

Write the result, not the change

State transitions, and the late webhook

Our payment moves only along allowed transitions (PENDING → UNKNOWN → SUCCEEDED, and so on), and every move is a compare-and-set. That makes a late or repeated answer harmless:

The order doesn't matter. The simulator also ran the webhook first, at 10.50: it made the same move (UNKNOWN → SUCCEEDED, outbox row ev_7), a second delivery of evt_1 at 10.90 was dropped as a duplicate, and at 11.00 the resolver found nothing UNKNOWN to do. The stored answer was still the pointer, so it rendered SUCCEEDED without being rewritten.

Versions from one counter

Idempotence alone doesn't cover order: an old message arriving after a newer one. Each copy is harmless on its own, but applying the old one last undoes the new one. The fix is a version from one counter, compared on write.

Branch V is the mobile news feed loop's case (step 2.2). Device D1 keeps a counter in its journal. It sends LIKE (counter 41), then UNLIKE (42); the LIKE is resent and arrives last. From the simulator:

Never order by a timestamp made on the phone. Phone clocks are set by users, carriers and time daemons, and they jump. The counter must also outlive anything it's stored next to: keep the per-device high-water mark on an item that never expires.

Condition on the durable record

A condition protects you only if it sits on a record that exists for as long as the effect does. Put it on an item that can move or expire, and a retry that comes after the move re-creates the effect.

Branch M is the email loop's case (round 2, delivery): a parser, fed by SQS, stores message m9, and SQS redelivers the parse after the user has moved m9 from Inbox to Archive. From the simulator:

Synthesizing vector architecture diagram...

What to notice: both writes are "put if absent". The left one conditions on a folder item that the move deleted, so the retry passes and m9 reappears in Inbox. The right one conditions on the locator M#m9, which never moves and becomes a tombstone when m9 is deleted, so the retry fails even after a delete (the simulator's second retry, after deletion, also did nothing).

The same rule appears across the loops: the payments loop's retry conditions on the payment row, not on the key store's item that expires (Part 4); YouTube's "ready" handler conditions on the video's state, not on a job token that MediaConvert forgets after a minute. When a condition fails, read what's there and answer from it; a failed condition on a retry is usually a success disguised as an error (Part 1).

---

Part 6. Queues and logs: dedup windows, redelivery and the inbox

Between the ledger and the worker, repeats are born in two places: the producer sends the same event twice (a relay that crashed after publishing), and the broker delivers the same message twice (a worker that died before deleting it). Brokers help with the first, only for a while. The second is always the consumer's job.

Two places repeats are born

Synthesizing vector architecture diagram...

What to notice: the producer-side window catches only fast repeats; everything it misses, and every redelivery, ends at the consumer's guard. The guard is the one that can't be skipped.

The FIFO dedup window, and what keeps the ID stable

Facts from the SQS documentation:

• A FIFO queue drops a message whose MessageDeduplicationId it has seen in the last 5 minutes: the send is "accepted successfully" but the message isn't delivered again.
• SQS keeps tracking the ID even after the first copy was received and deleted.
• Content-based dedup uses a SHA-256 hash of the message body (not its attributes) as the ID. So a body that carries an attempt count or a send timestamp makes every repeat look new.
• With high-throughput mode, the dedup scope is the message group, so a re-send must keep the same group ID as well as the same dedup ID.
• SNS FIFO topics have the same five-minute window for publishes.

The dedup ID is only as stable as the process that picks it. Ours is the event ID stored in the outbox row, read back by every re-send. The outbox relay (Change Streams & the Transactional Outbox, Part 3: claim, publish, delete) re-sends an undeleted row with the ID it finds there.

Trace: the relay crashes twice

Snapshot P3, after event 14:

Synthesizing vector architecture diagram...

After event 14: two sends, one message. The undeleted outbox row will be sent again, and nothing upstream can stop that.

The idempotent producer, and why a restart defeats it

A log such as Kafka offers an idempotent producer (enable.idempotence, on by default in current clients). The broker gives the producer an ID, and the producer numbers its batches per partition; a batch with a sequence number the broker has already written is dropped. Kafka's own description: "the producer will ensure that exactly one copy of each message is written in the stream". The catch is the scope: one producer session.

Branch K, from the simulator. The relay publishes ev_7 to a log instead of a queue:

Transactions in the log

A transactional ID makes the producer's identity survive restarts: Kafka documents that it "enables reliability semantics which span multiple producer sessions", and a new session with the same ID first completes or aborts whatever the previous session left open. Output records and consumed offsets can then commit together. Readers must set isolation.level=read_committed; the default, read_uncommitted, also returns records from aborted transactions.

Branch K again, with transactional ID relay-1:

• Session 1 commits ev_7, then crashes before deleting the outbox row. Session 2 starts (epoch 1), re-reads the row and commits it again. A read_committed reader sees ev_7 twice. The transaction covered the log; the outbox row that caused the re-send is outside it.
• Session 1 crashes before committing. Session 2's start aborts session 1's open record; read_committed sees ev_7 once, while read_uncommitted sees it twice.

So a transactional log removes some repeats, but an outbox re-send is a new record. Only the consumer's guard catches it.

The consumer's two guards

A consumer that applies an effect must recognise a repeat itself. Two ways:

1. An inbox keyed by event ID, written in the same transaction as the effect. This page's choice, for effects like credits that aren't naturally ordered.
2. Apply by version, for an ordered stream of changes to one key (a cache or an index following a row). Change Streams & the Transactional Outbox (Part 3) covers that.

The worker's transaction:

sqlCopy
BEGIN;
INSERT INTO processed (event_id) VALUES ('pay_7:SUCCEEDED')
  ON CONFLICT (event_id) DO NOTHING;             -- 1 row: first time; 0 rows: a repeat
-- only if 1 row was inserted:
INSERT INTO credits (merchant_id, cents, event_id) VALUES ('m1', 3000, 'pay_7:SUCCEEDED');
INSERT INTO notif (event_id, channel, state) VALUES ('pay_7:SUCCEEDED', 'push', 'PENDING')
  ON CONFLICT (event_id, channel) DO NOTHING;
COMMIT;
-- then, whether or not a row was inserted: if notif is PENDING, send the push (Part 7)

ON CONFLICT DO NOTHING matters in PostgreSQL: a plain INSERT that hits the unique key raises an error, and the error aborts the whole transaction. With DO NOTHING, the insert reports 0 rows and the worker decides what to skip.

Trace: the worker dies after acting

Synthesizing vector architecture diagram...

What to notice: the inbox stopped the second credit, and the notification record made w2 finish the one step w1 may not have finished. Whether the first push ever reached the phone, w2 can't know, so it sends again with the same replace ID.

Snapshot P4, after event 18:

Synthesizing vector architecture diagram...

After event 18: one credit, one notification on screen, and one outbox row still waiting to be sent a third time.

A re-send carries the original ID

Webhooks from outside are the same case. The PSP delivers each webhook at least once (Stripe retries for up to three days), so the webhook handler keeps an inbox keyed by the sender's event ID (evt_1 in event 13) and applies each event as an allowed state move.

Shared limits on this path. The FIFO message group is the merchant, so one busy merchant's events are serialised, and a message in flight (up to its 30 s visibility timeout) holds back the rest of its group. The inbox shares the consumer database's write path with the credits, and if the consumer also keeps a running balance per merchant, every credit writes that one item: a DynamoDB partition gives at most 1,000 write units a second, and a SQL row takes one lock at a time. Plan for the biggest merchant, not the average one.

---

Part 7. Side effects you can't take back: notifications

A push, an email or an SMS leaves our systems the moment we send it. It can't join our transaction, and the providers take no idempotency key. So "send exactly once" is impossible: the honest goals are send once per intent, and make an unavoidable repeat replace instead of stack.

Why providers can't help

FCM's collapse_key looks like the answer and isn't. It lets FCM keep only the latest message of a group while a device is offline, "so that only the last message gets sent when delivery can be resumed", and FCM keeps at most four different collapse keys per device. It never touches a notification already on the screen.

Intent, send, mark

textCopy
IN THE EFFECT'S TRANSACTION (the worker, Part 6)
  insert-if-absent notif[event_id, channel] = PENDING

AFTER THE COMMIT, and again on every redelivery of the event:
  if notif[event_id, channel] is SENT: skip
  if PENDING: send with replace id = the thing announced (pay_7)
              then set notif[event_id, channel] = SENT

The intent row makes "should I send?" a question with a durable answer. A crash between the send and SENT is the one case that sends twice, and the replace ID makes that second copy land on top of the first.

Branch N, from the simulator. The worker has no notification record and no replace ID, so it pushes on every delivery. After event 18 the phone shows two stacked notifications; after event 19's redelivery, three. With a replace ID but still no record: one notification shown, but three alerts. The record removes the repeats it can; the replace ID hides the rest.

Logical duplicates and windows

Some duplicates are not repeats of one request but the same message from different requests: a billing system that raises "your invoice is overdue" twice in a day. The notification loop (step 2.3) handles these with a producer-chosen dedup key (a fingerprint of recipient, template and business key) stored with SET key 1 NX EX <window> in Valkey or Redis:

• The first SET NX succeeds and the message goes out; a second within the window finds the key and is dropped.
• If the cache is down, fail open: send anyway. A duplicate reminder is better than a lost one.
• Run that cache with noeviction and alarm on memory: an evicted key silently turns into a duplicate.
• Count per-user caps once, at accept, after dedup, so a dropped duplicate doesn't use up the user's allowance.

Email: the send record, and choosing your failure

The email loop (steps 1.6 and round 2's delivery) keeps a send record keyed by the request's Idempotency-Key. It stops our repeats: a client that sends the same request twice finds the record and gets the same answer. It can't stop the provider's: if our call to the email provider times out, the email may or may not have gone. From the simulator, a request sent twice by its client, whose provider call timed out after the provider had accepted it:

Third parties with no key

For any outside call that takes no key: look it up by your own reference if the provider offers a search; otherwise accept a rare duplicate, or don't retry after an unknown, and choose per message type. A token that a client library generates for you inside one call covers only that library's retries of that call; if your process restarts and calls again, it is a new token.

Ready means notified

Branch R is the YouTube loop's rule, from step 1.5 ("Is My Video Ready Yet?"). A worker that marks the event DONE, then crashes before sending, leaves a trap: the retry finds DONE and, if it only acknowledges, the buyer is never told. From the simulator: after the crash at 14.50 and the redelivery at 44.50, the phone shows no notification, although m1 was credited. The rule: a detected duplicate checks the notification record and finishes it. "Done" isn't done until the notification is sent.

---

Part 8. Streams: exactly-once state plus idempotent sinks

A stream processor such as Apache Flink advertises exactly-once. It's true, for its state. What it writes to the outside world is another matter: after a crash it replays input and writes some output again. This Part follows a small job through a crash and a restore, and shows what the sink must do.

The job computes merchants' daily totals from the ledger's change stream. It runs alongside the payment story: it reads each change about 4 s after its commit (our assumption), and its watermark (how far it believes event time has got) is simply the newest event time it has read; the change stream carries a heartbeat every second. How real watermarks work is the job of page 07 (Event Time, Watermarks & Checkpoints).

Checkpoints make state exactly-once

A checkpoint saves, together, every operator's state and the input positions that state reflects. A restore loads both, so the job rewinds its state and its input to the same moment and replays from there. Each input event then affects the state exactly once, however many times it is read.

Output is different. Anything the job wrote to the sink after the checkpoint's barrier reached the sink was written by a run that the restore has now erased, and the replay will write it again. So:

Effectively-once = exactly-once state + a sink that makes repeated output harmless.

Our sink writes absolute totals, SET (merchant, day) = total, so writing the same total twice is harmless. Each sink write is also tagged with the id of the last checkpoint barrier the sink has passed (the id Flink hands the sink when it takes its snapshot; after a restore it starts at the restored checkpoint's id), and a sparse index on that tag lists recently written keys. Not the last completed checkpoint: the sink hears of a completion only later, and sometimes never, so writes made between the barrier and that notice would carry an older tag and escape the flush. The ad-click loop added this tag (step 2.4) so a restore can find keys that restored state knows nothing about.

Trace: the job restores

Synthesizing vector architecture diagram...

What to notice: the replay never touches m2, because v2 skips declined payments. Without the flush's delete, the 500 written by v1 would stay in the sink forever.

Snapshot P5, the stream job after S4:

Synthesizing vector architecture diagram...

After S4: the sink matches restored state plus the replay: m1 briefly showed 12,000 during the flush, and m2 is gone.

Idempotent sinks vs two-phase-commit sinks

There are two ways to make output safe to repeat, and they cost different things:

The ad-click loop picked the absolute sink because a two-phase-commit sink would have turned its 3-second freshness into the checkpoint interval.

Branch S, the same restore without the full flush. From the simulator, after the replay the sink holds m1 = 15,000 (correct) and m2 = 500: v1's wrong total for a declined payment, which the replay never touches and nothing will ever rewrite.

textCopy
FULL FLUSH(restored checkpoint c)          -- after every restore or rescale, before new output
  1. for every key in the restored state: SET key = its value, tag = c
  2. query the tag index for keys whose tag is c or later   -- written after barrier c reached the sink
  3. delete every one of them that the restored state doesn't hold
  4. replay from c's input positions
every sink write carries the id of the last checkpoint BARRIER this sink has passed
(after a restore, starting at c), not the last completed checkpoint, whose notice
arrives late or not at all; the index on that tag is sparse, so step 2 reads only
recently written keys

Dedup state, event time and the acceptance horizon

The change stream can also deliver a change twice (a connector that restarts from an older position). So the job keeps a dedup state: the event IDs it has seen. Two rules make it safe:

1. Keep it in the checkpointed state, not in an outside store. From the simulator: with the seen-IDs in an external cache, the cache still holds ev_7 after the restore, the replay of ev_7 is dropped as a duplicate, and m1 stays at 12,000: the restore under-bills by 3,000.
2. Expire it by event time, with an acceptance horizon. Each ID gets a timer at its event time + 2 hours (7,200 s); the timer fires when the watermark passes it. And a copy whose event time + 7,200 ≤ watermark is rejected as late (sent to a batch recount). Because the timer and the rejection use the same boundary, every copy that is accepted still finds its original in state, as long as the watermark the rejection uses never goes backwards (rule 3).
3. Keep a floor under the watermark. A stream processor like Flink doesn't checkpoint watermarks: after a restore the watermark starts from nothing until the sources send new ones. In that gap a copy that was already too late before the crash would look on time, find its original's ID gone (its timer fired before the checkpoint), and be counted twice. So the dedup operator saves the last watermark it saw in its checkpointed state, and after a restore rejects as late any copy whose event time + 7,200 ≤ the larger of that saved floor and the live watermark.

A processing-time TTL makes dedup depend on how far behind the job is: the longer the outage, the more duplicates. An event-time timer doesn't care. Keep a processing-time TTL only as a much longer guard against state growing without bound.

---

Part 9. Across Regions

Everything so far assumed one Region. With a second Region, every table that remembers a key is replicated, and replication is asynchronous in most designs. That changes what a condition means.

What replication does to a claim

Facts from the DynamoDB documentation for multi-Region eventual consistency (MREC) global tables, the default mode:

• Changes are replicated asynchronously, "typically within a second or less".
• "Conditional writes evaluate the condition expression against the version of the item in the Region": the local replica.
• Conflicting writes are resolved per item by last writer wins.
• Transactions are atomic only in the Region where they run; other Regions may see part of one for a while.

So during the lag, two Regions can both claim the same key, and both succeed. Our story uses a lag of 1.0 s (an assumption).

Trace: two Regions claim the same key (branch F2)

The PSP answers promptly in this branch (no gray failure). A1 claims K1 in R1 at 0.02. At 0.45 the phone's resend A1′ is routed to R2, before R1's claim has replicated (0.02 + 1.0 = 1.02). From the simulator:

Synthesizing vector architecture diagram...

What to notice: one charge, two credits. The derived PSP key protected the one system that dedups in one place, the PSP. Our own tables checked each claim locally, so both Regions ran the whole path.

The first row is the double credit. The other two show that "last writer wins" doesn't rescue you: it can hide a duplicate by accident, and it silently drops any other change to the same item made during the lag.

The fixes:

• A home Region per key. Route every request for a caller (or merchant) to one Region; in the home-Region run, A1′ found IN_PROGRESS and got 409.
• Or a claim store that is strongly consistent across Regions: a DynamoDB multi-Region strong consistency (MRSC) table, where "conditional writes always evaluate the condition expression against the latest version of an item". MRSC runs in exactly three Regions (three replicas, or two plus a witness), supports no transactions and no TTL, and a write that conflicts with one in progress in another Region fails with ReplicatedWriteConflictException.
• Or a single-writer ledger: one Region takes all ledger writes.

Trace: a Region fails in the middle of replication (branch F)

The main line, now with a standby Region R2 that is 1.0 s behind. From the simulator:

Synthesizing vector architecture diagram...

What to notice: R2 never saw R1's last second, yet it produced the same charge ID and the same event ID, because both are derived. That is what lets the PSP and the inbox recognise the replay.

Two counterfactuals, also from the simulator:

• A per-Region attempt key. If R2 replays with pay_7:R2:1, the PSP has never seen that key and charges again: ch_2, $60.00 in total.
• A random event ID in R1. If R1's outbox row had carried a random ID while R2 derives pay_7:SUCCEEDED, R1's leftover copy doesn't match anything in the home inbox, so it is credited again: m1 = 18,000. This is why the event ID is derived in any design with a Region failover (Part 2); a single-Region design may store a random one.

Where a claim can live

Regional dedup windows don't cross Regions. A FIFO queue's dedup IDs, a Valkey fingerprint cache and a Kafka producer session all live in one Region. After a failover, the new Region's copies of them start empty.

For quorums, lag and read-your-writes, see page 06 (Replication, Quorums & Read-Your-Writes); for detection, promotion and routing, page 09 (Multi-Region Failover). Meanwhile, Primitive #24: Disaster recovery and multi-Region covers the broad picture.

---

Part 10. Do you need it? Compared on equal terms

Idempotency keys cost storage, a write on the hot path and discipline in every client. Before paying that, it is fair to ask whether something people already know (a lock, a distributed transaction, a saga, an exactly-once broker) would do the job. Each gets the same test: the gray failure of events 3 to 9, where the PSP charges and its reply is lost.

The payment six ways

From the simulator, the same failure under each approach (the last row is this page's design):

The saga row works for the same reason the last row does: it is named after the client's key, and each step carries a derived key. Take the keys away and it behaves like the two-phase-commit row.

The comparison, on equal terms

What a saga gives up

A saga runs a series of local transactions, each with a compensation that undoes it (a refund undoes a charge). Each step commits on its own, so a saga gives up isolation: other readers can see the intermediate states. A merchant's dashboard can show a payment as charged, then refunded; a stock check can see an item reserved by an order that is about to be cancelled. Designs handle this with semantic counter-measures: a PENDING state that readers treat as not final, updates that give the same result in any order, and re-reading a record before acting on it. Its compensations are ordinary steps too: each has its own derived key (pay_7:cancel:refund), and an unknown one is retried, not abandoned (Part 4).

Why teams choose a saga over two-phase commit across services (the second question of the drill The Flight Booking That Charged Without a Seat): no locks are held across services or across a slow partner API; a crashed coordinator doesn't leave other services blocked, because each step already committed and the orchestrator resumes from its durable log; each service stays available on its own; and each step pays one local commit, not two rounds. The price is the lost isolation above and the compensation logic. For how orchestration works, see Primitive #10: Two-phase commit and saga orchestration.

---

Part 11. End to end: from the tap to the notification

"We made the API idempotent" protects one hop out of nine. A repeat can be born anywhere along the path, and each place needs its own ID and its own memory. This Part lists them all.

Every place a repeat is born

Synthesizing vector architecture diagram...

What to notice: every box except the load balancer, the ledger and the push provider can repeat something. We don't draw a retry at the load balancer: this page relies only on its arrival stamp, not on any retry behaviour.

The hop table

One tap, one sequence

A1 end to end, with the ID each hop carries:

Synthesizing vector architecture diagram...

What to notice: the IDs change name at each hop but never at random: K1 names pay_7, pay_7 names the PSP key, the event and the replace ID. That chain is the design.

---

Part 12. On AWS

The mechanism is the same on AWS. What changes is which service plays the durable guard, which ones offer short windows, and which only look like dedup.

Managed services that use it

Running it yourself

Shared limits

• The PSP's rate limit is shared by live charges and the resolver's replays. Stripe documents default live-mode limits of 100 requests a second per account and 25 a second per endpoint unless noted, and asks high-volume users to request increases in advance. Whatever your limit, give the resolver its own cap, so replays after an outage don't starve new payments.
• The durable guard's unique index is written by every create and shares the ledger's single writer with everything else.
• One merchant's balance item or row takes every credit for that merchant (Part 6).
• The FIFO message group serialises a merchant's events.
• The dedup cache's memory holds every fingerprint in its window; eviction turns into silent duplicates.
• The full flush shares the sink's write capacity with live output.

Look-alikes that are not this mechanism

---

Part 13. What you've learned

Back to the tap

The buyer tapped Pay once, and the network lost the answer at almost every hop. The simulator's final count: one charge (ch_1), one credit to m1 (12,000 → 15,000), one notification on the phone. Here is how each piece did its job:

• The app made K1 at the tap and reused it on every retry (Part 2). The server recorded the intent before the PSP call: a PENDING row, unique on (u1, K1) (Part 3, snapshot P1).
• The PSP's reply was lost. At the deadline the payment became UNKNOWN, not failed, and the key record stored a pointer, so the retry at 10.00 heard "processing" rendered from the payment itself (Part 3, snapshot P2).
• The resolver replayed the derived PSP key and got the saved ch_1; a compare-and-set made the late webhook harmless (Part 5).
• The relay sent ev_7 three times. The queue's window dropped the second; the worker's inbox dropped the third (Part 6, snapshot P3).
• A worker died after pushing. The notification record made the next worker finish the job, and the replace ID made the second push replace the first (Parts 6 and 7, snapshot P4).
• The stream job crashed and restored. Exactly-once state, an absolute sink and the full flush left the totals right (Part 8, snapshot P5).
• The buyer came back after two hours. The key store had forgotten K1; the durable guard answered (Part 4, snapshot P6).
• In the branches: a deadline and a void stopped the late original that opened this page; a home Region stopped the double credit in F2; derived IDs made R2's replay safe in F (Parts 4 and 9).

The whole story, event by event

Payment timeline:

Stream job, parallel timeline:

The cheat card

Failure checklist

• Does every client make its key before the first send, store it, and reuse it on every retry?
• Is the durable record of the intent written before any external call, with a unique key that lives as long as the effect?
• Does the read path treat an expired key record as absent, and does the durable guard answer after the window?
• Is the server's deadline, counted from the load balancer's stamp, shorter than the client's timeout minus the wait before the stamp? Does "did it happen?" void an absent key?
• Is every condition on the record that never moves or expires (or leaves a tombstone)?
• Is every write from an at-least-once consumer absolute, versioned or guarded by an inbox, never an increment?
• Does every re-send (relay, resolver, standby Region, re-run parser) carry the original or derived ID, never a fresh one?
• Does anyone rely on a broker's or producer's dedup as the only guard?
• Does a detected duplicate finish the steps the first attempt may not have finished (re-publish, re-send the notification)?
• Is every notification recorded as an intent with the effect, and sent with a replace ID?
• After a stream-job restore, does a full flush rewrite state and delete keys tagged with the checkpoint's id or later (tagged by barrier, not completion)? Is dedup state checkpointed and expired by event time?
• Does the late check use a watermark floor restored from the checkpoint?
• In more than one Region, does each key have one home Region (or a strongly consistent claim store), and does a failover resume unknowns with the same keys?

Think-first drills

Interview questions

Where to go next

• Write-Ahead Log, fsync & Group Commit: why an error or a timeout leaves a write's outcome unknown.
• Leases, Fencing Tokens & Distributed Locks: stopping a zombie holder with a fencing token, and the clock-and-margin rule a takeover uses.
• Change Streams & the Transactional Outbox: how an event leaves the database, relays vs change data capture, ordering, and apply-by-version consumers. Also Primitive #12: Change data capture and the outbox pattern.
• The Replication, Quorums & Read-Your-Writes (page 06), Event Time, Watermarks & Checkpoints (page 07) and Multi-Region Failover (page 09) loop primitives, all coming. Meanwhile, Primitive #24: Disaster recovery and multi-Region.
• Primitive #05: Message queues vs event streams and Primitive #10: Two-phase commit and saga orchestration.
• Drill: The Flight Booking That Charged Without a Seat, answered in Parts 4 and 10.
• Loops:
  • Payments (steps 1.1, 1.2, 1.4, 2.1, 2.4, 3.5, and round 2's key store), digital wallet (1.4, 2.2, 2.5), hotel reservations (1.5, 1.6, 3.5), transactional outbox and ledger (1.3, 1.5, 3.4, 3.5), stock exchange (3.4).
  • Message queue (1.2, 2.2, 3.2), key-value store (round 2, 3.2), rate limiter (3.2), URL shortener (3.2).
  • Chat (1.3, 1.4, 2.2), leaderboard (1.3, 3.2), notifications (1.3, 2.3, 2.4, 2.6, 3.4).
  • Job scheduler (1.3, 1.4, 2.4, 3.3, 3.4), S3-like storage (3.1, 3.5), web crawler (1.1).
  • Proximity (2.1), nearby friends (2.6), ride-sharing (round 1, 3.6).
  • YouTube (1.5, 2.2, 2.5), Drive (3.5, 3.6), ad-click aggregation (1.3, 2.4, 2.6, 3.3), email (1.3, 1.6, round 2, 3.4).
  • Mobile news feed (2.2), mobile chat (1.2, 2.2), mobile stock trading (1.2).