Distributed Transactions: Sagas vs Two-Phase Commit

Part 0. Start here

The problem: a sold-out hotel that cost $36,260

Wayfare, a travel site, sells trip packages: a flight seat and a few hotel nights, paid by card. Its first version books a trip the obvious way, inside the web request: charge the card, then take the seat, then take the rooms, and if a later step fails, refund the card.

Wayfare runs a one-hour Lisbon sale. 50 trips a minute are booked (our example's number), each $1,240. At minute 40 the hotel sells out. Nobody stops the sale, so for the last 20 minutes every attempt charges the card, takes a seat, fails at the hotel, gives the seat back and refunds.

That PSP, Stripe, states both costs plainly in its refund documentation: "Stripe's processing fees from the original transaction aren't returned", and "Your customer sees the refund as a credit approximately 5-10 business days later". The same page offers the way out: "You can cancel a payment before it's completed at no cost." A card payment can be authorized first (the money is held on the card, not taken) and captured later. An uncaptured authorization of an online, customer-initiated card payment stays valid for 7 days. Cancelling it (a void) is free.

"Up to 10" because a refund that timed out may still have happened. Nobody knows until someone asks the PSP.

The big picture

Synthesizing vector architecture diagram...

What to notice: the arrows give the order the orchestrator runs the steps: S1, S2, S3, SF, S4, S6, S7, S5, S8. Every step in "Before the pivot: undoable" has an undo, and SF is the last of them. "The pivot" has none. The steps in "After the pivot: retriable" have no undo either: they are built to succeed when retried, so the trip can only move forward once the ticket exists.

What you'll be able to do after this page

1. Say why one database transaction can't cover several databases and partner APIs, and name the two families of answers (Part 1).
2. Walk through two-phase commit's prepare, vote, decision and commit, and compute what its lock window costs a hot row (Part 2).
3. Say exactly what a coordinator crash blocks, for how long, and why an operator must never guess (Part 3).
4. Build a saga from local steps and compensations, and undo it in reverse order (Part 4).
5. Order the steps around a pivot, fence the holds before it, and tell the customer at the pivot (Part 5).
6. Use holds that expire (try-confirm-cancel), and handle a cancel that arrives before its hold (Part 6).
7. Treat a timeout as an unknown, resolve it, and handle a compensation that fails (Part 7).
8. Recover a saga after its orchestrator crashes, with a log, keys, a lease and an epoch (Part 8).
9. Name the three anomalies other bookings can see, and the countermeasure for each (Part 9).
10. Compare orchestration with choreography, including ordering on the bus (Part 10).
11. Compare 2PC, sagas, one-store transactions and choreography on equal terms (Part 11).
12. Trace one booking through every layer, with the durable record and the guard at each (Part 12).
13. Map all of it to AWS, and name the look-alikes (Part 13).

You may have arrived from a step that relies on this: step 2.1 of the payment processing loop (a payment is a saga, and "PSPs don't take part in anyone's 2PC"), step 2.2 of the digital wallet loop (a cross-shard transfer as try-confirm-cancel), steps 2.3 and 3.5 of the hotel reservation loop (fence the hold first, charge second; the refund as a durable step), step 3.1 of the job scheduler loop (a durable workflow engine with compensations) or step 3.4 of the Shopify case study (a void as the compensation, driven from an outbox). This page is the "why" behind all five, and behind the drill The Flight Booking That Charged Without a Seat, whose two questions it answers in full (Parts 5, 7 and 11). In all, 22 interview loops depend on this page's mechanisms in at least one step, 13 of them on sagas, 2PC, holds or compensations directly.

---

Part 1. One trip, three databases, two partners

The hook's code charged, then took a seat, then took the rooms. Each call committed on its own, so when the third failed, the first two had already happened. This Part says why no single transaction can fix that, and names the two ways out.

Charge, seat, rooms, refund

Synthesizing vector architecture diagram...

What to notice: the first message already moves money, so every failure after it is repaired with a refund. The two steps most likely to fail (the rooms, then the seat) come after the only step that is expensive to undo.

What one transaction covers

A database transaction is atomic because one database writes one log: the commit record is either on disk or not. A second database has its own log, and an HTTP partner has none that you can see. So BEGIN; charge; take seat; take rooms; COMMIT can include none of them. Writing to each in turn is a dual write: a crash between two writes leaves one done and the other not. The Change Streams & the Transactional Outbox loop primitive shows this failure for a database and a broker (Part 1) and fixes it with an outbox; here the second party is another service or a partner, and an outbox alone can't undo a charge.

Two words get mixed up here:

• Atomicity: all of the changes happen or none do.
• Isolation: nobody else sees the changes half-done.

A trip needs atomicity across systems. Whether it also needs isolation is a separate question, answered in Part 9.

Two families

The example we follow

One story runs through the page: one trip booking: flight, hotel, card. Wayfare sells packages from seat and room allotments it owns. At 10:00:00.000 customer c-88 books trip trip-4417: seat 14C on flight FL-212 to Lisbon ($520) and 3 nights at hotel **`H-31`** ($240 a night, $720), **$1,240** on a card. Three services each own a PostgreSQL database; two partners sit outside:

• seats owns the row flight_inventory(FL-212, available), one row per flight (every booking on FL-212 updates it: the hot row), and a row per seat.
• rooms owns a row per hotel night with its available count, and a row per room hold.
• trips owns the trip record, the saga log and the outbox; it runs the orchestrator.
• The PSP (a partner) authorizes, captures, voids and refunds.
• The airline (a partner) issues tickets against a fare quote, here quote q-77.

The saga has nine steps, run in the order S1, S2, S3, SF, S4, S6, S7, S5, S8. The numbers are names, not the order; Parts 5 and 6 explain the order. Every step carries a key trip-4417:<step>; every undo carries the step's key plus :undo.

The orchestrator writes a record to the saga log (5 ms) after every reply and before the next call. The network adds nothing in our example, so every time on this page is exact arithmetic from these numbers. Timeouts: 2 s for seats and rooms, 5 s for the PSP, 10 s for the airline. The full settings table is in Part 14.

The story has twelve beats: one trip, three databases, two partners (Part 1); all or nothing, with locks (2); the coordinator that died (3); undo instead of rollback (4); the point of no return (5); holds that expire (6); the reply that never came (7); the orchestrator that crashed (8); what other bookings see (9); nobody in charge (10); the same trip four ways (11); and one trip, end to end (12). The main line (the booking that succeeds) runs through Parts 4, 5, 6 and 12. Every failure, and every two-phase commit, runs on a copy rewound to 10:00:00.000, so one never swamps another. Every trace comes from running a private reference implementation of this setup, not from working by hand. The full event table is in Part 14.

At 10:00:00.000 the request arrives with the client's key k-4417 (the Idempotency & Effectively-Once Processing loop primitive), trips creates trip-4417, and the orchestrator writes STARTED to the saga log, done at .005 (times are seconds after 10:00:00.000).

A few words used on this page:

Background reading, not relied on for any fact here: Primitive #10: Two-phase commit and saga orchestration.

---

Part 2. All or nothing, with locks

On a copy: suppose we want seats, rooms and trips to commit trip-4417 together, as one atomic unit, whatever happens. The protocol for that is two-phase commit (2PC). This Part runs it on our three databases, shows what it can't include, and measures what it costs when nothing fails.

Three databases, one decision

The trips service acts as the coordinator. Replay P0, from the reference implementation (1 ms per round trip; each PREPARE TRANSACTION, the decision record and each COMMIT PREPARED is one forced write of 5 ms):

Synthesizing vector architecture diagram...

What to notice: three forced writes sit on the path (the participants' prepare, the coordinator's decision, the participants' commit), and the FL-212 row stays locked from the first write at .005 until the commit is acknowledged at .062. That lock window, not the forced writes, is what other bookings pay for.

Prepare, vote, decide, commit

textCopy
COORDINATOR (the trips service), global transaction g-4417
  1. record g-4417 and its participants          (so recovery knows whom to ask)
  2. run the writes on each participant, each in its own open transaction
  3. send PREPARE to every participant; wait for every vote
  4. all YES  -> force the decision record COMMIT to disk
     any NO, or a vote missing at the timeout -> ABORT
                (nothing to force: no record means abort, "presumed abort", Part 3)
  5. send the decision to every participant; resend until each acknowledges
  6. once every participant acknowledged, forget the decision record

PARTICIPANT (seats), on PREPARE g-4417
  1. can I still commit? (constraints checked, rows still locked)
  2. yes -> force the changes and a "prepared" record to disk, vote YES
            from now on: never decide alone, keep the locks, wait for the decision
     no  -> roll back, vote NO

Each forced write has a job. The participant's prepared record is its promise: after a crash it must still be able to commit. The coordinator's decision record is the only place the outcome exists between the votes and the last acknowledgement: if it crashes after deciding COMMIT and before telling everyone, the record is what it reads on restart.

What it looks like in PostgreSQL and MySQL

The standard for this is XA: a transaction manager drives the protocol, and each database acts as a resource manager.

After PREPARE TRANSACTION, PostgreSQL says, "the transaction is no longer associated with the current session; instead, its state is fully stored on disk", and "the transaction continues to hold whatever locks it held". Part 3 needs both sentences.

What 2PC can't include

The trip also needs the card and the ticket. Put them inside the lock window (replay P0x), so that "all or nothing" covers them:

Two things are wrong. The PSP and the airline never prepared: they did their work at once, so if a prepare fails at 1.651 the ticket is already issued and the authorization already holds money. Undoing them needs compensations: a saga is needed anyway. And the hot row is now locked for 1.657 s instead of 57 ms. 2PC can cover only our own databases; the partners force a saga regardless.

Isolation is still the participants'

2PC gives atomicity across participants: all commit or all abort. It adds no isolation. Each database isolates its own rows at its own level, and a reader can see seats committed and rooms not yet, between two COMMIT PREPAREDs. MySQL's own XA page says it: "REPEATABLE READ may not be sufficient for distributed transactions". "2PC gives serializability" is a common wrong answer.

The lock window

A row that every booking updates can commit one change at a time. Formula 1:

$$\text{a row's commits per second} \;\le\; \frac{1}{\text{how long each change holds its lock}}$$

Replay H: bookings arrive at FL-212 at random, 2 a second during the sale (our number), for 60 seconds, and a waiting booking gives up after lock_timeout = 5 s.

Under P0x the queue doesn't grow for ever: it levels off. The row serves 0.60 a second, so about 2 − 0.60 = 1.4 bookings a second wait out the 5 s and fail, and about 2 × 5 = 10 are waiting at any moment, each holding a database connection. P0 and the saga both have plenty of headroom at 2 a second; the gap shows when a flash sale pushes the rate past 17.5.

One store, one call

On another copy: if all three services' tables lived in one DynamoDB account and Region, the store's own transaction could replace the protocol (replay D). One TransactWriteItems writes 6 items (the flight counter with the condition available >= :one, seat 14C, the 3 room nights with conditions, and the trip), each under 1 KB, with the client token trip-4417:

The facts it relies on, from AWS's documentation: TransactWriteItems "groups up to 100 write actions in a single all-or-nothing operation … within the same AWS account and in the same Region", the items total at most 4 MB, and no two actions may target the same item. "DynamoDB performs two underlying reads or writes of every item in the transaction: one to prepare the transaction and one to commit", so transactional writes cost 2 write units per KB, consumed even when a condition cancels the transaction. A client token "is valid for 10 minutes after the request that uses it finishes". Each item's partition keeps its own limit too (1,000 write units a second per partition, the figure the loops use), so a hot flight counter is still one hot item. Not BatchWriteItem: its up to 25 puts and deletes are each atomic, but "BatchWriteItem as a whole is not", and failed ones come back in UnprocessedItems. The last row shows why the conditions matter more than the token: after 10 minutes the token no longer protects the retry, and the seat's condition does.

The saga shrinks to four steps: the D transaction (compensatable: its undo is a second transaction that gives everything back), authorize, the ticket (the pivot), and capture. Its "booked" would come at 1.640 and its end at 1.945. It still needs a saga, because the partners are still outside. Several loops use exactly this one-store answer: step 2.3 of the ride-sharing dispatch loop (three items instead of "an ordered two-step with compensation"), and the effect-plus-idempotency-record writes in the chat, email, Drive, nearby-friends and bot-defense loops.

---

Part 3. The coordinator that died

On a copy of P0: the coordinator crashes at .051, right after all three participants voted yes and before it wrote a decision. Another instance of the trips service takes over when the dead one's lease runs out, 10 s later, the same takeover the saga's orchestrator gets in Part 8. Even that short gap costs about ten bookings on FL-212. With nobody to take over, it costs about 170.

Ten seconds, or ninety, of a frozen flight

Synthesizing vector architecture diagram...

What to notice: between the crash and the scan, the three participants hold their locks and do nothing, because each has promised to wait. The new coordinator has to go and look: nothing in PostgreSQL asks it.

From the reference implementation:

Why a prepared participant waits

Blocking needs one specific moment: every participant has voted yes, and none has heard the decision. Before that moment, a participant can decide alone:

• it hasn't voted yet: it may abort (the coordinator will then decide ABORT);
• it learns that another participant voted no, or never prepared: the decision must be ABORT, so it may abort.

After that moment it may not. The coordinator might have decided COMMIT and told another participant already; aborting alone would split the trip. So it waits, holding its locks. Participants asking each other doesn't help either: all of them voted yes, so none of them knows more.

Presumed abort and the scan

The R* system's paper (Mohan, Lindsay and Obermarck, 1986) named the rule most systems use, presumed abort: it is "safe for a coordinator to 'forget' a transaction immediately after it makes the decision to abort it". No record means abort, so the coordinator never forces an ABORT record, and a coordinator that finds nothing on restart aborts.

The R* participants periodically ask the coordinator what happened. PostgreSQL participants never ask. They sit in pg_prepared_xacts until someone runs COMMIT PREPARED or ROLLBACK PREPARED. So the recovering coordinator must know every participant it used (step 1 of the protocol), scan each one's pg_prepared_xacts, and for every gid it finds: resend COMMIT PREPARED if its log says COMMIT, otherwise ROLLBACK PREPARED.

What exactly is blocked

"The whole cluster freezes" is a common wrong answer. What is locked is the rows the prepared transactions wrote: FL-212, seat 14C, three nights at H-31 and one trip row. Bookings on other flights and other hotels share no lock with them and are unaffected.

And prepared transactions hold no connections: PostgreSQL detached them from their sessions at PREPARE. What fills a connection pool is the new transactions that wait for those rows, each holding its own connection. Formula 2 (Little's law):

$$\text{waiting transactions} \;=\; \text{arrival rate to the locked rows} \;\times\; \text{how long each waits}$$

Each booking waits at most lock_timeout, then fails. The bookings that arrive in the last 5 s before the rollback are still waiting when the rows unlock, and succeed. So the failures are about the arrival rate × (the blocked time − lock_timeout).

Synthesizing vector architecture diagram...

Snapshot T4, P1 at 30 s. What to notice: the three transactions in "Prepared" hold the locks and no connection. The bookings in "Waiting" hold the connections. Nothing in "Unaffected" waits for anything.

Replay P1h, the 90 s of P1 under the sale's load, from the reference implementation (seeds 1 to 5):

With the 10 s takeover of P1s, the same arithmetic gives 2 × (10.058 − .005 − 5) ≈ 10 failed bookings (seeds 1 to 5: 5 to 15, mean 9.4), and 8 to 11 more that waited up to 5 s and then succeeded.

The operator's temptation

On a copy of P2 (the coordinator decided COMMIT, then died): at 60 s, with the flight frozen, an operator runs ROLLBACK PREPARED on seats to unblock FL-212 (replay P3):

On a copy of P1 (no decision) the same action happens to be right: the restart rolls back the other two, and the trip is cleanly aborted. The operator can't tell P1 from P2 without the coordinator's log. The R* paper describes exactly this operator interface, to "forcibly commit or abort" a prepared process, warns that its "misuse … could lead to inconsistencies", and suggests the operator "could use the telephone to find out the coordinator site's decision". The rule: read the coordinator's decision log before touching an in-doubt transaction. COMMIT recorded: COMMIT PREPARED. Nothing recorded, or ABORT: ROLLBACK PREPARED. Never guess. XA calls an outcome forced this way a heuristic decision.

When a participant crashes

If seats itself crashes after preparing, its prepared transaction survives: its state "is fully stored on disk", and PostgreSQL restores it, with its locks, when it restarts. FL-212 stays locked through the participant's recovery until the decision arrives. What a managed failover does with prepared transactions depends on the service; this page doesn't state it for Aurora, because AWS doesn't document it.

Coordinators that don't die: replication

Distributed databases such as Google Spanner and CockroachDB run 2PC inside, between their own shards, without this blocking. They don't use a better protocol; they replicate the coordinator. It isn't the clocks: Spanner's TrueTime gives transactions their timestamps (external consistency); what removes the blocking is the replication. In Spanner, "If a transaction involves more than one Paxos group, those groups' leaders coordinate to perform two-phase commit", and "The state of each transaction manager is stored in the underlying Paxos group". A dead leader is replaced by another replica with the same state, so "Running two-phase commit over Paxos mitigates the availability problems". CockroachDB replicates each transaction's record with Raft; other transactions check the coordinator's heartbeat, and if it stops, the record is moved to ABORTED. The Replication, Quorums & Read-Your-Writes loop primitive explains the elections (Part 6); background in Primitive #09.

Replacement isn't instant. Spanner's leader leases were "10 seconds by default" in its 2012 paper, which reports that after killing a leader, "Approximately 10 seconds after the kill time, all of the groups have leaders". Replay P4 puts our coordinator's state in such a group: its leader dies at .051, a new leader takes over at 10.051, finds no decision in the replicated log (presumed abort), and rolls back by 10.057: about 10 failed bookings on FL-212 (seeds 1 to 5: 5 to 15), the same as P1s, with no takeover machinery of your own and no human. And all of it is still one database: replication doesn't let a PSP or an airline prepare.

---

Part 4. Undo instead of rollback

Back to the main line. trip-4417 is booked as a saga: each step commits in its own database at once, and nothing holds a lock across services. But on a copy where night 2 at H-31 has no rooms left (replay F1), the seat is already held when the rooms fail. There is no rollback across databases. Something must undo the seat.

Seat held, rooms gone

Synthesizing vector architecture diagram...

What to notice: the orchestrator writes ABORTING before the first undo, so a crash in the middle of undoing continues backwards, never forwards. The card was never touched, and the whole failure costs $0.

A saga

The name comes from Hector Garcia-Molina and Kenneth Salem's 1987 paper Sagas, written for long-lived transactions inside one database: "A LLT is a saga if it can be written as a sequence of transactions that can be interleaved with other transactions." Each step Tᵢ has a compensating transaction Cᵢ that "undoes, from a semantic point of view, any of the actions performed by Tᵢ". Services and partners fit the same idea: each step is a local transaction in its own system, and a failure is repaired by running the compensations of the steps that happened, newest first.

The main line so far, from the reference implementation:

Snapshot T1, at .480 (after the fence of Part 5):

Semantic undo

A compensation is a new local transaction that restores the business meaning, not a physical rollback of the old bytes:

• Undoing a seat hold writes available = available + 1 and marks the hold RELEASED. It doesn't delete the hold row (Part 6 needs it) and doesn't write back the number it read (Part 9 shows why).
• Undoing an authorization is a void, a call to the PSP.
• Undoing a transfer between accounts writes a balanced reversal, a pair of entries. The digital wallet loop's history has the counter-example: a compensation that inserted a one-sided credit line created money, and was replaced by a try-confirm-cancel flow with an in-transit account and balanced pairs (step 2.2 of the wallet loop).

A compensation is never one-sided and never a delete of the step's record. It is itself a complete local transaction, with its own key.

The idea reaches past servers. A phone app that shows a like at once and removes it when the server rejects the action runs a client-side compensation (step 2.5 of the mobile news feed loop).

Backwards, in order

On a copy where the card is declined (replay F2), two steps must be undone:

textCopy
ON a step Sk rejected for a business reason (sold out, declined, fare changed):
  write "ABORTING (reason)" to the saga log
  FOR each step with a DONE record, newest first:
      IF it has a compensation Ck:
          run Ck with key trip-4417:<step>:undo      (safe to repeat)
          write "Ck DONE"  (the last one: "Ck DONE + ABORTED", one record)
  tell the customer

Why newest first: a later step may depend on an earlier one (a capture needs its authorization), so undoing in reverse never leaves a step whose basis is gone. Only an explicit rejection starts this loop. A timeout doesn't (Part 7).

---

Part 5. The point of no return

The hook charged first and refunded later: the expensive undo was the first step. A saga lets us choose the order. This Part puts the cheap, likely-to-fail steps first, puts the one step that can't be undone at a chosen point, and answers the customer there.

Three zones

Azure's saga pattern guide names the three kinds of step:

• Compensatable steps "can be undone or compensated for by other transactions with the opposite effect": our holds and the authorization.
• The pivot: "Pivot transactions serve as the point of no return in the saga. After a pivot transaction succeeds, compensable transactions are no longer relevant." Ours is issuing the ticket.
• Retriable steps "follow the pivot transaction. Retryable transactions are idempotent and help ensure that the saga can reach its final state". Ours: confirm the rooms, confirm the seat, capture, send the email.

Synthesizing vector architecture diagram...

Snapshot T2, at 1.685. What to notice: everything in "Compensatable" is done and undoable, the one step in "Pivot" is done and isn't, and "Retriable" holds the four steps still to run, in the order shown. The customer node hangs off the pivot, not off the last step.

Ordering rules

1. The steps most likely to fail go first, if they're cheap to undo. The seat and the rooms are the ones that sell out (the hotel did, in the hook), and each hold is released in 15 to 20 ms, so both holds come before the card.

2. Cheap undos before expensive ones. A hold costs 15 to 20 ms to release. A void costs a call. A refund keeps the fee and takes 5 to 10 business days.

3. Authorize; don't charge. The authorization reserves the money; the capture takes it after the pivot. A void is free:

   Undo of a card step	Cost	When the customer sees it	Possible when
   Void (cancel the authorization)	"at no cost"	The hold disappears from the statement when the bank releases it (bank-dependent)	Before capture, within the authorization's life (7 days here)
   Refund	The fee is kept: $36.26 on$1,240	"approximately 5-10 business days later"	After capture

4. The irreversible step last among the risky ones: that is the pivot.

5. After the pivot, only steps that can't fail for a business reason.

Fence, then the pivot

Rule 5 has a trap. The holds expire in 15 minutes by their own database's clock (Part 6). If the saga stalls between the ticket and the confirmations for longer than that, a sweeper releases the holds, and "confirm the rooms" fails for a business reason after the ticket is issued. So just before the pivot, step SF fences both holds: one conditional update each, HELD → CONFIRMING only if still HELD and unexpired. A sweeper never touches CONFIRMING. SF is the last compensatable step: if a hold already expired, SF fails before anything irreversible, and the saga undoes itself. The hotel loop learned this the hard way: its confirm flow once charged the card before fencing the hold, and now moves HOLD → CONFIRMING first and charges second (step 2.3 of the hotel reservation loop). Part 6 shows the fence failing and saving the trip.

The main line, from the reference implementation:

Telling the customer at the pivot

Once the ticket exists, the trip can't fail: every step left is built to succeed when retried. So the customer is told "booked" at 1.685, not at 2.030, and the rest runs without anyone waiting. Snapshot T3, at 2.030:

Confirm, then capture

The confirmations run before the capture: secure the inventory, then take the money. If the order were reversed and a confirmation could still fail, the customer would have paid for a room Wayfare doesn't hold. With the fence, neither can fail; the order is the second line of defence.

Retriable, not "retried forever"

"Retriable" means designed to succeed when retried: a capture of a valid authorization, a confirmation of a fenced hold, an email from the outbox. It doesn't mean "retry anything forever". On a copy (replay F7), the PSP answers 503 to every capture from 2.025 to 122.025 (two minutes, our number). The capture is retried with backoff and full jitter (base 1 s, cap 30 s: the Retries, Timeouts, Backpressure & Load Shedding loop primitive, Part 4), always with the same key:

The ticket is never undone, the seat and rooms are already confirmed, and the customer already saw "booked". Our model's 503 is refused before the PSP starts executing, so nothing is saved under the key and the retry runs fresh. That matters: Stripe saves "the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails … including 500 errors", but only "after the execution of an endpoint begins". A capture whose key saved a 500 needs a status lookup, not the same key again.

A retriable step still has limits. If captures kept failing for 7 days, the authorization would expire; the saga then needs a human queue and the nightly reconciliation against the PSP's report (step 2.1 of the payment loop).

When the pivot fails

The pivot is the go/no-go step: it can still say no. On a copy (replay F3), the airline rejects quote q-77 at 1.680 because the fare changed:

The drill's first answer follows from this Part and Part 7: The Flight Booking That Charged Without a Seat charged before the flight step; with holds and an authorization first, a flight failure costs a void, and the money never leaves the card.

---

Part 6. Holds that expire

On a copy (replay F5): the hold request for the rooms gets stuck in a retrying proxy at .025. At 1.000, with S2 still in flight, the customer cancels. The orchestrator sends cancel_rooms, and rooms answers "no such hold, OK". At 3.025 the stuck request finally arrives. What stops it from holding three nights for nobody?

Try, confirm, cancel

A saga whose forward steps are reservations that expire has a name: try-confirm-cancel (TCC). The main line is one:

The digital wallet's cross-shard transfer has the same shape: the money moves into an in_transit account (try), then to the receiver (confirm) or back (cancel), and a timeout is never a cancel (step 2.2 of the wallet loop).

Why holds expire

A hold is a lease on inventory: its owner (the saga) must confirm it before it runs out. The Leases, Fencing Tokens & Distributed Locks loop primitive's rules apply (Parts 3 and 6):

• It expires by the participant database's own clock (expires_at checked by now() in the same database), never by an app server's clock, whose time can differ.
• It expires on its own. The expiry is what bounds a lost cancel, a dead orchestrator, and a hold that arrives after its cancel.

The cost is inventory off sale. Rooms off sale on average = holds a second × how long each hold stays open (Little's law again). On our main line a hold is confirmed about 1.7 s after it is taken, so at the hotel loop's 20 holds a second only about 20 × 1.7 = 34 are open at once. The 15-minute expiry is the worst case, reached only by holds nobody confirms or cancels: if every hold ran to expiry, 20 × 900 = 18,000 would be open. Shorter expiries put abandoned stock back sooner and give slow customers less time.

Synthesizing vector architecture diagram...

What to notice: CONFIRMING has no arrow to EXPIRED: the sweeper can't touch a fenced hold, and only a cancel releases it. And a cancel for a hold that doesn't exist yet leaves a TOMBSTONE, which is where a late try ends.

Empty rollback and hanging

Synthesizing vector architecture diagram...

What to notice: the cancel reaches rooms two seconds before the hold it cancels. Because it leaves a tombstone, the hold that arrives at 3.025 is refused instead of taking three nights.

Seata, an open-source transaction framework, names the two problems: an empty rollback is a cancel for a try that never arrived, and hanging (Seata's English pages also say "dangling") is a try that arrives after its cancel. From the reference implementation:

In F5 the orchestrator logged ABORTING at 1.005, sent C2 for the in-flight step at once (it works whether or not the step arrived), then C1 for the done step, and ABORTED at 1.050. Snapshot T5, at 3.045: the trip is ABORTED, the seat RELEASED (available back to 5), no room hold exists, and the tombstone for trip-4417 has just refused the late hold.

textCopy
hold(hold_id):                          try (S2)
  row = the hold row for hold_id
  if row exists: answer from the row, never hold again
                 (TOMBSTONE, RELEASED, EXPIRED -> "refused"; REJECTED -> "sold out"; otherwise "HELD")
  if any of the nights has available = 0: insert (hold_id, REJECTED); return "sold out"
  each night: available = available - 1
  insert (hold_id, HELD, expires_at = now() + 15 minutes)

cancel(hold_id):                        C2, key trip-4417:hold_rooms:undo
  row = the hold row for hold_id
  none               -> insert (hold_id, TOMBSTONE)      empty rollback
  HELD, CONFIRMING   -> each night: available = available + 1; status = RELEASED
  RELEASED, EXPIRED  -> nothing: already given back
  BOOKED             -> refuse: the pivot has passed

The tombstone must live at least as long as a delayed try can: longer than the retry window of every proxy and client that might resend it. The expiry bounds the damage if the tombstone is missing. The wallet loop's "reject if not yet processed" is the same rule.

The fence, and twenty minutes without an orchestrator

The fence and the sweeper, as SQL in rooms:

sqlCopy
-- SF: fence the hold, only if it is still held and unexpired (or already fenced)
UPDATE room_hold SET status = 'CONFIRMING'
 WHERE hold_id = 'trip-4417'
   AND (status = 'CONFIRMING' OR (status = 'HELD' AND expires_at > now()));
-- 1 row: fenced (or already fenced by an earlier run of SF). 0 rows: expired or cancelled, so SF fails before the pivot.

-- the sweeper, every 10 s: only HELD holds, never CONFIRMING
UPDATE room_hold SET status = 'EXPIRED'
 WHERE status = 'HELD' AND expires_at <= now()
RETURNING hold_id;   -- and add 1 back to each night of each returned hold, in the same transaction

The status = 'CONFIRMING' branch makes SF safe to re-run. On a copy, SF commits at .475, its reply is lost, and the orchestrator dies at 1.000 before logging it. Epoch 2 takes over at 11.000, finds S3 DONE as the last record, and re-runs SF: it finds both holds already CONFIRMING, counts that as done, and the trip completes at 12.575. With the stricter status = 'HELD' condition, the same re-run finds 0 rows and voids a good trip (ABORTED at 11.375).

On copies (replay F8), the whole orchestrator fleet is down for 20 minutes from just before the pivot:

The fence's cost: a CONFIRMING hold has no expiry of its own. If the orchestrator never comes back, the stock stays off sale. So alarm on CONFIRMING holds older than the longest normal saga (a few seconds here; minutes if a partner is slow): that alarm is the backstop the expiry used to be. Shopify's flash-sale design has the same shape: a PAYING state that no timer touches (step 2.4 of the Shopify case study).

---

Part 7. The reply that never came

On a copy (replay F4): at .450 the PSP authorizes $1,240, and the reply is lost on the way back. At 5.050 the orchestrator's 5 s timeout fires. Did the card get authorized? The orchestrator can't know, and the drill's failure lives right here.

Timed out at 5 s

Synthesizing vector architecture diagram...

What to notice: nothing is undone at 5.050. The orchestrator writes UNKNOWN, asks the PSP again with the same key, and gets the saved answer: the card was authorized, so the saga continues forwards.

From the reference implementation:

Unknown, not failed

A timeout says only that the answer didn't arrive in time. The request may never have arrived, may have been refused, or may have succeeded with its reply lost. The Idempotency loop primitive owns this rule, the key record behind it, and "When a compensation times out" (Parts 2 to 4); step 1.2 of the payment loop and step 1.2 of the mobile stock trading loop apply it. For a saga it means one thing: only an explicit rejection starts backward recovery. A timeout is logged as UNKNOWN and resolved first.

What guessing costs (replay F4x, the naive rule "a timeout is a decline"):

Resolve, then decide

Once resolved: if the step happened and the saga continues, write DONE and go on. If the saga is aborting, the unknown step is compensated too, once you know it happened; AWS's own saga sample does the same when a step fails, reverting that step as well as the ones before it. The one exception: when the saga is already aborting, the orchestrator may send an in-flight step's compensation without resolving first, because the compensation works either way (it releases the hold, or leaves a tombstone for it: Part 6).

When the undo fails

On a copy of F3 (replay F4r), the void's reply is lost:

This is the drill's first question: a compensating refund (or void) that fails halfway. What keeps the money safe:

1. The intent is recorded first. ABORTING is in the log before C3 is sent, so a crash continues backwards.
2. The undo has its own key, so every retry is the same void, never a second one.
3. Retry with backoff and jitter while the PSP is failing, the same key each time. After the retry budget, the saga stays ABORTING, raises an alarm and goes to a human queue: never "failed, stop". While the void is stuck, the fenced holds stay off sale too, and the CONFIRMING age alarm of Part 6 fires as well.
4. The nightly reconciliation against the PSP's report has the last word: every authorization the PSP holds must match a trip, or be voided. The hotel loop's refund is the same durable step (step 3.5 of the hotel loop), and the scheduler loop marks such a saga COMPENSATION_STUCK (step 3.1 of the scheduler loop).

Two limits are shared. FL-212's row is taken by every hold, release and confirmation on the flight, so a burst of compensations after an outage competes with new bookings for the same lock. And the PSP's rate limit is shared: Stripe's live-mode limit is 100 requests a second per account, and each API endpoint gets 25 a second unless noted. So 1,000 voids after an outage use the cancel endpoint's own 25 a second for 1,000 ÷ 25 = 40 s, and share the account's 100 a second with every authorization and capture. The crowdfunding loop's mass refund draws on the same budget (step 3.4 of the crowdfunding loop).

---

Part 8. The orchestrator that crashed

On a copy (replay F6): the orchestrator (epoch 1) sends "issue ticket" at .480 and dies at .800, right after renewing its lease. The airline still issues the ticket at 1.680, and its reply goes nowhere. The last record in the saga log is SF DONE. What does the replacement do?

The saga log

The saga log is the saga. The orchestrator writes one record (5 ms) after every reply and before the next call:

Writing after the reply means a crash loses at most "which step was in flight". Writing before the next call means the log always shows every step that could have started.

Taking over

sqlCopy
-- every saga-log write is conditional on the writer's epoch (a fencing token)
BEGIN;
SELECT owner_epoch FROM saga_owner WHERE trip_id = 'trip-4417' FOR SHARE;
-- not 1: ROLLBACK and stop; another instance owns this saga now
INSERT INTO saga_log (trip_id, epoch, record) VALUES ('trip-4417', 1, 'S4 DONE');
COMMIT;
-- the takeover's UPDATE of owner_epoch waits for this lock, so it reads the log after this write, never before

textCopy
OWN each saga with a lease of 10 s, renewed on each log write and every 3 s

ON taking over trip-4417 (the lease ran out):
  owner_epoch = owner_epoch + 1             (conditional write; we are epoch 2)
  read the saga log
  IF it contains ABORTING: continue backwards (compensations without a DONE record)
  ELSE: re-run the first step without a DONE record, with its SAME key, and continue

From the reference implementation:

Synthesizing vector architecture diagram...

What to notice: the new owner doesn't know whether the ticket was issued, and doesn't need to: the same key makes the airline return the ticket it already issued. The last two messages show the fence: a paused old owner that wakes up can't write to the log any more.

Snapshot T6, at 10.800: the log's last record is SF DONE; the owner epoch is now 2; 14C and the three nights are CONFIRMING (fenced, so no sweeper touches them during the outage); the authorization is uncaptured; the airline holds TKT-1, which the log doesn't know about yet; the customer still sees "processing".

In the paused variant, epoch 1 wasn't dead, only stalled, and wakes at 20.000 with the airline's reply waiting. Its write S4 DONE at 20.005 is refused: the owner epoch is 2. A lease alone would not stop it; the epoch on every log write does (the Leases, Fencing Tokens & Distributed Locks loop primitive, Parts 4 and 5).

Why the key matters

On a copy where the airline call carries no key (replay F6x), the re-run at 10.800 issues TKT-2 at 12.000. Wayfare now holds two tickets for one seat and pays $520 more, unless the airline's rules let it void one. Every step needs a key for exactly this moment. Where a partner has no key, look the result up by your own reference before re-sending (the payment loop's lookup).

Determinism and versions

Recovery works only if the orchestrator's decisions depend on nothing but the log: "if S3's logged result said declined, abort" is safe to recompute; "if a random number is below 0.5" or "if the clock says after 10:00" is not, unless the value itself was logged. And a running saga must keep the step list it started with: a deploy that inserts a step between S3 and S4 must not change what a half-finished saga does next. Step Functions does this for you: after UpdateStateMachine, "Running executions will continue to use the previous definition". Temporal replays each workflow's history through the code, so code changes that alter the order of decisions need its versioning APIs.

Commands through the outbox

The orchestrator's trips database holds the saga log and an outbox. Writing "S4 DONE" and the next command in one local transaction, then relaying the command, means a crash can't lose the command or send it without its record: the Change Streams loop primitive, Parts 2 and 3. The relay sends at least once, so every participant must accept a repeated command, which the keys already guarantee.

Durable workflow engines are this Part, packaged: Step Functions Standard keeps each execution's history as its saga log, Temporal keeps an event history per workflow and retries activities, and the job scheduler loop builds its own (step 3.1). Part 13 has their limits. After a Region failover, sagas restart from the replicated log in the new Region (step 3.5 of the payment loop; the Multi-Region Failover loop primitive).

---

Part 9. What other bookings see

On a copy (replay I1), FL-212 has 5 seats. trip-4417 holds one at .020 (5 → 4). trip-4418 starts at .300 and holds another at .320 (4 → 3). Then trip-4417 takes F3's path: the fare changed, and C1 releases its seat at 2.030. If C1 writes back the number it read, available becomes 5 while one seat is still held by trip-4418.

The undo that oversold

Synthesizing vector architecture diagram...

What to notice: both panels run the same three writes; only the last one differs. "Absolute undo" writes back the value trip-4417 read before trip-4418 held its seat, which erases that hold. "Delta undo" adds 1 to whatever is there.

The 1987 paper made this exact point about seats: a compensation "cannot simply store in the database the number of seats that existed when Tᵢ ran".

Three anomalies

A saga commits each step at once, so every intermediate state is visible to others. Azure's guide lists what goes wrong:

• Lost updates: one saga overwrites another's change (I1).
• Dirty reads: a saga reads data another saga changed but hasn't finished, and may still undo (I2).
• Fuzzy (non-repeatable) reads: two steps of one saga read the same data and see different values, because another change came in between (I3).

I2, on a copy with 1 seat left: trip-4417 holds it at .020 (available 0). trip-4419 arrives at .480, its S1 finds 0 at .500, and it is told "sold out" at .505. Then trip-4417 aborts (F3), and the seat comes back at 2.030: 1.530 s after trip-4419's hold was refused (.500). The seat was never sold; trip-4419 read a hold as if it were a sale.

I3, on a copy: the fare for FL-212 changes from $520 to **$560** at 1.000, while S4 (sent at .480) is in flight. S4 carries quote q-77, and the airline checks it when it issues, at 1.680, so it rereads the fare, finds it changed, and rejects: exactly F3. The trip is voided and released by 2.035 at a cost of $0. Without the quote ID, the airline issues at$560: the trip now costs $560 +$720 = **$1,280**, against a$1,240 authorization. By default a capture takes at most the authorized amount (some online card payments can be overcaptured, within card-brand limits, on some pricing plans or on request), so either Wayfare absorbs $40, or it goes back to the customer after telling them "booked".

Countermeasures

Two more, one sentence each. A version file logs every operation on a record so they can be applied in the right order even if they arrive out of order. Risk-based concurrency by value picks the mechanism per operation: a saga for a $1,240 trip, and 2PC or a single store's transaction where the money at risk justifies the locks. For the isolation levels inside one database, see Primitive #21. The wallet loop's rule "never check the total across shards mid-transfer" is the same lesson (step 2.3 of the wallet loop).

---

Part 10. Nobody in charge

On a copy (replay C), the same trip runs with no orchestrator: each service reacts to events on a bus and publishes its own. Night 2 is sold out, as in F1. The saga aborts in 205 ms instead of 70, and "where is trip-4417?" has no single answer.

Two styles

Synthesizing vector architecture diagram...

What to notice: in "Orchestration" one component sends every command and knows the order. In "Choreography" the order exists only as the chain of who reacts to which event; no box knows the whole trip.

From the reference implementation (50 ms per bus hop, our number):

Same trip, same failure as F1: .205 against .070. The difference is three bus hops (150 ms), less the orchestrator's three log writes (15 ms). The larger difference is where the logic lives: seats must know that RoomsUnavailable means "release", and the answer to "where is trip-4417?" is spread over trips, seats and rooms.

Ordering on the bus

On a copy (replay Cx), the bus doesn't order events. SeatHeld (published at .070) is delayed on its way to rooms until 1.300 (our number). The customer cancels at 1.000, and TripCancelled reaches rooms at 1.055, before the event it cancels:

Amazon EventBridge now offers both: the Custom Event Bus has FIFO subscribers that deliver "in the order that they were published within an event group" (set by EventGroupId), and the Custom Event Bus - Classic doesn't order. Part 13 has the details. A cycle is the other trap: AWS's choreography guide warns of "cyclic dependencies" when services react to each other's events.

The loops chose: "for money we want one place that knows the state" (step R2.7 of the payment loop). Uber's multi-entity changes are sagas too (step 2.7 of the Uber case study). Choreography fits short, loosely coupled reactions that need no multi-step undo.

---

Part 11. The same trip four ways

So which one? Every row below runs the same trip, the same participants, the same partners and the same failures. 2PC covers only our three databases wherever it appears, because they are the only participants that can prepare; the PSP and airline steps are identical in every row. A crashed coordinator and a crashed orchestrator are both recovered by the same 10 s lease takeover.

On equal terms

Read each row from both sides. 2PC does win something: the rows it covers are isolated until commit, and it needs no undo code for them. The saga loses that isolation (Part 9) and pays with undo code. The one-store transaction is the cheapest atomicity of all, but only if every item lives in one account and Region, and it still needs a saga for the partners. P4's replicated coordinator (Part 3) blocks for about the same 10 s as P1s, without machinery of your own, but inside one database.

Choose this when

• One store's own transaction (TransactWriteItems, one PostgreSQL database's BEGIN … COMMIT): every item lives in that store. Always the first choice when it applies.
• A distributed SQL database (its internal 2PC over replicated groups): the data is in one such database and you want serializable transactions across its shards.
• 2PC (XA) across your own databases: rarely. Only when the databases can't be merged, the transactions are short, the rows aren't hot, and you run a transaction manager whose log survives it. AWS advises against XA on Aurora MySQL (Part 13).
• A saga with holds and an orchestrator: anything that involves a partner API, a long wait, or several services. The default for payments, bookings and transfers.
• Choreography: short reactions between loosely coupled services with no multi-step undo; never for money.

This is the drill's second question: why a saga over 2PC for a cross-service booking. The partners can't prepare, so 2PC can't include them; 2PC would hold the hot row locked across every round trip and for the whole outage if the coordinator dies after the votes; and a saga's crash blocks no other booking. The price is isolation and undo code.

---

Part 12. One trip, end to end

Each guarantee on this page comes from one layer's durable record and one layer's key. Follow trip-4417 from the click to the email, on the main line.

Synthesizing vector architecture diagram...

What to notice: the boxes run top to bottom in the saga's order, and each names what makes a repeat at that layer harmless: a key, a conditional state change or an event ID. Only "Airline: ticket" can't be undone; the boxes after it need no undo, because they are retriable.

Two rules come out of the table. The saga log decides, and the partners' records confirm: the nightly reconciliation compares the two. And only the pivot is irreversible; everything else repeats safely.

The scheduler loop's summary fits: the engine's state changes exactly once, and every task runs at least once (step 3.4 of the scheduler loop). "The saga is exactly-once" is only true of the log.

---

Part 13. On AWS

Where each piece runs in AWS production, stated only from AWS's public documentation (and, for the self-run engines, their own).

Managed services that use it

Step Functions Standard vs Express, for a saga:

At the hook's 50 trips a minute, 0.83 sagas a second × about 12 state transitions each (nine tasks and a few choices, our design) ≈ 10 transitions a second, against 5,000; and 0.83 StartExecution calls a second against a refill of 300. Both are shared by every workflow in the account and Region, and compensations are transitions too. The hotel loop rejected one workflow per hold for this reason: 500 holds a second is over the 300 a second default (step R2.7 of the hotel loop).

Running it yourself

Look-alikes that are not this mechanism

---

Part 14. What you've learned

Back to the sale

The Lisbon sale charged first and refunded later: $36,260 in kept fees and up to 10 customers charged for nothing. With the design on this page:

• The sold-out hotel costs $0. The seat and rooms are held, not taken; the card is authorized, not charged. F1 aborts at .070 by releasing the seat, and the card is never touched (Part 4). Had the fare changed at the pivot, a free void (F3, Part 5).
• The refund that timed out can't strand a customer. A timeout is logged UNKNOWN and resolved with the same key; a failed void is retried with its own key, alarmed, and reconciled, never dropped: 0 unrefunded (Part 7).
• Nothing irreversible happens early. The ticket is the pivot; the holds are fenced just before it; "booked" is said at 1.685, and the rest is retriable (Part 5). A 20-minute outage with the fence completes the trip; without it, a ticket with no room (Part 6).
• A crash anywhere resumes, never repeats (Part 8), and other bookings see holds as holds (Part 9).
• 2PC wasn't the answer: the partners can't prepare, and a coordinator crash after the votes freezes the flight's row for as long as it's down (Parts 2, 3 and 11).

What it costs

1. No isolation between steps: semantic locks, deltas and rereads in every design that shares data (Part 9).
2. Undo code for every step, each with its own key (Parts 4 and 7).
3. Inventory off sale while held: holds a second × how long each hold stays open (Part 6).
4. An orchestrator, its log, a lease and an epoch to run (Part 8).
5. A sweeper, tombstones, and an alarm on old fenced holds (Part 6).
6. Unknowns to resolve, a review queue, and a nightly reconciliation (Part 7).

The settings

The whole story, event by event

From the reference implementation. Times are seconds after 10:00:00.000; replays run on copies.

The cheat card

Failure checklist

• Does every partner call run as its own saga step, outside any database transaction?
• Is every step and every compensation keyed (<saga>:<step> and :undo), and does every participant return the saved result for a repeat?
• Are compensatable steps first, the one irreversible step the pivot, and only retriable steps after it?
• Does the card get authorized before the pivot and captured after it, never charged early?
• Is every hold expiring by the participant's own clock, with a sweeper that touches only unfenced holds?
• Are the holds fenced (CONFIRMING) just before the pivot, with an alarm on fenced holds older than the longest normal saga?
• Does a cancel for a missing hold leave a tombstone that refuses a late try?
• Is a timeout logged as UNKNOWN and resolved before any compensation runs?
• Is a failed compensation retried with its own key, then alarmed and queued, never dropped, and is there a nightly reconciliation against the PSP?
• Is the saga log written after every reply, with every write conditional on the owner's epoch?
• Does recovery re-run the first unfinished step with the same key, and does a running saga keep its definition?
• Do undos use deltas, do readers treat holds as not final, and does the pivot carry the quote or version it depends on?
• If 2PC is used anywhere: does the coordinator's decision log survive it, does recovery scan every participant, and does the runbook forbid guessing?
• Is lock_timeout set on the rows a prepared transaction may lock, so waiters fail instead of filling the pool?

Think-first drills

Interview questions

Where to go next

• Idempotency & Effectively-Once Processing: the keys, the durable guard, "a timeout is an unknown" and "When a compensation times out" (Parts 2 to 4), and the payment six ways, including the 2PC and saga rows (Part 10).
• Change Streams & the Transactional Outbox: the outbox each step's command and reply leave through (Parts 2 and 3), and the outbox vs 2PC with a broker (Part 10).
• Leases, Fencing Tokens & Distributed Locks: holds as leases, whose clock decides (Part 3), the epoch that fences a paused owner (Parts 4 and 5), and orphaned locks (Part 6).
• Replication, Quorums & Read-Your-Writes: the elections behind a replicated coordinator (Part 6).
• Sharding, Hot Keys & Rebalancing: transactions across shards, and choosing keys to avoid them (Part 9); hot rows.
• Retries, Timeouts, Backpressure & Load Shedding: timeouts from percentiles (Part 2), one retry layer (Part 3), full jitter (Part 4).
• Queues & Delivery Semantics: the ack point, dead-letter queues and redrive for the commands between the orchestrator and its participants.
• Multi-Region Failover: sagas restarted from the replicated log in another Region.
• Background: Primitive #10: Two-phase commit and saga orchestration, Primitive #21: Database isolation levels, Primitive #09: Distributed consensus.
• Drill: The Flight Booking That Charged Without a Seat (question 1 in Parts 5 and 7, question 2 in Parts 2, 3 and 11).
• Loops: payment processing (steps 1.2, 2.1, R2.7, R2.11, 3.5, R3.9), the digital wallet (steps 2.2, 2.3, R2.7, R2.8, R2.11), hotel reservations (steps 1.6, 2.3, 3.1, 3.5, R2.7), the transactional outbox and ledger (steps 1.1, R1.8, 2.2, R3.11), crowdfunding (steps 2.3, 3.4), the job scheduler (steps 3.1, 3.4, R3.11), the key-value store (step R2.3), chat (step 1.3), the news feed (step 3.3), the gaming leaderboard (step 3.5), nearby friends (step 1.2), ride-sharing dispatch (step 2.3), YouTube (step 2.1), Google Drive (step 2.5), ad-click aggregation (step 2.4), the email service (step R1.6), the mobile news feed (step 2.5), mobile stock trading (step 1.2), and the Uber (steps 2.6, 2.7, 3.2, 3.3, R2.9), Shopify (steps 2.4, 3.4, R1.8), Figma (step 3.4) and bot defense (step R1.6) case studies.