Design a Payment Processing System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Payment System?

You Already Use One: the Checkout Button

You click Pay on a shopping site, and a second later the page says "Order confirmed". In that second, money did not move. What happened is that several companies that don't trust each other agreed that it will move, and each of them wrote that down.

Synthesizing vector architecture diagram...

The solid arrows happen in about a second, while the buyer waits. The dotted arrow happens days later. Everything in between is a promise that somebody has to keep track of.

Five words carry this whole loop. Each gets one line now:

• Authorize: the issuer checks the card and puts a hold on the amount. No money moves yet.
• Capture: the merchant says "take the held money now". Online shops often capture at shipping.
• Settle: the banks actually move the money, usually a day or more after capture, and the PSP pays the merchant, minus its fees.
• Refund: the merchant sends money back to the card on purpose.
• Chargeback (dispute): the buyer asks their bank to take the money back, and the bank does, often weeks later. The merchant can contest it with evidence.

What Makes It Hard

Three things make payments harder than most systems that just store data:

1. The network lies. A request to the PSP can time out after the PSP charged the card. From our side, "no answer" looks exactly like "failed", but the money moved.
2. Retries charge twice. The easy fix for a timeout, "try again", is exactly how a customer gets charged twice.
3. Our books must match the bank's to the cent. A missing cent is not a rounding detail. It means money went somewhere we can't explain, and an auditor will ask.

So every component in this loop defends one of three rules:

The Question the Whole Loop Answers

How do we make sure every payment happens exactly once in effect, and every cent is accounted for, when the network can't tell us what happened?

The answer gets sharper every round:

• Round 1: idempotency keys, a state machine, "never trust a timeout", and a double-entry ledger.
• Round 2: the machinery of a real processor: sagas, locking, an outbox, routing between providers, reconciliation, payouts and a card vault.
• Round 3: money across borders, rules and adversaries: sharded ledgers, hot accounts, 3-D Secure, fraud scoring, currencies, home regions and audit.

---

Round 1 · Mid-level · "Card Payments for One Marketplace"

~35 min · SDE II (L5) · 1 region, 3 AZs · ~10 payments/s peak · 99.9% · no double charges

R1.1 Establish Design Scope

The interviewer says: "We run an online marketplace. Design the part that takes the buyer's payment." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Several PSPs. One provider, one integration.
• Our own card storage. The PSP keeps every card.
• Other currencies and countries. US dollars only, but we store the currency anyway.
• Fraud checks. The PSP's built-in checks are enough for now.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into a requirement:

Not yet: authorizing now and capturing later, payouts, and more than one PSP. The interviewer may bring these back.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions, in the order that matters for money:

• Correctness. No double charges, and no lost payments: a buyer who was charged must get their order. This beats every other requirement. A slow payment is an annoyance; a double charge is a support ticket, a refund, and maybe a chargeback.
• Availability. If payments are down, the marketplace sells nothing. But "available" must never mean "guessing".
• Auditability. Finance must be able to explain every dollar: who paid, what we kept, what we owe each seller.
• Latency. About a second at checkout, most of which is the PSP.

R1.4 The API

Create a payment. The checkout makes one idempotency key per payment attempt: a random ID the client generates once and sends again on every retry of the same attempt, so the server can recognize the retry. It is not the order ID: if a card is declined and the buyer tries a different card, that's a new attempt with a new key.

httpCopy
POST /v1/payments HTTP/1.1
Host: payments.internal.example.com
Idempotency-Key: 5f1c2a9e-8d4b-4c61-9a3e-2b7f0d6c1e45
Content-Type: application/json

{
  "order_id": "ord_20260927_000184",
  "amount_minor": 4000,
  "currency": "USD",
  "payment_method_token": "pm_tok_8GQ2x",
  "seller_id": "sel_7731",
  "description": "Order ord_20260927_000184"
}

amount_minor is the amount in the currency's minor unit (cents for US dollars): 4000 means $40.00. Step 1.6 explains why it's an integer.

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{
  "payment_id": "pay_01J9ZK3M7Q",
  "status": "SUCCEEDED",
  "amount_minor": 4000,
  "currency": "USD",
  "psp_reference": "pi_3Q8xYz",
  "created_at": "2026-09-27T14:32:01.458Z"
}

What a retry with the same key returns. The same status code and the same body as the first time, plus a header Idempotent-Replayed: true. A retry never charges again.

Refund:

httpCopy
POST /v1/payments/pay_01J9ZK3M7Q/refunds HTTP/1.1
Host: payments.internal.example.com
Idempotency-Key: 0b6e4d1a-7c3f-4e2a-8f5d-91c2e7a4b380
Content-Type: application/json

{ "amount_minor": 4000, "reason": "ITEM_NOT_RECEIVED" }

It returns 201 with a refund object ("status": "PENDING" until the PSP confirms, then SUCCEEDED or FAILED), and follows the same idempotency rules.

Status: GET /v1/payments/{id} returns the payment with its refunds.

How the card gets to the PSP without touching us:

Synthesizing vector architecture diagram...

The card number crosses from the buyer's browser to the PSP and nowhere else. Our servers only ever see a token that is useless outside our PSP account.

Recap

• One create call with an idempotency key; one refund call; one status call.
• A retry returns the stored result; a different body with the same key is an error.
• "We don't know yet" is a real answer (202 PROCESSING), not a failure.
• Card numbers never touch our servers.
• Correctness first: at most one charge per attempt, and every dollar written down.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From "Call the PSP" to a Ledger

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

The checkout calls the PSP. If the PSP says "succeeded", we write paid = true on the order.

Synthesizing vector architecture diagram...

What's good about it: it's two lines of code, and on a good day it works.

What it costs us: it trusts every message to arrive exactly once, and every answer to come back. Neither is true. The next six steps are the ways it breaks.

Step 1.1: The Buyer Was Charged Twice

The key table, in the same Aurora PostgreSQL database as the payments (so claiming the key and creating the payment can share one transaction):

sqlCopy
CREATE TABLE idempotency_keys (
    scope_key        TEXT PRIMARY KEY,         -- 'payments:5f1c2a9e-...'
    request_hash     BYTEA NOT NULL,           -- SHA-256 of the request body
    status           TEXT NOT NULL CHECK (status IN ('IN_FLIGHT', 'COMPLETED')),
    locked_until     TIMESTAMPTZ NOT NULL,     -- set from the database clock: now() + 60 s
    payment_id       TEXT,                     -- filled once known
    response_code    INT,
    response_body    JSONB,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

The claim is one statement:

sqlCopy
INSERT INTO idempotency_keys (scope_key, request_hash, status, locked_until)
VALUES ($1, $2, 'IN_FLIGHT', now() + interval '60 seconds')
ON CONFLICT (scope_key) DO NOTHING;

If it inserted zero rows, somebody else holds the key, and we read the row to decide between replay, 409 and 422. The lock time comes from the database's clock (now()), so two app servers with slightly different clocks can't disagree about whether a lock expired. A request that finds an IN_FLIGHT row whose locked_until has passed takes over by updating the row only if locked_until still has the value it read, and then checks whether a payment already exists before doing anything (step 1.2 explains what it does next). A nightly job deletes rows older than 7 days, in batches of a few thousand so it never locks the table for long.

Step 1.2: The PSP Call Timed Out. Did It Charge?

Why 5 seconds? The PSP's normal P99 is 300 to 600 ms, so 5 seconds is far above normal and a timeout really means trouble. A shorter timeout would turn slow-but-successful charges into unknowns we then have to resolve; a much longer one leaves the buyer staring at a spinner.

Synthesizing vector architecture diagram...

The replay is safe only because the key is the same. It does not charge again; it reads the first request's saved result.

Primitive: Circuit Breaker, Bulkhead and Fault-Tolerance Patterns (timeouts, retries and backoff with jitter)

Step 1.3: The Statuses Are a Mess of Booleans

Synthesizing vector architecture diagram...

There is no arrow out of FAILED and no arrow from UNKNOWN to anything but a confirmed answer. The only way to leave UNKNOWN is for the PSP to tell us.

Step 1.4: The PSP Tells Us Things Later

Disputes after refunds are allowed. It's tempting to write "a refunded payment ignores dispute events". That's wrong. A buyer can still dispute a partially refunded charge, and Stripe notes that a dispute can arrive while a refund is still on its way (it fails such a refund with the reason charge_for_pending_refund_disputed). So DISPUTED must be reachable from SUCCEEDED, PARTIALLY_REFUNDED and REFUNDED, and the dispute handling must check how much was already refunded so the buyer isn't paid back twice. Round 2 builds the dispute flow; in Round 1 we record the event and alert support.

We confirm a refund from the refund's own status (refund.updated moving to succeeded), not from charge.refunded: Stripe sends charge.refunded when a refund is created, before the refund has actually gone through. | charge.dispute.created | SUCCEEDED, PARTIALLY_REFUNDED or REFUNDED | Record the dispute, alert support |

Step 1.5: Finance Can't Tell Where the Money Went

Debits and credits in one paragraph. Assets (things we have or are owed, like psp_receivable) go up with a debit. Liabilities (things we owe, like seller_payable) and revenue go up with a credit. Expenses (like psp_fees) go up with a debit. So "the PSP owes us more" is a debit, and "we owe the seller more" is a credit, and in every transaction the two sides are equal.

One $40.00 payment, with the PSP's fee of 2.9% + 30¢ ($1.16 + $0.30 = $1.46) and our 10% commission ($4.00):

The PSP keeps its fee and will pay us $38.54; we owe the seller $36.00 and keep $4.00. The fee is the amount the PSP reports in its response, never one we compute ourselves: our entries must match what the PSP actually does.

A full refund of that payment. PSPs generally don't return their processing fee on a refund (Stripe says so), and our policy (a choice) is to return our commission:

After both: the seller is owed $0, our revenue is $0, and psp_receivable is down $1.46 net, which is exactly the fee the PSP kept, still visible in psp_fees. Every cent is explained.

sqlCopy
CREATE TABLE accounts (
    account_id     BIGSERIAL PRIMARY KEY,
    account_code   TEXT UNIQUE NOT NULL,    -- 'seller_payable:sel_7731'
    account_type   TEXT NOT NULL CHECK (account_type IN ('ASSET','LIABILITY','REVENUE','EXPENSE','EQUITY')),
    currency       CHAR(3) NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE journal_transactions (
    journal_id     BIGSERIAL PRIMARY KEY,
    journal_key    TEXT UNIQUE NOT NULL,    -- 'payment:pay_01J9ZK3M7Q:capture'; a repeat post fails here
    payment_id     TEXT,
    kind           TEXT NOT NULL,           -- 'CAPTURE', 'REFUND', 'DISPUTE', 'REVERSAL'
    posted_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE ledger_entries (
    entry_id       BIGSERIAL PRIMARY KEY,
    journal_id     BIGINT NOT NULL REFERENCES journal_transactions(journal_id),
    account_id     BIGINT NOT NULL REFERENCES accounts(account_id),
    direction      CHAR(1) NOT NULL CHECK (direction IN ('D','C')),
    amount_micros  BIGINT NOT NULL CHECK (amount_micros > 0),
    currency       CHAR(3) NOT NULL
);
CREATE INDEX ON ledger_entries (account_id, entry_id);
CREATE INDEX ON ledger_entries (journal_id);

Two rules make the ledger trustworthy, and both are enforced by the database, not by hoping every code path is right:

• Balanced journals. A deferred constraint trigger runs at commit and rejects the transaction if, for any journal it touched, debits and credits differ in any currency.
• Append-only. The application's database role may only insert into the ledger tables: REVOKE UPDATE, DELETE ON ledger_entries, journal_transactions FROM payments_app;

A balance is a query:

sqlCopy
SELECT SUM(CASE WHEN direction = 'C' THEN amount_micros ELSE -amount_micros END) AS owed_micros
FROM ledger_entries
WHERE account_id = $1;   -- a liability account: credits minus debits

Primitive: Database Isolation Levels, ACID and Concurrency Anomalies (why the whole journal commits or nothing does)

Step 1.6: The Amounts Are Off by a Cent

A worked example: a $33.33 order, for a seller whose contract says the seller bears the PSP fee. Our 10% commission is 333.3 cents; half-even to the cent gives $3.33. The PSP reports its fee: 2.9% × $33.33 + $0.30 = $1.26657, which it rounds to $1.27. We record the fee the PSP reports, compute our commission, and derive the seller's share by subtraction: $33.33 − $1.27 − $3.33 = $28.73.

The seller's line is derived by subtraction, so the journal balances by construction. psp_fee_recovery offsets psp_fees, so our net fee cost for this order is zero, and both facts stay visible.

(Who bears the PSP fee is a business rule. In the $40 example above, we absorbed it; here the seller does. Pick one rule per contract and write it in the journal code, not in each report.)

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Follow a payment: the browser sends only a token; WAF filters abuse; the service claims the key, creates the payment and calls the PSP; the result and the ledger entries commit in one Aurora transaction. Webhooks come in through the same front door and change payments through the same allowed-moves rules. The resolver works only on payments stuck in UNKNOWN.

The pieces:

• Payment service: a stateless container on ECS Fargate (containers without managing servers), three tasks, one per AZ. It serves the API and the webhook endpoint.
• Aurora PostgreSQL: one writer and one replica in another AZ, which Aurora promotes if the writer fails. It holds the key table, payments, refunds, webhook events and the ledger. One database means one transaction can cover "claim key, create payment, post ledger".
• Resolver: a scheduled Fargate task that runs every 30 seconds, picks payments that have been in UNKNOWN or PROCESSING for more than 10 seconds, and asks the PSP with the backoff from step 1.2 (it stores each payment's next try time, so a run skips payments that aren't due). It also runs the nightly key cleanup.
• Secrets: the PSP API key and the webhook signing secret live in AWS Secrets Manager, read at start-up.
• Egress: calls to the PSP leave through NAT gateways (one per AZ).

The remaining tables:

sqlCopy
CREATE TABLE payments (
    payment_id       TEXT PRIMARY KEY,             -- 'pay_01J9ZK3M7Q'
    scope_key        TEXT UNIQUE NOT NULL,             -- permanent guard; key rows are deleted after 7 days
    order_id         TEXT NOT NULL,
    seller_id        TEXT NOT NULL,
    amount_minor     BIGINT NOT NULL CHECK (amount_minor > 0),
    refunded_minor   BIGINT NOT NULL DEFAULT 0,
    currency         CHAR(3) NOT NULL,
    status           TEXT NOT NULL,
    psp_reference    TEXT,
    next_check_at    TIMESTAMPTZ,                  -- resolver schedule while UNKNOWN
    version          INT NOT NULL DEFAULT 0,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CHECK (refunded_minor >= 0 AND refunded_minor <= amount_minor)
);

CREATE TABLE refunds (
    refund_id        TEXT PRIMARY KEY,
    payment_id       TEXT NOT NULL REFERENCES payments(payment_id),
    scope_key        TEXT UNIQUE NOT NULL,
    amount_minor     BIGINT NOT NULL CHECK (amount_minor > 0),
    status           TEXT NOT NULL CHECK (status IN ('PENDING','SUCCEEDED','FAILED','UNKNOWN')),
    psp_reference    TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE payment_events (
    event_id         BIGSERIAL PRIMARY KEY,
    payment_id       TEXT NOT NULL REFERENCES payments(payment_id),
    from_status      TEXT,
    to_status        TEXT NOT NULL,
    cause            TEXT NOT NULL,                -- 'api', 'resolver', 'webhook:evt_1Q...'
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE webhook_events (
    psp_event_id     TEXT PRIMARY KEY,             -- dedupe
    event_type       TEXT NOT NULL,
    received_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    payload          JSONB NOT NULL
);

Trace 1: a successful payment.

Synthesizing vector architecture diagram...

The payment row exists before the PSP is called. If our service dies between the PSP call and the second transaction, the resolver finds a PROCESSING payment and asks the PSP; nothing is lost.

Trace 2: a timeout resolved by asking. The PSP charges, but the reply never arrives. At 5 s the service sets UNKNOWN, sets next_check_at, and returns 202 PROCESSING; the page starts polling. On its next run, the resolver replays the charge with the same PSP key and gets the saved "succeeded". In one transaction it moves the payment to SUCCEEDED, posts the journal, and completes the key with a 201 response, so a buyer's retry now replays 201. The page's next poll shows "Payment confirmed". The PSP's payment_intent.succeeded webhook arrives later and finds nothing left to do.

Trace 3: a refund confirmed by webhook.

Synthesizing vector architecture diagram...

The refund journal is posted when the PSP confirms the refund, not when we ask for it, because until then the money hasn't moved. If a refund.failed arrives later, a reversing journal undoes it and the payment moves back.

R1.7 Numbers

Traffic.

$$\text{Average} = \frac{50{,}000}{86{,}400 \text{ s}} \approx 0.58 \text{ payments/s}$$

For the peak we assume (and would confirm with the interviewer) that 30% of the day's payments land in the busiest two hours, and that a promotion can multiply that five-fold for a few minutes:

$$\frac{0.30 \times 50{,}000}{7{,}200 \text{ s}} \approx 2.1/\text{s} \;\Rightarrow\; 2.1 \times 5 \approx 10 \text{ payments/s at peak}$$

Each payment is about ten row writes (key claim, payment insert, status update, two history events, journal, four entries, key completion, rounded), so the peak is about 100 row writes a second. One small database does that easily.

Ledger rows per day. Four entries per payment, and we assume 5% of payments get a refund journal of three entries:

$$50{,}000 \times 4 + 2{,}500 \times 3 = 207{,}500 \text{ entries/day}$$

Storage. About 1 KB per payment all in: the payment row (~350 bytes), four ledger entries (~100 bytes each, ~400 bytes), and events and journal header (~250 bytes).

$$50{,}000 \times 1 \text{ KB} = 50 \text{ MB/day} \;\Rightarrow\; 50 \text{ MB} \times 365.25 \approx 18.3 \text{ GB/year}$$

With indexes (about 1.6× the raw data, an assumption), about 29 GB a year.

Idempotency keys. About 52,500 a day (payments plus refunds), kept 7 days, at up to ~1 KB each with the stored response:

$$52{,}500 \times 7 \times 1 \text{ KB} \approx 368 \text{ MB}$$

Availability. 99.9% of a 30.4-day month is $43{,}776 \text{ min} \times 0.001 \approx 43.8$ minutes. Note that the PSP is in our path: if the PSP is down, we are down for new payments, whatever our own uptime.

Latency. Dependent steps, so they add: key claim and payment insert (~5 ms), PSP call (300 to 600 ms at P99), commit (~5 ms), network inside the region (~10 ms). That's about 620 ms at P99, inside the "about a second" target, and almost all of it is the PSP.

Monthly cost (us-east-1 on-demand prices, 730 hours a month):

Now compare with what the PSP charges. At a common published card rate of 2.9% + 30¢ and our $40 average order, each payment costs $1.46 in fees:

$50{,}000 \times \$1.46 = \$73{,}000/\text{day} \approx \$2.2\text{M/month}$

Say it in the interview: the infrastructure is about 0.03% of the payment fees. Saving money on servers is irrelevant here. Every design choice in this round is about correctness.

R1.8 Trade-Offs

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Puts correctness first and says so: no double charge, no lost payment.
• Uses an idempotency key with an atomic claim, stores the response, and passes a key to the PSP.
• Treats a timeout as "unknown", never as "failed", and knows how to find out: replay with the same key inside the PSP's window, or look up.
• Models the payment as a state machine with allowed moves, enforced where the data lives.
• Verifies webhooks, dedupes them, and doesn't assume their order.
• Writes money movements as balanced double entries in integer units.

Follow-up questions

1. "Why not use the order ID as the idempotency key?" Answer: because an order can have several legitimate payment attempts. If the first card is declined and the buyer tries another, an order-ID key would replay the decline forever. The key identifies one attempt; the order can have many, and the service can still refuse a second successful payment for one order by checking the order's paid state inside the same transaction.

2. "The buyer closes the tab while the payment is UNKNOWN. What happens?" Answer: nothing depends on the tab. The resolver finishes the payment, the order moves to paid or cancelled, and the buyer gets an email. If they come back and click Pay again, the checkout reuses the same key for the same attempt (it keeps the key with the order attempt), so they get the stored result instead of a second charge.

3. "Why store the response instead of just a 'done' flag?" Answer: because a retry must get the same answer as the first request, including the payment ID and the PSP reference. With only a flag, the retry would have to rebuild the answer, and anything that changed since (a refund, say) would make it different.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "20M Payments a Day, Many PSPs, Payouts"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 2,500 payments/s peak · 12,500 reads/s · 99.99% · the books balance to the cent

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built card payments for one marketplace: 50,000 payments a day, about 10 a second at peak, one PSP, US dollars. The browser sends the card to the PSP through hosted fields, so we only see a token. Every request carries an idempotency key; we claim it with a conditional insert in the same Aurora PostgreSQL database as the payment, store the response, and replay it on retries. We pass our own key to the PSP. A timeout is never a failure: the payment goes to UNKNOWN, the buyer sees 'confirming', and a resolver asks the PSP by replaying with the same key inside the PSP's key window, or by looking the payment up, with backoff and full jitter. Statuses are one state machine with allowed moves enforced by compare-and-set. Webhooks are verified with HMAC, rejected after 300 seconds, deduped by event ID and applied only as allowed moves, and a dispute can follow a refund. Money is a double-entry ledger in integer micros: balanced journals, derived balances, append-only. About $600 a month, against $2.2 million a month in PSP fees. Open costs: one PSP, one database for everything, and nobody checks our books against the PSP's."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: one database holds keys, payments and the ledger, so one transaction keeps them consistent; the resolver turns unknowns into answers.

Round 1 step summary

Open costs: one PSP is a single point of failure; one database does everything; nothing proves our books match the PSP's and the bank's.

R2.1 The Scope Raise

Interviewer: "The marketplace worked so well that we're turning it into a payments platform. Twenty thousand merchants, 20 million payments a day, 2,500 a second at peak. Many merchants want to authorize at checkout and capture when they ship. We'll use three PSPs, routed by cost and approval rate, and we must keep taking payments when one of them is failing. We now pay merchants out to their bank accounts ourselves. Finance wants the nightly bank and PSP files to match our ledger to the cent. We want our own card vault so we're not locked into one PSP. Disputes are ours to handle. And customers now have a stored wallet balance, which they spend from their phone and their laptop."

As in Round 1, we ask back before fixing anything.

Scope change

The "Not yet" list from R1.2 is now mandatory: authorize and capture, payouts, and several PSPs.

About "availability": we promise 99.99% for our API. We can't promise that end to end, because PSPs, card networks and issuers are in the path and publish their own targets. We measure our own availability separately from each PSP's, and routing (step 2.4) is how we stay up when one PSP isn't.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

Authorize now, capture later

httpCopy
POST /v1/payments HTTP/1.1
Host: api.payments.example.com
Authorization: Bearer <merchant API key>
Idempotency-Key: 7a0c4b2e-1f6d-4c8a-b3e9-5d2f8e1a6c07
Content-Type: application/json

{
  "amount_minor": 12999,
  "currency": "USD",
  "payment_method": "vtok_9f3kQ2",
  "capture_method": "MANUAL",
  "merchant_reference": "order-551-2207"
}

It returns 201 with "status": "REQUIRES_CAPTURE" and "capture_before": "2026-10-04T09:12:00Z", the deadline after which the authorization lapses (taken from the PSP's answer, never assumed).

httpCopy
POST /v1/payments/pay_01JA2B/capture HTTP/1.1
Idempotency-Key: 3e8d1c7a-cap-2207
Content-Type: application/json

{ "amount_minor": 11999 }

A capture may be for less than the authorized amount; the rest is released. For most card payments you can capture only once (Stripe documents this), so the API rejects a second capture with 409. POST /v1/payments/{id}/cancel releases an uncaptured authorization.

Routing rules (an admin API, versioned, read by the router every 30 seconds):

jsonCopy
{
  "version": 42,
  "rules": [
    { "match": { "card_country": "US", "card_type": "debit" }, "prefer": ["PSP_B", "PSP_A"] },
    { "match": { "card_country": "*" }, "prefer": ["PSP_A", "PSP_B", "PSP_C"] }
  ],
  "min_approval_rate_to_stay_primary": 0.85
}

Payouts: GET /v1/merchants/{id}/payouts lists payouts with their status (PENDING, SENT, SETTLED, RETURNED) and the payments each one covers. A merchant sets its bank account and schedule through the dashboard.

Reconciliation reports: GET /v1/reconciliation/runs/{date} returns counts and amounts per category (matched, amount mismatch, missing on one side, timing) per PSP, for finance.

Dispute events to merchants (our own webhooks, signed the same way PSPs sign theirs):

jsonCopy
{
  "id": "evt_01JA3K9",
  "type": "payment.disputed",
  "created": "2026-10-19T08:14:02Z",
  "data": {
    "payment_id": "pay_01JA2B",
    "dispute_id": "dp_01JA3K",
    "amount_minor": 11999,
    "reason": "fraudulent",
    "evidence_due_by": "2026-11-02T23:59:59Z"
  }
}

Wallets: POST /v1/wallets/{id}/debits with an idempotency key, which fails with 402 and insufficient_funds if the balance can't cover it.

R2.4 Design Evolution: The Machinery of a Real Processor

Step 2.1: A Payment Is Five Steps Across Three Systems

Synthesizing vector architecture diagram...

The saga for one manual-capture payment. There is no separate notify state: PostLedger writes the journal, the status and an outbox row in one transaction, and the merchant notification flows from the outbox (step 2.3). Two arrows lead to Capture because the merchant may call capture before the execution even starts waiting; the next paragraph shows why that race is safe.

The capture race, made safe by the row lock. The workflow saves its task token on the payment row; the capture API marks the row capture_requested. Each does it with one UPDATE ... RETURNING on the same row, so Postgres runs them one after the other:

• The workflow runs UPDATE payments SET task_token = $t WHERE payment_id = $1 RETURNING capture_requested. If capture_requested is already true, it goes straight to Capture.
• The capture API runs UPDATE payments SET capture_requested = true WHERE payment_id = $1 AND status = 'REQUIRES_CAPTURE' RETURNING task_token. If a token is there, it calls SendTaskSuccess with it; if not, the workflow will see the flag when it registers.

Whichever runs second sees the first one's write, so a capture request can never be missed.

Primitive: Two-Phase Commit and Saga Orchestration

Drill: The flight booking that charged without a seat (a half-failed compensation is answered above; saga versus 2PC is the first wrong answer)

Step 2.2: Two $40 Spends From a $50 Wallet Both Succeeded

Primitive: Database Isolation Levels, ACID and Concurrency Anomalies

Drill: The on-call doctor anomaly (why snapshot isolation misses write skew, and why not SERIALIZABLE everywhere, are both answered above)

Step 2.3: The Ledger and the Event Stream Disagree

The same stream also starts the manual-capture sagas: a small consumer reads payment.requires_capture events and calls StartExecution with the payment ID as the name, which is idempotent while the execution runs.

Primitive: Change Data Capture and the Outbox Pattern

Drill: The dual-write that broke search consistency (the commit-succeeded-publish-failed case and CDC versus a polling outbox are both answered above)

Step 2.4: PSP A Is Failing, and Approvals Are Dropping

Primitive: Circuit Breaker, Bulkhead and Fault-Tolerance Patterns

Drill: The slow recommendation service that took down checkout (why a slow dependency exhausts threads and why tiny timeouts aren't the answer are both covered above)

Step 2.5: Our Books Don't Match the Bank's

Synthesizing vector architecture diagram...

Two checks, both required: the line-by-line join proves every payment is on both sides, and the net check proves the money that actually landed in the bank is what the lines add up to.

Step 2.6: Merchants Need Their Money

Synthesizing vector architecture diagram...

The ledger always changes before money moves out. Settlement is posted on the settlement date for every payout, so payouts_in_transit always empties; a return moves the money back onto the merchant's balance, so the next payout can pay it once the bank details are fixed.

Step 2.7: We're Locked Into One PSP's Card Tokens

sqlCopy
CREATE TABLE card_vault (
    token_id         TEXT PRIMARY KEY,             -- 'vtok_9f3kQ2', random
    fingerprint      BYTEA NOT NULL,               -- HMAC-SHA256(card number, fingerprint key)
    kms_key_arn      TEXT NOT NULL,
    key_version      INT NOT NULL,
    encrypted_dek    BYTEA NOT NULL,               -- data key, encrypted by KMS
    ciphertext_pan   BYTEA NOT NULL,               -- card number, AES-256-GCM under the data key
    last4            CHAR(4) NOT NULL,
    brand            TEXT NOT NULL,
    exp_month        SMALLINT NOT NULL,
    exp_year         SMALLINT NOT NULL,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    rewrapped_at     TIMESTAMPTZ
);
CREATE INDEX ON card_vault (fingerprint);

Round 2 Step Summary

R2.5 Architecture v2

Now the concepts get AWS names. Two pictures: the request path, then the money and data path.

Synthesizing vector architecture diagram...

The request path. The API claims the key in DynamoDB, authorizes through the router, and commits the payment, its ledger entries and an outbox row in one Aurora transaction. Card numbers exist only inside the vault and on the wire to the PSP.

Synthesizing vector architecture diagram...

The money and data path. Every change leaves the ledger through the WAL, so sagas, notifications, the lake and reconciliation all see the same committed history. Payouts wait for reconciliation.

What moved since Round 1, and why:

• Idempotency keys moved to DynamoDB. The key check is the first thing every request does, including abusive retry loops, and the Aurora writer is the one machine we can't scale out. DynamoDB takes the conditional writes, and its TTL feature deletes 7-day-old keys without costing us write capacity. Aurora stays the source of truth: the payments table keeps a UNIQUE (merchant_id, scope, idempotency_key) constraint forever, so even a retry after the DynamoDB item is gone can't create a second payment. (R2.8 covers what happens when the two disagree.)
• Webhooks are verified and stored by the API tier (insert into webhook_events, dedupe by event ID, plus an outbox row, then 200); the processing happens behind the stream, so a burst of deliveries can't overwhelm payments.
• Reads go to three read replicas. Anything that decides something (a balance check before a debit, "is this captured?" before a refund) reads the writer.

The DynamoDB key item:

The claim is a PutItem with ConditionExpression: attribute_not_exists(pk). Two details that matter:

• DynamoDB has no "current time" in conditions, so lease_until comes from our servers' clocks. A request that finds an IN_FLIGHT item treats the lease as expired only when lease_until is at least 5 seconds (our clock-skew margin) in the past, and takes it over with an UpdateItem conditioned on lease_until still equalling the value it read. Only one taker can win, and no decision depends on DynamoDB's clock.
• TTL deletes items eventually, typically within a few days of expiry, not at the instant. So an item can outlive its 7 days; replaying it is harmless because it's the true answer. And we never use TTL as the lease: the lease is lease_until, checked by us.

Trace 1: authorize, then capture three days later.

Synthesizing vector architecture diagram...

The buyer waits only for the top half. Everything after "three days later" is the saga, and each step is idempotent.

Trace 2: PSP A is failing. At 14:05, PSP A's timeouts pass 20% for 30 seconds and its breaker opens. New payments with a vault token route to PSP B (the vault forwards the card there); payments with a PSP-A-only token (older cards not yet in our vault) get 503 with "try again shortly". The 400 or so payments that timed out at A before the breaker opened are UNKNOWN; the resolver asks A, gets answers as A recovers, and only those A confirms never happened are left for the buyer to retry. At 14:20 the breaker goes half-open, 1% of eligible traffic tries A, succeeds, and the router shifts A back to its normal share over a few minutes.

Trace 3: a reconciliation break. At 06:20, the Glue job finds one PSP A line with no ledger match: du_1QK..., a $50.00 dispute plus a $15.00 fee. The webhook for it had failed signature verification during a secret rotation and was never processed. The net check for A's batch is off by the same $65.00, so the break is explained, not just detected. The merchant is held from today's payout; an operator replays the event from the PSP (Stripe keeps events available to resend for weeks), the dispute journal posts, the merchant's balance drops by $65, and tomorrow's run shows zero breaks for A.

Trace 4: a payout with a return. On Tuesday the payout job debits merchant_payable:m_4410 $8,200.00 and credits payouts_in_transit, then sends the ACH file. On Wednesday, the settlement date, it debits payouts_in_transit and credits bank_cash $8,200.00. On Thursday the return file shows R03 for that entry. The job debits bank_cash, credits merchant_payable:m_4410 $8,200.00, marks the payout RETURNED and blocks the bank account. The merchant updates its bank details on Friday and the next payout includes the $8,200.00.

Ledger entries for one payment and one refund (the $129.99 authorization captured at $119.99; PSP fee 2.9% + 30¢ = $3.78, which PSP A reports after rounding $3.77971; platform fee 1% = $1.20; the merchant bears the PSP fee):

Check the capture: $119.99 − $3.78 − $1.20 = $115.01 to the merchant, and $116.21 + $3.78 = $119.99. The refund returns our 1% on the refunded part ($0.1999, half-even to $0.20) and takes the rest from the merchant; the PSP fee is not returned.

R2.6 Numbers and Cost

Traffic.

$$\text{Average} = \frac{20 \times 10^6}{86{,}400 \text{ s}} \approx 231.5/\text{s} \qquad \text{Peak} = 231.5 \times 10 \approx 2{,}315 \;\Rightarrow\; \text{plan for } 2{,}500/\text{s}$$

The 10× covers the daily peak hour plus flash sales. Status reads are 5 per payment at peak: $2{,}500 \times 5 = 12{,}500$ reads/s.

Storage. About 1 KB per payment all in, as in Round 1 (payment row, entries, events, journal):

$$20 \times 10^6 \times 1 \text{ KB} = 20 \text{ GB/day} \qquad 20 \text{ GB} \times 365.25 \approx 7.3 \text{ TB/year} \qquad \times 5 = 36.5 \text{ TB}$$

With indexes (1.6×, an assumption): about 11.7 TB after one year and 58.4 TB after five, inside one Aurora cluster's limit (128 TiB, or 256 TiB on recent Aurora PostgreSQL versions). Aurora keeps six copies across three AZs but bills storage once, so the storage bill is for 58.4 TB, not three times that. We partition the ledger tables by month, so old months can later be detached and archived to S3 without a giant delete.

Idempotency keys.

$$20 \times 10^6/\text{day} \times 7 \text{ days} \times 500 \text{ B} = 70 \text{ GB}$$

That's the steady state for 7 days of keys. Because TTL deletes lag by up to a few days, we budget for somewhat more.

Bandwidth (4 KB per request or response, from Round 1's payloads plus headers):

$$\text{In} = 2{,}500 \times 4 \text{ KB} = 10 \text{ MB/s} = 80 \text{ Mbps} \qquad \text{Out} = (12{,}500 + 2{,}500) \times 4 \text{ KB} = 60 \text{ MB/s} = 480 \text{ Mbps}$$

Out counts status replies plus our calls to PSPs. Averaged over a month: $(1{,}157.5 + 231.5) \times 4 \text{ KB} \approx 5.6$ MB/s, about 14.6 TB a month leaving AWS.

Concurrency to the PSPs (Little's law): $2{,}500/\text{s} \times 0.6 \text{ s (P99)} = 1{,}500$ calls in flight at peak, split across the three bulkheads by traffic share.

Database sizing. Each payment is roughly ten row writes, so peak is about 25,000 row writes a second on one writer. We plan on a db.r6g.8xlarge writer (32 vCPUs) and an identical standby in a second AZ that Aurora promotes on failure. This is a planning assumption we load-test before launch: the writer is the one component that doesn't scale out, and Round 3 shards it. For reads, we plan 8,000 simple indexed reads a second per db.r6g.2xlarge replica (also to be load-tested) and run three, one per AZ: losing an AZ leaves two, $2 \times 8{,}000 = 16{,}000 \ge 12{,}500$ (78% busy).

API fleet. We plan 500 requests a second per vCPU (an assumption to load-test): $15{,}000 / 500 = 30$ vCPUs at peak. To survive losing one of three AZs, the remaining two-thirds must carry peak: $30 / (2/3) = 45$ vCPUs. We run 24 tasks of 2 vCPU (48 vCPUs), 8 per AZ.

Step Functions. Standard workflows cost $0.000025 per state transition ($25 per million) in us-east-1. Only manual-capture payments run a saga, 4 transitions each on the happy path (RegisterToken, WaitForCapture, Capture, PostLedger):

$$0.40 \times 20 \times 10^6 \times 4 = 32 \times 10^6 \text{ transitions/day}$$

Disputes (0.1% of payments, assumed, about 6 transitions each) and payouts (20,000 merchants × about 5) add about 0.22 million a day:

$32.22 \times 10^6 \times \$0.000025 \approx \$806/\text{day} \approx \$24.5\text{K/month}$

If every payment ran a 10-transition saga, the same math gives $5,000 a day, about $152K a month. That's why auto-capture payments don't use a saga.

Step Functions quotas. One default in us-east-1 is below our peak: StartExecution for Standard refills at 300 a second (bucket 1,300), and our manual-capture starts peak at $0.4 \times 2{,}500 = 1{,}000$ a second. The others have headroom: state transitions refill at 5,000 a second against roughly 2,300 at peak, and SendTaskSuccess at 500 a second against about 93 (captures arrive spread over days, at about the average rate of $0.4 \times 231.5$). These are soft quotas that AWS can raise; we request a StartExecution increase, load-test it, and have the saga starter pace StartExecution calls, since a saga that starts a few seconds late only waits for a capture that's hours away anyway (the capture race above is safe in either order).

Monthly cost (us-east-1 on-demand prices, 730 hours a month, storage at the end of year one):

Aurora storage grows by about $1,170 a month each year, to about $5,840 a month at year five (58.4 TB). Watch the I/O line: if Aurora I/O ever passes about a quarter of the Aurora bill, Aurora I/O-Optimized (no per-I/O charge, higher instance and storage prices) becomes cheaper.

Now the number that matters. At 2.9% + 30¢ on a $40 average, PSP fees are 20 \times 10^6 \times \1.46 = $29.2M a day, about **\888M a month**. Our whole platform is about 0.005% of that. Large platforms negotiate far better rates than list price, but the conclusion holds: a 0.1-point better routing decision is worth more than the entire AWS bill. That's why routing by cost and approval rate (step 2.4) is a revenue feature, not an optimization.

Availability budget. 99.99% of a 30.4-day month is $43{,}776 \times 0.0001 \approx 4.4$ minutes. An Aurora failover of tens of seconds uses a real part of that, so we don't spend it on deploys: rolling deploys, one AZ at a time.

Latency budget (P99, dependent steps add): key claim in DynamoDB (~10 ms) + payment insert (~5 ms) + router and vault decrypt (~20 ms) + PSP (≤ 600 ms) + commit with ledger and outbox (~15 ms) + network inside the region (~20 ms) ≈ 670 ms, under the 800 ms target. The PSP is 90% of it.

R2.7 Trade-Offs

Four ways to keep several writes consistent, with verified limits:

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Rejects 2PC with the right reason (external PSPs, locks held across network calls) and designs a saga with compensations that can themselves fail safely.
• Chooses between Standard and Express workflows by execution semantics, duration and price, and knows the quotas.
• Explains write skew under snapshot isolation and fixes the balance race with explicit locks where the invariant lives.
• Uses the outbox with CDC instead of dual writes.
• Routes between PSPs with bulkheads and breakers, and never fails over a payment in an unknown state.
• Reconciles line by line with a full outer join, accounts for every row, and holds payouts on breaks.
• Designs payouts ledger-first with returns, and a vault with envelope encryption, knowing what KMS rotation does and doesn't do.
• Says out loud that PSP fees dwarf infrastructure.

Follow-up questions

1. "A merchant captures, then immediately refunds, and the capture's ledger post is still in flight. What happens?" Answer: the refund API checks the payment's status on the writer. While it's CAPTURING, a refund isn't an allowed move yet, so the API returns 409 with Retry-After. Once the capture journal commits (seconds), the refund proceeds. We never refund money we haven't recorded as captured.

2. "Why not let merchants call PSP B directly when PSP A is down?" Answer: because every payment must go through our idempotency, routing and ledger. A payment made around us is money we don't know about: it would show up in reconciliation as "missing in our ledger", and the merchant's balance and payouts would be wrong.

3. "Your reconciliation says everything matched, but a merchant says they're owed $200 more. Who's right?" Answer: we list the merchant's ledger entries for the period; each links to a payment, refund, dispute or payout, and each of those links to a reconciled PSP or bank line. Either the entries explain the balance (and we show them), or one is missing, which reconciliation would have flagged unless both we and the PSP missed it, and then the bank net check is the tiebreaker. The ledger's value is that the answer is a query, not an argument.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Global Money: Currencies, Rules, Fraud, Scale"

~45 min · Principal (L7) · 4 home regions, each with a disaster-recovery copy · 200M payments/day · 25 currencies · no double-spend through a region failover

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Round 2.

Round 2 in 60 seconds. "We built a payments platform in one region: 20,000 merchants, 20 million payments a day, 2,500 a second at peak, 12,500 status reads a second. Idempotency keys are claimed in DynamoDB with a conditional put and a lease timed by our own clock, and Aurora's unique constraint is the source of truth. Payments authorize inline; the 40% that capture later run a Step Functions Standard saga that waits for the merchant with a task token, voids before the authorization expires, and compensates safely. Wallet spends lock the account row, with a per-account never-negative constraint, because snapshot isolation allows write skew. Every ledger change leaves through an outbox read by Debezium into MSK. The router picks among three PSPs by card, cost and live approval rate, with a bulkhead and a circuit breaker per PSP, and never fails over a payment whose outcome is unknown. A nightly Glue job reconciles every line three ways and holds payouts on breaks. Payouts are ledger-first ACH batches with returns handled. Cards live in an isolated vault with KMS envelope encryption. About $46K a month, against about $888 million a month in PSP fees. Open costs: one database writer, accounts that every payment touches, and one region."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: one writer holds all the money; everything else reads its log.

Round 2 step summary

Open costs: one Aurora writer caps us; accounts that every payment touches serialize on a row lock; one region, one currency, and no defense of our own against fraud.

R3.1 The Scope Raise

Interviewer: "We're going global. Forty countries, 25 currencies, 200 million payments a day. Cards issued in Europe need strong customer authentication. Last month fraudsters ran stolen cards through one of our merchants at a thousand attempts a minute. One of our markets requires payment data to stay in the country. The regulators audit us now. Our own revenue account is written by every single payment. And a whole region can fail, but money must never be double-spent."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

The payment intent with a "requires action" state. A payment intent is one object that tracks a payment from "the buyer wants to pay" to the final result, across however many steps it takes.

httpCopy
POST /v1/payment_intents/pi_01JB7/confirm HTTP/1.1
Host: api.eu.payments.example.com
Idempotency-Key: 2c9f0e6b-confirm-1
Content-Type: application/json

{ "payment_method": "vtok_eu_77Hc", "return_url": "https://shop.example/checkout/return" }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "pi_01JB7",
  "status": "REQUIRES_ACTION",
  "amount_minor": 8900,
  "currency": "EUR",
  "next_action": { "type": "three_ds_challenge", "url": "https://acs.issuer.example/challenge/abc" },
  "expires_at": "2026-10-02T10:44:00Z",
  "risk": { "score": 34, "decision": "CHALLENGE" }
}

FX quotes:

httpCopy
POST /v1/fx/quotes HTTP/1.1
Content-Type: application/json

{ "sell_currency": "EUR", "buy_currency": "USD", "sell_amount_minor": 10000 }

jsonCopy
{
  "quote_id": "fxq_01JB8",
  "rate": "1.08500000",
  "sell_amount_minor": 10000,
  "buy_amount_minor": 10850,
  "expires_at": "2026-10-02T10:15:30Z"
}

Rates are decimal strings with a fixed number of places, never floats. A quote expires 60 seconds after it's issued (our choice); a payment presenting an expired quote gets 409 quote_expired and a fresh quote.

Risk: every payment carries a risk block (score 0 to 99 and the decision taken). Merchants can see it but not change it.

Residency attributes: every merchant and every customer record has a home_region (for example eu-central-1) and a residency (for example M or NONE), set at onboarding and changed only by a supervised migration.

R3.4 Design Evolution: Money Across Borders, Rules and Adversaries

Step 3.1: Every Payment Writes the Platform Revenue Account

Primitive: Database Sharding and Partition Keys

Drill: One customer, one shard, one outage (the giant tenant's move is the "very biggest merchants" paragraph; cross-tenant reports read the lake, in "What it costs us")

Step 3.2: European Cards Need a Challenge Step

Synthesizing vector architecture diagram...

REQUIRES_ACTION is the only state where we wait for a person, and it has a deadline enforced by the database's own clock.

Step 3.3: Card-Testing Attacks

Primitive: Bot Defense, Sybil Resistance and Registration Abuse

Step 3.4: Twenty-Five Currencies

The €100.00 payment at our quoted rate of 1.0850 (the rate already includes our margin):

Each currency balances on its own: €100.00 = €100.00, and $108.50 = $108.50. (Fees are left out to keep the example small.) Later, treasury sells €100.00 at the bank for $109.00:

Both clearing accounts are back to zero, and the $0.50 is a realized gain, recorded.

Rounding, worked. €10.00 converted at 1.08235 is $10.8235 = 10,823,500 micros. The merchants are paid in cents, so the outward amount is $10.82 (half-to-even: the digits after the cent are 35, below half). The $0.0035 = 3,500 micros goes to fx_rounding_variance. If the $10.82 is split among three sellers, largest remainder gives $3.61, $3.61 and $3.60. The USD side: debit fx_clearing:USD 10,823,500 micros; credit the three sellers 10,820,000 micros in total and fx_rounding_variance 3,500 micros. It balances to the micro, which is why the ledger stores micros.

Step 3.5: Data Must Stay In-Country, and a Region Can Fail

Why not synchronous replication for every write? Every commit would wait for a round trip to another region (tens of milliseconds between nearby regions, more across oceans) on the ledger's hottest path, and a slow or unreachable second region would stall payments in the first. We keep strong consistency where it's needed, one writer per account, and accept asynchronous copies for recovery.

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active

Drill: The booking that existed in Frankfurt but not in Virginia (reading a lagging copy for a decision is point 3; synchronous replication for all writes is the paragraph above)

Step 3.6: Auditors Want Proof That Nothing Was Altered

Synthesizing vector architecture diagram...

The chain makes tampering detectable; Object Lock makes the evidence undeletable; the verifier belongs to someone who can't write the ledger.

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each home region is a complete Round 2 stack with its own ledger shards, vault, risk service and PSP connections. The only global piece is the small directory. Dotted arrows are disaster-recovery copies; market M's copy never leaves the country.

Each home region also runs its own Step Functions (disputes, payouts, cross-shard and cross-region transfers), MSK cluster, S3 lake and nightly reconciliation. A global reconciliation job reads every region's lake summaries (totals and break counts, not raw data from market M) to produce the company-wide books. The FX service quotes from a rate provider and caches rates per region; treasury converts balances daily.

What changed for manual captures. At ten times the volume, running a Step Functions saga for every manual-capture payment would cost about $245K a month (Round 2's $24.5K × 10), mostly to pay for waiting. We moved captures onto the same pattern the auto-capture path already uses: the payment row holds the state, the capture API runs the capture inline with its PSP key and the "unknown, then ask" protocol, and a sweeper selects WHERE status = 'REQUIRES_CAPTURE' AND capture_before < now() + interval '6 hours' to void authorizations nobody captured, using the database's clock. Step Functions stays for flows that are rarer and need a visible multi-step history.

Trace 1: a 3DS payment in the EU.

Synthesizing vector architecture diagram...

The webhook happened to arrive first here; if the return had come first, it would have triggered the same fetch and the same compare-and-set, and the webhook would have found nothing left to do.

Trace 2: a card-testing attack blocked. At 10:14 the risk service sees merchant m_4410's distinct-cards-per-minute counter jump from its usual 40 to 900, with 85% declines and $1 amounts. The rule "distinct cards per merchant per minute above 10× its 7-day baseline" fires: new cards at m_4410 now require a bot check on the hosted fields and a 3DS challenge, and the merchant's limit on new cards drops to 60 a minute (429 above it). Attempts that reach the PSP fall from about 1,000 a minute to under 60, most of those with a challenge the script can't pass. The merchant gets an alert; after an hour without the pattern, the limits relax step by step.

Trace 3: a regional failover. eu-central-1 has a severe outage at 09:00.

About 13 minutes from failure to traffic in the recovery region, most of it detection and decision.

What is waiting in the recovery region, and what isn't. Each shard's secondary already runs one writer-sized instance, so the promoted database can take the full write load at once. It has no read replicas yet: for the first minutes, status reads go to the writer (a db.r6g.8xlarge has room for them only because writes are what we sized it for; we shed status polling with 429 before it competes with payments), and we add read replicas after promotion, which takes several more minutes each. The API, risk and vault services run a warm fleet at a quarter of the home region's size (about 16 of eu-central-1's 63 API tasks) and scale out at failover. Fargate adds tasks in minutes, so for roughly the first 10 minutes in the recovery region, peak-hour traffic above a quarter of normal gets 503 and retries. We accept that, and we cost the warm fleets in R3.6.

R3.6 Numbers and Cost

Traffic per home region (the interviewer's split of 200M payments a day):

Shards. We keep Round 2's planning figure: one shard (a db.r6g.8xlarge writer) carries up to 2,500 payments a second at peak, the same figure we load-tested. So $\lceil 10{,}000 / 2{,}500 \rceil = 4$, $\lceil 7{,}000/2{,}500 \rceil = 3$, and $\lceil 3{,}500/2{,}500 \rceil = 2$ for each of the smaller regions: 11 shards. Reads per shard stay at or below Round 2's 12,500 (us-east-1: $50{,}000/4 = 12{,}500$), so each shard keeps three db.r6g.2xlarge readers.

Storage.

$$200 \times 10^6 \times 1 \text{ KB} = 200 \text{ GB/day} \;\Rightarrow\; \times 365.25 \approx 73 \text{ TB/year raw} \;\Rightarrow\; \times 1.6 \approx 117 \text{ TB/year with indexes}$$

That's about 10.6 TB per shard per year. We keep two years hot in Aurora (a choice) and detach older monthly partitions to S3, so each shard holds up to about 23 TB (the busiest shards take about 11.7 TB a year with indexes), well inside the Aurora limit.

Audit archive. The sealed copy is Parquet, which we assume compresses the ledger about 4×: 50 GB a day. Storage from a retention period is writes per day × days kept:

$$50 \text{ GB/day} \times 365.25 \times 7 \approx 128 \text{ TB at steady state}$$

The newest year (18.3 TB) in S3 Standard at $0.023/GB-month is about $420; the other six years (about 110 TB) in S3 Glacier Deep Archive at about $0.00099/GB-month is about $110. About $530 a month. Object Lock itself adds no storage charge.

Risk-scoring latency budget (checkout P99; dependent steps add, parallel steps take the max):

Inside the 50 ms: about 8 counter reads and increments against Valkey in one pipelined round trip (~2 ms), the rules (~1 ms) and the model (a few ms), with the rest as margin for garbage collection and queueing. Counter load: us-east-1 peaks at $10{,}000 \times 8 = 80{,}000$ Valkey operations a second, spread over 2 shards. We plan each region with 2 shards of a primary and a replica (4 nodes).

Monthly cost (us-east-1 on-demand prices for every region; the other three regions cost more, especially in Asia-Pacific, where both instance and data-transfer prices are higher):

Against the money. At the same list rate, fees on 200M payments are about $292M a day, roughly $8.9 billion a month. Infrastructure is about 0.003% of that. The costs that matter at this level are processing fees, fraud losses and chargebacks, and FX spread: a 0.1% improvement in authorization rate or fraud loss is worth more than the whole AWS bill. That's why routing, risk and FX are the architect's real levers.

R3.7 Trade-Offs

The opening question was: how do we make sure every payment happens exactly once in effect, and every cent is accounted for, when the network can't tell us what happened? The answer is now: at most once by construction (idempotency keys at our edge and at every PSP, one writer per account, compare-and-set state machines), every cent in a balanced journal (micros, per-currency balancing, explicit variance and FX accounts, hot accounts without shortcuts), and proven every day (three-way reconciliation, a hash chain sealed in write-once storage). The network still lies; the design never believes a silence.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per region and per PSP OPS 8 · REL 6

PSP failover procedure OPS 10

1. Confirm it's the PSP: its error and latency metrics are bad while the other PSPs are fine.
2. The breaker usually opens on its own. If the PSP is degraded but not failing enough to trip it, set its routing weight to zero with the parameter below.
3. Never move UNKNOWN payments to another PSP. Watch the resolver drain them as the PSP recovers.
4. Tell affected merchants whose customers only have PSP-specific tokens.
5. After recovery, bring the weight back in steps (10%, 50%, 100%) while watching its approval rate.

Reconciliation-break procedure

1. Check the inputs: did every expected file arrive, with the expected row count and a net total matching the bank deposit?
2. Group the breaks by category and cause. One cause behind many rows (a missed webhook batch, a new fee type) gets one fix.
3. For "missing in our ledger": fetch the object from the PSP by reference and replay the event through the normal webhook processor, which posts the journal with its usual idempotency.
4. For "amount mismatch": the PSP's number wins for money the PSP moved; post a correcting journal, reviewed by a second person.
5. Release the merchant's payout hold only when its breaks are zero.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the names and ARNs with real ones.

textCopy
# 1. Alarms currently firing for payments in a region
aws cloudwatch describe-alarms --region eu-central-1 --state-value ALARM --alarm-name-prefix payments-

# 2. Take PSP A out of routing (the router reads this parameter every 30 s)
aws ssm put-parameter --region eu-central-1 --name /payments/routing/psp-a/weight --value 0 --type String --overwrite

# 3. Failed dispute or payout sagas in the last hour
aws stepfunctions list-executions --region eu-central-1 --state-machine-arn arn:aws:states:eu-central-1:111122223333:stateMachine:payout-saga --status-filter FAILED --max-results 50

# 4. Redrive a failed Standard execution after fixing the cause (possible for 14 days after it ends)
aws stepfunctions redrive-execution --region eu-central-1 --execution-arn arn:aws:states:eu-central-1:111122223333:execution:payout-saga:po_20261002_m4410

# 5. Replication state of one ledger shard's global database
aws rds describe-global-clusters --global-cluster-identifier ledger-shard-05

# 6. Planned move of a shard's writer to another region (no data loss)
aws rds switchover-global-cluster --global-cluster-identifier ledger-shard-05 --target-db-cluster-identifier arn:aws:rds:eu-west-1:111122223333:cluster:ledger-shard-05-euw1

# 7. Unplanned failover when the primary region is down (may lose the replication lag)
aws rds failover-global-cluster --global-cluster-identifier ledger-shard-05 --target-db-cluster-identifier arn:aws:rds:eu-west-1:111122223333:cluster:ledger-shard-05-euw1 --allow-data-loss

# 8. Point shard 5's writer flag at the new region (services poll it every 10 s)
aws ssm put-parameter --region eu-west-1 --name /payments/shards/05/writer-region --value eu-west-1 --type String --overwrite

# 9. Rerun last night's reconciliation for one date
aws glue start-job-run --region eu-central-1 --job-name recon-nightly --arguments '{"--batch_date":"2026-10-01"}'

# 10. List yesterday's unexplained breaks
aws athena start-query-execution --region eu-central-1 --work-group finance --query-string "SELECT psp, category, COUNT(*) AS n, SUM(amount_micros) AS micros FROM recon.breaks WHERE batch_date = DATE '2026-10-01' GROUP BY psp, category"

The failover in command 7 can lose the last moments of replication. Command 8 flips the writer flag, but it can't reach the old region's copy while that region is down, so the fence doesn't depend on it: services in the old region stop writing shard 5 as soon as they can't refresh their flag lease (within 10 seconds), and services in the new region wait 15 seconds after the flip before writing. Aurora adds its own fence: the old primary can only rejoin as a read-only secondary. (When the old region is reachable, the runbook updates its copy of the parameter too.)

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Shards the ledger so that a payment's journal stays on one shard, and treats hot accounts with a strategy per account type, not one trick.
• Designs checkout as a state machine that can pause for a person, with deadlines enforced by the database's clock.
• Puts fraud scoring on the critical path with a latency budget, graduated actions and a feedback loop, and says what happens when it fails.
• Models FX in the ledger: original amounts, quotes, per-currency balancing, rounding variance and realized gain or loss.
• Chooses home regions with one writer per account, explains exactly why that can't double-spend and what a failover can lose, and bounds it.
• Makes the ledger tamper-evident with checks owned by someone else.
• Names the real economics: fees, fraud and FX, not servers, and can argue build versus buy.

Follow-up questions

1. "A merchant moves from Europe to the US. How do you move their ledger?" Answer: as a planned migration, not a replication trick. Freeze their writes briefly, copy their accounts' entries to a US shard, verify the balances and hash chain match, flip the directory and the routing table, then unfreeze. In-flight payments finish in the old home first. If residency rules forbid the move, their European data stays in Europe and only new activity starts in the US.

2. "Why not keep a global balance for each customer's wallet, readable anywhere?" Answer: we can offer a display balance anywhere, from replicas, clearly marked as possibly a moment old. A spendable balance can only be checked on the one writer for that wallet, in its home region. Two regions deciding about one balance is exactly how money gets double-spent.

3. "Could we use DynamoDB global tables for the ledger instead?" Answer: global tables replicate asynchronously by default and resolve conflicting writes with last-writer-wins, so two regions could both approve a spend and one record would be discarded. The multi-Region strong-consistency mode avoids that, but it runs in exactly three Regions and doesn't support transactions, and a journal is a multi-row transaction. It's a good fit for our small global directory, not for the ledger.

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design, I'll ask what we must never get wrong. For payments that's charging twice or losing a payment, so every piece I add will defend 'at most once', 'every cent accounted for', or 'our books match the bank's'."
• When the network fails: "A timeout isn't a failure, it's an unknown. We don't guess: we ask the PSP with the same idempotency key, and until it answers, the payment is UNKNOWN."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels