Design a Payment Processing System

1. Problem Statement & Scope Clarification

System Mission

Design a mission-critical, bank-grade global payment processing platform (equivalent to Stripe, Adyen, or Square) engineered to orchestrate millions of card authorizations, captures, and refunds daily. The system must guarantee absolute zero double-charges via distributed idempotency keys, enforce an immutable double-entry bookkeeping ledger, safely navigate third-party Payment Service Provider (PSP) gray failures via stateful saga orchestration, and reconcile billions of dollars nightly against external banking settlement files.

Payment Processing System at a Glance

Functional Requirements

1. Card Pay-In (Authorization & Capture): Ingest, tokenize, and execute credit card charges via multiple external PSPs (e.g., Stripe, Adyen, Chase Paymentech) with sub-second turnaround. Support two-phase authorize-and-capture (auth-capture) workflows as well as immediate single-phase charges.
2. Ironclad Distributed Idempotency: Guarantee that network timeouts, client retries, or duplicate button taps never result in multiple charges or duplicated ledger entries for the same transaction coordinate.
3. Immutable Double-Entry Ledger: Record every cent of money movement as balanced credit and debit rows adhering to the fundamental accounting equation ($\sum \text{Debits} - \sum \text{Credits} = 0$). Account balances must be mathematically derivable from the immutable transaction audit trail.
4. Smart Multi-PSP Routing & Failover: Dynamically route payments based on card BIN (Bank Identification Number), issuing country, transaction fees, and live PSP health/authorization rates to maximize payment conversion.
5. Asynchronous Settlement & Automated Nightly Reconciliation: Ingest multi-gigabyte daily bank and PSP settlement clearing files (NACHA, CAMT.053, CSV/BAI2), match them against internal ledger entries using automated big-data batch pipelines, and flag discrepancies for human review.

Non-Functional Requirements (SLAs & SLOs)

• Data Durability & Consistency: $100\%$ zero data loss guarantee. Strict ACID transactions for all internal balance and ledger state transitions; no eventual consistency on monetary balances.
• Availability: $99.999\%$ ("five nines") uptime for the payment ingestion and authorization API (less than 5.26 minutes of unscheduled downtime per calendar year).
• Latency (P99):
  • Synchronous Card Charge Path: $P99 < 800\text{ ms}$ (including external PSP round-trip latency of $300\text{--}600\text{ ms}$).
  • Internal Ledger & Balance Mutations: $P99 < 15\text{ ms}$.
  • Balance & Payment Status Inquiries: $P99 < 10\text{ ms}$.
• Security & Regulatory Compliance: PCI-DSS Level 1 compliant architecture. Zero raw Primary Account Numbers (PAN), CVVs, or expiration dates stored in unencrypted databases. Tokenization handled via dedicated isolated HSM-backed vaults using AWS KMS Customer Managed Keys (CMK).

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Transaction Volume & Throughput (QPS)

• Daily Completed Transactions: $20,000,000\text{ payments/day}$ ($20\text{ Million}$).
• Average Transaction QPS: $\text{TPS}_{\text{avg}} = \frac{20 \times 10^6\text{ transactions}}{86,400\text{ sec}} \approx \mathbf{231.5\text{ TPS}}$
• Peak Ingestion QPS ($10\times$ diurnal + flash-sale burst): $\text{TPS}_{\text{peak}} = 231.5 \times 10 \approx \mathbf{2,315\text{ TPS}} \quad (\text{dimensioned for } \mathbf{2,500\text{ TPS}})$
• Read & Status Inquiry QPS ($5:1$ Read/Write Ratio): $\text{QPS}_{\text{read, peak}} = 2,500 \times 5 = \mathbf{12,500\text{ QPS}}$

Storage Footprint & Capacity Growth (5-Year Horizon)

Every financial transaction generates three interconnected records:

1. Transaction Header: ID, buyer ID, merchant ID, status, currency, timestamps, metadata $\approx 350\text{ bytes}$.
2. Balanced Double-Entry Rows: Minimum 2 entries (Source Debit + Destination Credit) $\times 200\text{ bytes} = 400\text{ bytes}$.
3. Audit Trail & Settlement State: PSP reference tokens, state transition timestamps $\approx 250\text{ bytes}$.

• Total Storage per Payment: $\approx 1,000\text{ bytes } (1\text{ KB})$.

$$\text{Daily Raw Ledger Storage} = 20 \times 10^6\text{ payments} \times 1\text{ KB} = \mathbf{20\text{ GB/day}}$$

$\text{Annual Raw Ledger Storage} = 20\text{ GB/day} \times 365.25 \approx \mathbf{7.3\text{ TB/year}}$

$$\text{5-Year Raw Storage Footprint} = 7.3\text{ TB} \times 5 = \mathbf{36.5\text{ TB}}$$

• Accounting for B-tree index structures ($1.6\times$) and 3-AZ storage replication: $\text{Effective 5-Year Aurora Storage} = 36.5\text{ TB} \times 1.6 \times 3 \approx \mathbf{175.2\text{ TB}}$

Network Bandwidth Sizing

• Average Payment Request/Response Payload: $4\text{ KB}$ JSON envelope.
• Peak Ingress Bandwidth: $\text{Bandwidth}_{\text{in}} = 2,500\text{ TPS} \times 4\text{ KB} = 10\text{ MB/sec} = \mathbf{80\text{ Mbps}}$
• Peak Egress Bandwidth (Inquiries + PSP Calls): $\text{Bandwidth}_{\text{out}} = (12,500\text{ Read QPS} + 2,500\text{ PSP calls}) \times 4\text{ KB} = 60\text{ MB/sec} = \mathbf{480\text{ Mbps}}$

Ephemeral Idempotency Key Storage (DynamoDB)

• Key Record Size: Idempotency Key (UUID), Payment ID, Status, Cached Response Hash $\approx 500\text{ bytes}$.
• TTL Window: 7 days ($604,800\text{ seconds}$).
• Working Dataset in DynamoDB: $\text{Active Idempotency Working Set} = 20 \times 10^6\text{/day} \times 7\text{ days} \times 500\text{ bytes} \approx \mathbf{70\text{ GB RAM/SSD}}$

---

3. AWS-First High-Level Architecture

The architecture partitions responsibilities into four distinct domains: Ingress & Perimeter Tokenization, Saga Orchestration & PSP Execution, Core ACID Double-Entry Ledger, and Asynchronous Settlement & Reconciliation.

Synthesizing vector architecture diagram...

Follow a payment from the top. In the "Perimeter, Security & Tokenization" panel, WAF and mutual TLS guard the API, and the tokenizer replaces the card number with a token using HSM-backed keys, so raw card data stays inside this one small PCI-audited area. In the "Ingestion & Fast Idempotency Tier" panel, the payment API checks the idempotency key in DynamoDB, so a retried request returns the original result instead of charging twice, and queues the payment in SQS FIFO. In the "Distributed Payment Saga & PSP Execution" panel, Step Functions runs the payment and its smart router picks Stripe, falls back to Adyen, and then to Chase if a provider fails. The "Immutable Double-Entry Ledger Tier" records every movement as balanced debit and credit rows in Aurora, and the "Asynchronous Settlement & Reconciliation Data Lake" streams the ledger to S3 and matches it daily against bank files, paging on any mismatch.

Data Flow Walkthrough

1. Tokenization: The client streams raw card data directly to the isolated TokenizerSvc. Card numbers are encrypted using an envelope encryption key from AWS KMS/CloudHSM; only a synthetic token (tok_visa_9918) is returned to the client and forwarded to the payment backend.
2. Ingress & Idempotency Locking: The client submits POST /v1/payments with an Idempotency-Key header. The ECS Payment API initiates a conditional write to DynamoDB (attribute_not_exists(idempotency_key)). If the key already exists and is SUCCEEDED, the cached response is returned immediately. If IN_FLIGHT, HTTP 409 Conflict is returned.
3. Saga Orchestration: A fresh transaction begins by launching an AWS Step Functions Express/Standard Workflow. The state machine creates an internal record in Aurora PostgreSQL marked PENDING.
4. Smart Routing & Charge Execution: The state machine evaluates routing weights (BIN, latency, fees) and invokes the chosen external PSP API over mTLS.
5. Ledger Commit: Upon receiving an authorization/capture acknowledgement from the PSP, the saga opens an atomic transaction in Aurora PostgreSQL, inserts balanced debit and credit entries into ledger_entries, updates the transaction state to COMMITTED, and updates DynamoDB to SUCCEEDED.
6. Nightly Settlement Reconciliation: Daily clearing reports from banks and PSPs are ingested via AWS Transfer Family into S3. An AWS Glue Spark job performs a full outer join against internal ledger entries, ensuring zero financial drift.

Core Request Tracing Execution Walkthrough

---

4. API Interface Design & Data Contracts

1. Execute Charge (POST /v1/payments)

Executes an idempotent payment charge against a tokenized payment instrument.

Request Headers

httpCopy
POST /v1/payments HTTP/1.1
Host: api.payments.platform.aws.internal
Authorization: Bearer sec_live_99a8b7c6d5e4f3a2b1
Idempotency-Key: 9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d
X-Correlation-ID: corr_01J8N6K3P4V9QZ2W8M1Y7R4X
Content-Type: application/json

Request Payload

jsonCopy
{
  "amount_cents": 4999,
  "currency": "USD",
  "payment_method_token": "tok_visa_4242_pci_token",
  "customer_id": "cust_usr_88124",
  "order_id": "ord_ecommerce_990145",
  "merchant_account_id": "m_acct_nike_store_us",
  "capture_mode": "AUTOMATIC",
  "metadata": {
    "shipping_country": "US",
    "ip_address": "198.51.100.42"
  }
}

Response: 201 Created

jsonCopy
{
  "payment_id": "pay_live_718293847501",
  "status": "SUCCEEDED",
  "amount_cents": 4999,
  "currency": "USD",
  "psp_reference": "ch_stripe_3N8vK2Lkd0918",
  "routing_selected_psp": "STRIPE",
  "idempotency_key": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "captured": true,
  "fee_cents": 175,
  "net_merchant_cents": 4824,
  "created_at": "2026-09-16T14:32:01.458Z"
}

2. Capture Authorized Hold (POST /v1/payments/{payment_id}/capture)

Captures previously authorized funds (e.g., after physical order shipment).

httpCopy
POST /v1/payments/pay_live_718293847501/capture HTTP/1.1
Host: api.payments.platform.aws.internal
Idempotency-Key: idemp_cap_882194
Content-Type: application/json

{
  "amount_cents": 4999,
  "currency": "USD"
}

3. Issue Full or Partial Refund (POST /v1/payments/{payment_id}/refunds)

httpCopy
POST /v1/payments/pay_live_718293847501/refunds HTTP/1.1
Host: api.payments.platform.aws.internal
Idempotency-Key: idemp_ref_330198
Content-Type: application/json

{
  "amount_cents": 2000,
  "reason": "CUSTOMER_RETURN"
}

Status Codes & Error Contracts

• 200 OK: Idempotent query or repeat of previously successful idempotent request.
• 201 Created: Payment newly executed and ledger successfully committed.
• 400 Bad Request: Validation failure (malformed payload, invalid currency).
• 402 Payment Required: PSP declined (insufficient funds, expired card, fraud rule rejection).
• 409 Conflict: Idempotency key currently IN_FLIGHT on another worker; client must retry after 500ms.
• 422 Unprocessable Entity: Currency mismatch or amount exceeds authorized ceiling.
• 429 Too Many Requests: Client or tenant rate limit exceeded.
• 504 Gateway Timeout: PSP call timed out; transaction placed in indeterminate gray-failure recovery.

jsonCopy
{
  "error": {
    "code": "CARD_DECLINED",
    "message": "Your card was declined due to insufficient funds.",
    "decline_code": "insufficient_funds",
    "payment_id": "pay_live_718293847501",
    "psp_code": "do_not_honor"
  }
}

---

5. Data Models & Storage Architecture

Database Selection Justification

1. Amazon Aurora PostgreSQL Multi-AZ: Selected for the core ledger. Money movement requires serializable row integrity, multi-record ACID transactions, check constraints, and relational queries for double-entry balancing.
2. Amazon DynamoDB: Selected for the low-latency distributed idempotency key cache. Supports sub-5ms conditional writes (PutItem with attribute_not_exists) and automatic 7-day TTL cleanup with zero operational burden.
3. Amazon S3 + AWS Glue + Amazon Athena: Selected for the financial data lake and nightly reconciliation engine. Handles petabyte-scale CSV, Parquet, and bank clearing statements at minimum storage cost.

1. Double-Entry Bookkeeping Ledger DDL (Aurora PostgreSQL)

In financial systems, balances are never mutated via raw UPDATE accounts SET balance = balance + 10 statements. Balances are mathematically derived from an immutable audit trail of balanced debit and credit entries. To eliminate fractional-cent rounding drift across international currencies and fee splits, monetary amounts are modeled as 64-bit Signed Integers in Micro-Units ($10^{-6}$ base currency units, where $\text{\textdollar}1.00\text{ USD} = 1,000,000\text{ micro-units}$).

sqlCopy
-- 1. Supported Currency Metadata Table
CREATE TABLE currencies (
    currency_code CHAR(3) PRIMARY KEY, -- 'USD', 'EUR', 'JPY', 'GBP'
    exponent INT NOT NULL DEFAULT 2, -- Minor unit exponent (USD=2, JPY=0, BHD=3)
    micro_multiplier BIGINT NOT NULL DEFAULT 1000000 -- Conversion to micro-units
);

-- 2. Accounts Master Table
CREATE TABLE accounts (
    account_id VARCHAR(64) PRIMARY KEY, -- e.g. "USER_WALLET:usr_102", "MERCHANT_PAYABLE:m_402", "PSP_RECEIVABLE:stripe"
    account_type VARCHAR(32) NOT NULL, -- ASSET, LIABILITY, EQUITY, REVENUE, EXPENSE
    currency CHAR(3) NOT NULL REFERENCES currencies(currency_code),
    status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
    -- Read-optimized SNAPSHOT of SUM(ledger credits) - SUM(ledger debits). It is never written by
    -- application code directly: it is updated in the same transaction that inserts the ledger rows
    -- (Section 9.2) and is re-derived from ledger_entries by the nightly audit job.
    balance_micros BIGINT NOT NULL DEFAULT 0 CHECK (balance_micros >= 0),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- 3. Master Payment Transactions Table
CREATE TABLE payment_transactions (
    transaction_id VARCHAR(64) PRIMARY KEY, -- e.g. "pay_live_718293847501"
    idempotency_key VARCHAR(128) UNIQUE NOT NULL,
    merchant_account_id VARCHAR(64) NOT NULL REFERENCES accounts(account_id),
    customer_id VARCHAR(64) NOT NULL,
    amount_micros BIGINT NOT NULL CHECK (amount_micros > 0),
    fee_micros BIGINT NOT NULL DEFAULT 0 CHECK (fee_micros >= 0),
    currency CHAR(3) NOT NULL REFERENCES currencies(currency_code),
    status VARCHAR(20) NOT NULL CHECK (status IN ('PENDING', 'COMMITTED', 'REJECTED', 'REFUNDED', 'INDETERMINATE')),
    psp_provider VARCHAR(32) NOT NULL, -- STRIPE, ADYEN, CHASE
    psp_reference VARCHAR(128),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- 4. Immutable Double-Entry Ledger Lines
CREATE TABLE ledger_entries (
    entry_id BIGSERIAL PRIMARY KEY,
    transaction_id VARCHAR(64) NOT NULL REFERENCES payment_transactions(transaction_id),
    account_id VARCHAR(64) NOT NULL REFERENCES accounts(account_id),
    direction VARCHAR(6) NOT NULL CHECK (direction IN ('DEBIT', 'CREDIT')),
    amount_micros BIGINT NOT NULL CHECK (amount_micros > 0),
    currency CHAR(3) NOT NULL REFERENCES currencies(currency_code),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Indexing for Fast Settlement & Auditing Queries
CREATE INDEX idx_ledger_account_lookup ON ledger_entries(account_id, created_at DESC);
CREATE INDEX idx_ledger_tx_id ON ledger_entries(transaction_id);
CREATE INDEX idx_payment_created ON payment_transactions(created_at DESC);

The Fundamental Accounting Invariant & Fractional-Cent Drift Protocol

Every transaction must mathematically balance across micro-units:

$$\sum \text{Debits} - \sum \text{Credits} = 0$$

Fractional-Cent Rounding Drift in Multi-Currency & Fee Allocations

When converting currencies or splitting percentage fees across basket items (e.g., splitting a $10.00 charge across 3 merchant payouts yielding $3.333333... each), integer division causes fractional cents to drift. If unhandled, $\sum \text{Debits} \ne \sum \text{Credits}$, violating the double-entry invariant.

To eliminate drift:

1. Banker's Rounding (Half-Even): Standardized to IEEE 754-2008 half-even rounding to prevent statistical accumulation toward zero or infinity.
2. Micro-Unit Precision: Calculations execute in $10^{-6}$ micro-units prior to display truncation.
3. Synthetic Rounding Variance Line: If an irreducible fractional remainder exists ($\Delta \neq 0$), the transaction engine generates a compensatory balancing ledger line attributed to a dedicated system equity account (EQUITY:FX_ROUNDING_VARIANCE): $\Delta_{\text{drift}} = \sum \text{Debits} - \sum \text{Credits}$ The engine appends (account: "EQUITY:FX_ROUNDING_VARIANCE", direction: (Δ > 0 ? "CREDIT" : "DEBIT"), amount: |Δ|). This preserves strict mathematical equality down to the single micro-unit without manual journal interventions.

2. DynamoDB Single-Table Idempotency Schema (PaymentIdempotencyTable)

3. PCI-DSS Tokenization Vault Schema & KMS Envelope Key Rotation

To comply with PCI-DSS Level 1 while permitting seamless annual and emergency key rotation, PAN data is encrypted using envelope encryption. Raw cards are encrypted under unique Data Encryption Keys (DEKs) wrapped by an AWS KMS Customer Managed Key (CMK).

sqlCopy
CREATE TABLE card_token_vault (
    token_id VARCHAR(64) PRIMARY KEY, -- e.g. "tok_live_991823a"
    customer_id VARCHAR(64) NOT NULL,
    kms_key_arn VARCHAR(256) NOT NULL, -- ARN of root CMK used
    key_version VARCHAR(16) NOT NULL, -- e.g. "v1", "v2"
    encrypted_dek BYTEA NOT NULL, -- AES-256 DEK wrapped by KMS CMK
    ciphertext_pan BYTEA NOT NULL, -- Primary Account Number encrypted with DEK
    pan_last4 CHAR(4) NOT NULL, -- Plaintext masked last 4 digits for UI display
    card_brand VARCHAR(20) NOT NULL, -- VISA, MASTERCARD, AMEX
    expiration_month INT NOT NULL,
    expiration_year INT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    reencrypted_at TIMESTAMPTZ -- Populated by background key rotation daemon
);

CREATE INDEX idx_vault_key_version ON card_token_vault(key_version) WHERE reencrypted_at IS NULL;

Tokenization Key Rotation Workflow

1. Active Write Key Rotation: AWS KMS CMK rotates automatically or administrators create a new key version (v2). New tokens immediately use v2.
2. Dual-Key Transparent Decryption: Decryption inspects kms_key_arn and key_version. KMS decrypts the token's encrypted_dek using the corresponding key version, enabling zero-downtime transitions.
3. Asynchronous Batch Re-Encryption: An AWS ECS background daemon runs off-peak, querying rows where key_version != current_version. It decrypts using the legacy key, re-wraps with the new CMK under v2, updates encrypted_dek, and commits without modifying token_id or disrupting client operations.

---