Design Shopify's Flash Sale Architecture & Payment Resilience

1. Problem Statement & Scope Clarification

System Mission

Design Shopify's high-concurrency commerce engine and payment resilience platform capable of handling extreme Black Friday Cyber Monday (BFCM) flash sales (> $9.3B+ GMV across a single weekend). The architecture must survive catastrophic viral traffic surges ($100,000+$ checkout requests/sec on a single hot merchant brand like Gymshark or Kylie Cosmetics), protect core transactional databases via an edge Virtual Waiting Room, isolate merchant blast radiuses through autonomous Cellular Pods, and prevent inventory overselling via atomic two-phase Redis Lua reservation leases before committing payments to MySQL.

Synthesizing vector architecture diagram...

Functional Requirements

1. Edge Virtual Waiting Room: Intercept traffic spikes exceeding merchant checkout capacity at the edge; queue shoppers fairly and admit them with cryptographically signed, short-lived HMAC tokens.
2. Atomic Inventory Reservation: Guarantee zero inventory overselling on limited-stock items (e.g., $5,000$ sneakers sought by $500,000$ buyers) using atomic Redis Lua scripts with $10\text{-minute}$ lease expiration tokens.
3. Cellular Pod Multi-Tenant Isolation: Bound blast radiuses such that noisy-neighbor traffic spikes in one merchant store never degrade neighboring merchants.
4. Resilient Idempotent Payment Processing: Provide exactly-once payment authorization and capture across external payment gateways (Stripe, Adyen, PayPal, Shop Pay) despite transient network drops or webhooks arriving out of order.
5. Compensating Lease Release: Automatically return reserved inventory to the available pool if the shopper abandons checkout or payment authorization fails.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.99\%$ uptime during BFCM peak windows ($72\text{ h} \times 3600 \times 10^{-4} \approx 26$ seconds allowed downtime over the 72-hour weekend).
• Checkout Latency:
  • Waiting Room Pass-Through: $P99 < 15\text{ ms}$ at the edge.
  • Inventory Lease Check: $P50 < 2\text{ ms}$, $P99 < 8\text{ ms}$ in Redis.
  • End-to-End Order Creation: $P95 < 600\text{ ms}$, $P99 < 1,500\text{ ms}$ in MySQL Aurora.
• Inventory Overselling Rate: Exactly $0.00\%$ (strict linearizability for stock decrement).
• Peak Throughput: Sustain $120,000\text{ orders/minute}$ ($2,000\text{ orders/sec}$) platform-wide, and absorb $100,000\text{ QPS}$ flash sale bursts against a single merchant store.
• CAP / PACELC Classification:
  • Inventory Reservation & Checkout: CP system under CAP; PC/EC under PACELC (Partition $\to$ Consistency; Else $\to$ Consistency over Latency). Financial balances and limited physical inventory require strict serializability.

Out-of-Scope

• Physical warehouse robotics and supply chain freight fulfillment.
• In-store Point-of-Sale (POS) offline hardware Bluetooth card-reader processing.
• Dynamic localized tax rate table derivation (delegated to third-party services like Avalara).

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Traffic Scale Calculations (BFCM Peak)

• Total Merchants: $2,000,000$ active stores.
• Platform Peak Ingress Throughput: $30,000,000\text{ HTTP req/min} = \mathbf{500,000\text{ Ingress QPS}}$.
• Peak Completed Orders: $120,000\text{ orders/min} = \mathbf{2,000\text{ completed orders/sec}}$ platform-wide.
• Viral Flash Sale Peak on a Single Merchant: $\text{Flash Sale Checkout Ingress} = \mathbf{100,000\text{ requests/sec}}$ $\text{Available Stock} = 10,000\text{ limited edition items (Sold out in } < 60\text{ seconds)}$ $\text{Oversubscription Factor} = \frac{100,000\text{ buyers}}{10,000\text{ items}} = 10\times\text{ demand ratio}$

Database & Storage Calculations (3-Year Horizon)

• Order Record Size:
  • orders master record: 800 bytes
  • order_line_items (avg 3 items): 600 bytes
  • payment_transactions & auth tokens: 400 bytes
  • Total per order: $\approx \mathbf{1.8\text{ KB}}$.
• Annual Orders Stored: $1.5\text{ Billion orders/year} \times 1.8\text{ KB} = \mathbf{2.7\text{ TB raw/year}}$.
• 3-Year Replicated Pod Volume Across 200 Pods: $\text{3-Year Raw Storage} = 2.7\text{ TB} \times 3 = 8.1\text{ TB}$ $\text{Replicated Storage (Multi-AZ Aurora, RF = 6)} = 8.1\text{ TB} \times 6 = \mathbf{48.6\text{ TB Global SSD}}$ Aurora bills the logical volume; the 6 copies are its internal storage layer. Average logical storage per Pod: $\frac{8.1\text{ TB}}{200\text{ Pods}} \approx \mathbf{40\text{ GB per Pod}}$, which fits entirely in a db.r6g.4xlarge (128 GB) buffer pool.

Cache & Memory Sizing (Redis Inventory Tier)

• Active SKUs per Hot Merchant: $50,000$ active items.
• Redis Memory per SKU Entry: Key: inventory:{store_id}:{variant_id} $\to$ hash map with available count, reserved count, and lease zset ($\approx 256\text{ bytes}$). $50,000 \times 256\text{ bytes} \approx \mathbf{12.8\text{ MB RAM}}$ per hot merchant.
• Active 10-Minute Lease Tokens: $10,000$ active leases $\times 128\text{ bytes} \approx \mathbf{1.28\text{ MB RAM}}$.
• Pod Redis Provisioning: Each Pod provisions an AWS ElastiCache Redis cache.r6g.2xlarge cluster (52 GB RAM, Multi-AZ) with $< 5\%$ memory utilization, dedicating 95% of capacity to I/O bandwidth and single-threaded Lua script execution.

Fleet Sizing & Pod Allocation

• Number of Cellular Pods: $200$ autonomous Pods.
• Compute Tasks per Pod: Each Pod runs $20 - 100$ ECS Fargate tasks ($2\text{ vCPU}, 4\text{ GB RAM}$) auto-scaling based on CPU and request queue depth. Total platform compute: $200\text{ Pods} \times 50\text{ tasks (avg)} = \mathbf{10,000\text{ compute containers}}$.

---

3. AWS-First High-Level Architecture

Synthesizing vector architecture diagram...

Follow a shopper from the top during a flash sale. In the "Edge Traffic Shaping & Virtual Waiting Room" panel, a Lambda@Edge function holds shoppers in a waiting room and admits them at a rate the store can handle, giving each an HMAC-signed admission token. In the "Global Ingress Routing Tier" panel, the pod router looks up which pod hosts this store in the tenant directory and forwards the request. In the "Isolated Merchant Pod 14" panel, the checkout fleet reserves inventory with an atomic Lua script in Redis, writes the order to the pod's own Aurora, and pushes receipts and webhooks to SQS for background workers. In the "External Payment Gateway Orchestration" panel, payments pass through a PCI-scoped vault proxy to Stripe, Adyen or Shop Pay. The waiting room caps the arrival rate, and the pod contains the load: together they keep one merchant's sale from affecting other merchants.

Data Flow Walkthrough

1. Edge Traffic Shaping & Queueing:
   • Millions of shoppers arrive at the merchant store during a flash sale.
   • Amazon CloudFront and AWS WAF inspect the incoming request. If traffic to gymshark.com/checkout exceeds the configured threshold ($2,000\text{ req/sec}$), Lambda@Edge intercepts the request.
   • Unadmitted shoppers receive a lightweight Virtual Waiting Room page showing their dynamic queue position.
   • When the waiting room admits a batch, Lambda@Edge generates a cryptographically signed HMAC token (X-Shopify-Queue-Token) with an expiration of $5\text{ minutes}$ (the inventory lease minted later lasts 10 minutes; the two TTLs are independent) and redirects the client to checkout.
2. Pod Routing:
   • The request hits the Pod Router Service. The router inspects the Host header (gymshark.com) or merchant ID, queries its in-memory tenant directory, and routes the traffic directly to Pod 14.
3. Atomic Inventory Lease Reservation (Redis Lua):
   • Before allowing the shopper to enter payment details, the checkout service invokes an atomic Lua script on Pod 14’s ElastiCache Redis cluster.
   • The script verifies that available_inventory >= requested_quantity. If true, it decrements available_inventory, increments reserved_inventory, and stores a lease token with a $600\text{-second}$ expiration.
   • If stock is exhausted, the Lua script returns SOLD_OUT in $< 2\text{ ms}$, immediately rendering an out-of-stock modal without touching the MySQL database.
4. Idempotent Payment Authorization:
   • The shopper inputs payment credentials. The checkout service submits the transaction through the isolated PCI-DSS payment proxy to the external gateway (Stripe/Adyen) using a deterministic Idempotency-Key = hash(checkout_id, payment_attempt).
5. Order Commit to MySQL Aurora:
   • Upon confirmed payment authorization, the checkout service executes an ACID transaction in Pod 14’s Aurora MySQL: inserts the orders and order_line_items records, confirms payment capture, and flushes the Redis lease into permanently committed state.
   • An asynchronous job enqueues into Amazon SQS for email receipts and shipping notifications.

---

End-to-End Request Tracing Walkthrough

---

4. API Interface Design & Wire Protocol

1. Checkout Initiation with Edge Waiting Room Validation (POST /v1/checkouts)

httpCopy
POST /v1/checkouts HTTP/1.1
Host: checkout.shopify.com
Content-Type: application/json
X-Shopify-Store-Id: store_gymshark_9482
X-Shopify-Queue-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdG9yZSI6Ijk0ODIiLCJ1c2VyIjoidXNyXzAxSFo...
Idempotency-Key: chk_init_01HZX894KMNPQ9283401

{
  "cart_token": "cart_8928340192834019",
  "line_items": [
    {
      "variant_id": 9814028192,
      "quantity": 1,
      "unit_price_cents": 6500
    }
  ],
  "shipping_address": {
    "postal_code": "90210",
    "country_code": "US"
  }
}

Response: 201 Created

jsonCopy
{
  "checkout_id": "chk_8492019481029",
  "store_id": "store_gymshark_9482",
  "status": "INVENTORY_RESERVED",
  "inventory_lease": {
    "lease_token": "lease_01HZX894KMNPQ",
    "expires_at_epoch_ms": 1773648600000,
    "lease_duration_seconds": 600
  },
  "total_amount_cents": 7150,
  "payment_vault_url": "https://vault.shopify.com/sessions/sess_91823a"
}

---

2. Payment Completion & Capture (POST /v1/checkouts/{id}/complete_payment)

httpCopy
POST /v1/checkouts/chk_8492019481029/complete_payment HTTP/1.1
Host: checkout.shopify.com
Content-Type: application/json
X-Shopify-Store-Id: store_gymshark_9482
Idempotency-Key: chk_pay_01HZX894KMNPQ9283401

{
  "payment_token": "tok_1N4m2b2eZvKYlo2C",
  "payment_gateway": "STRIPE",
  "lease_token": "lease_01HZX894KMNPQ",
  "expected_total_cents": 7150
}

Response: 200 OK

jsonCopy
{
  "order_id": "ord_9840192840192",
  "checkout_id": "chk_8492019481029",
  "status": "PAID",
  "financial_status": "CAPTURED",
  "confirmation_number": "SHPF-984019",
  "created_at": "2026-09-16T21:40:00Z"
}

---

3. Status Codes & Error Contracts

---

5. Data Models & Storage Architecture

Relational Schema: Aurora MySQL Pod Storage

Each merchant Pod hosts an independent Aurora MySQL cluster containing the merchant's relational transactional records.

Synthesizing vector architecture diagram...

Read the relationships from STORE: a store receives ORDERs, and each order has line items and payment transactions, while INVENTORY_LEVEL tracks stock per product variant and store. Two unique keys protect against retries: checkout_id on ORDER means a double-submitted checkout creates one order, and idempotency_key on PAYMENT_TRANSACTION means a retried charge is recorded once, even if the network failed after the gateway charged the card. INVENTORY_LEVEL keeps available and committed quantities separately, so stock reserved by in-progress checkouts is not sold twice. Every table sits in the pod's own database (see pod_id on STORE), so a merchant's data never needs a cross-pod join.

Production MySQL DDL for Orders & Inventory

sqlCopy
CREATE TABLE orders (
    order_id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    store_id BIGINT UNSIGNED NOT NULL,
    checkout_id VARCHAR(64) NOT NULL,
    customer_id BIGINT UNSIGNED,
    email VARCHAR(255) NOT NULL,
    total_amount_cents BIGINT UNSIGNED NOT NULL,
    financial_status ENUM('PENDING', 'AUTHORIZED', 'CAPTURED', 'REFUNDED') NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (order_id),
    UNIQUE KEY uq_checkout (store_id, checkout_id),
    KEY idx_store_customer (store_id, customer_id),
    KEY idx_created (store_id, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

CREATE TABLE payment_transactions (
    transaction_id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    order_id BIGINT UNSIGNED NOT NULL,
    store_id BIGINT UNSIGNED NOT NULL,
    idempotency_key VARCHAR(128) NOT NULL,
    gateway VARCHAR(32) NOT NULL,
    gateway_reference VARCHAR(128),
    amount_cents BIGINT UNSIGNED NOT NULL,
    status ENUM('INITIATED', 'SUCCESS', 'FAILED') NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (transaction_id),
    UNIQUE KEY uq_idempotency (store_id, idempotency_key),
    FOREIGN KEY (order_id) REFERENCES orders (order_id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

-- Authoritative stock: Redis leases are a fast path, this row is the source of truth (see §6.2)
CREATE TABLE inventory_levels (
    store_id BIGINT UNSIGNED NOT NULL,
    variant_id BIGINT UNSIGNED NOT NULL,
    available_quantity INT NOT NULL,
    committed_quantity INT NOT NULL DEFAULT 0,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (store_id, variant_id),
    CONSTRAINT chk_available_positive CHECK (available_quantity >= 0)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

---

Redis Lua Script: Atomic Inventory Reservation Lease

To guarantee zero overselling under 100,000 QPS, inventory reservation is executed entirely inside an atomic Redis Lua script.

luaCopy
-- KEYS[1]: inventory_key  = "inventory:{store_id}:{variant_id}"   (hash: available, reserved)
-- KEYS[2]: lease_zset_key = "leases:{store_id}:{variant_id}"      (zset: token -> expiry epoch)
-- KEYS[3]: lease_qty_key  = "leaseqty:{store_id}:{variant_id}"    (hash: token -> qty, so the reaper knows how much to return)
-- ARGV[1]: requested_quantity (e.g., 1)
-- ARGV[2]: lease_token (e.g., "lease_01HZX894KMNPQ")
-- ARGV[3]: lease_expiration_epoch (e.g., current_epoch + 600)
-- All keys share the {store_id}:{variant_id} hash tag so they land on one cluster slot.

local available = tonumber(redis.call('HGET', KEYS[1], 'available'))
local req_qty = tonumber(ARGV[1])

if available == nil or available < req_qty then
    return 0 -- SOLD OUT / INSUFFICIENT STOCK
end

-- Atomic Decrement of Available Stock & Increment of Reserved Stock
redis.call('HINCRBY', KEYS[1], 'available', -req_qty)
redis.call('HINCRBY', KEYS[1], 'reserved', req_qty)

-- Record Lease in Sorted Set with Expiration Timestamp as Score
redis.call('ZADD', KEYS[2], ARGV[3], ARGV[2])
redis.call('HSET', KEYS[3], ARGV[2], req_qty)

return 1 -- SUCCESS: INVENTORY RESERVED

---