Design a Mobile Stock Trading App

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

In this loop the phone is part of the system. It has a flaky network, a screen that redraws 120 times a second, a secure chip that can sign things, and a user whose thumb is on the Buy button. Every round designs both halves: the app on the device and the backend it talks to. We are the broker's app, not the exchange. How an exchange matches orders is the subject of the matching engine loop, and how money moves through a ledger is the subject of the digital wallet and payment processing loops. We link to them instead of teaching them again.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Trading App?

You Already Know One: a Live Price Board With a Buy Button

Picture the departures board at a train station, except the numbers are prices and they change many times a second. Under the board there's a button. You watch a price, decide it's right, and press the button. Someone at the other end must do exactly what you pressed, once, and tell you honestly what happened.

A mobile trading app is that board and that button in your pocket. The board is fed by the stock market; the button sends an order to a broker, who sends it on to a market to be executed.

A few words carry this whole loop. Each gets one line now:

Synthesizing vector architecture diagram...

Two flows cross in our backend. Market data flows down and is huge; orders flow up and are few, but each one is money.

What Makes It Hard

• Prices move faster than a phone can show them. A busy stock can change a hundred times a second. A person can't read that, a cellular radio shouldn't carry it, and a phone shouldn't redraw on every change.
• The network drops just as you tap Buy. The app often can't know whether the order arrived.
• A duplicated order is real money. Two orders instead of one means twice the shares, twice the cost, and an angry customer.
• A stale price is a trap. A user who taps Buy while looking at a price from 30 seconds ago is trading blind.
• Everyone arrives at 9:30 a.m. Eastern. The market opens, millions of people open the app in the same minute, and the busiest seconds of the day come first.

The Question the Whole Loop Answers

How do we show prices that are fresh enough to trust, and make every order happen exactly once, only if the user meant it?

The answer gets sharper every round:

• Round 1: stream quotes instead of polling, give every order a key born on the phone so retries can't duplicate it, and let the server, never the phone, decide buying power and order state.
• Round 2: shrink the market-data firehose to what a phone can use (conflation), keep the screen smooth, refuse to trade on stale prices, and sign every order with a key only the user's biometric can unlock.
• Round 3: run it as a regulated financial service: records that can be proven years later, routing that can be defended, a second region that loses no accepted order, and releases that can't break trading.

---

Round 1 · Mid-level · "Quotes and Orders for a Small Brokerage"

~35 min · SDE II (L5) · 1 region, 3 AZs · 50K users online at the open · 500K quote deliveries/s at peak · 100K orders/day, ~100/s at peak · 99.9% in market hours

R1.1 Establish Design Scope

The interviewer says: "We're a small brokerage. Our customers want a phone app with a watchlist, a quote screen, and a way to buy and sell. Design it." We ask before we draw.

Out of scope for this round: options, margin accounts, smooth 120 Hz charts, biometric order signing, and the scale of a meme-stock morning.

R1.2 Functional Requirements, Derived Step by Step

Not yet: 120 Hz rendering, biometric signing, the opening-bell scale. The interviewer will bring them back.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Freshness. How old may a price on screen be before it's misleading? And how does the app know its prices are old, instead of silently showing the last one?
• No duplicate orders. A tap must produce at most one order, whatever the network does. And the user must never have to guess whether an order exists.
• Clear status. "Accepted by us", "acknowledged by the market" and "filled" are three different facts. The app must never show one as another.
• Availability during market hours. The regular US session runs 9:30 a.m. to 4:00 p.m. Eastern, 6.5 hours a day, about 252 days a year: $6.5 \times 252 = 1{,}638$ hours. So 99.9% in market hours means at most $1{,}638 \times 60 \times 0.001 \approx 98$ minutes a year, about 8 minutes a month, and it's measured when customers need us, not at 3 a.m. on a Sunday.

R1.4 The API

Three pieces: an order API, one streaming connection for quotes and order updates, and a small history API for the chart. Every call carries the user's access token.

1. Place an order.

httpCopy
POST /v1/orders HTTP/1.1
Host: api.example-broker.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "client_order_id": "0192f3c4-5b1e-7a42-9c3d-2f8e1a6b7c90",
  "account_id": "acc_5521",
  "symbol": "AAPL",
  "side": "BUY",
  "type": "LIMIT",
  "quantity": 10,
  "limit_price": "227.50",
  "time_in_force": "DAY"
}

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{
  "order_id": "ord_8KZ2M4Q1",
  "client_order_id": "0192f3c4-5b1e-7a42-9c3d-2f8e1a6b7c90",
  "status": "RECEIVED",
  "reserved_cash": "2275.00",
  "account_event_seq": 1042
}

• client_order_id is the idempotency key (a unique ID the phone creates once per order intent, so a repeat of the same request is recognised). It's a UUIDv7, whose first bits are a timestamp, so IDs sort roughly by creation. It lives in the body, not a header, because in Round 2 it must sit inside the bytes the user signs.
• Prices are decimal strings, never floating point, so 227.50 stays 227.50.
• 201 means "we recorded it and reserved the money". Not "it traded", not even "the market has it". Those come later as status updates.
• The same request again returns 200 OK with this same body.

2. Look up and cancel.

httpCopy
GET /v1/orders/ord_8KZ2M4Q1 HTTP/1.1
Authorization: Bearer <token>

httpCopy
DELETE /v1/orders/ord_8KZ2M4Q1 HTTP/1.1
Authorization: Bearer <token>

A cancel returns 202 Accepted with "status": "PENDING_CANCEL": cancelling is a request to the market, and a fill can still win the race (step 1.3).

3. Resolve an order in doubt. When the app never got an answer to POST /v1/orders, it asks by its own key:

httpCopy
POST /v1/orders:resolve HTTP/1.1
Authorization: Bearer <token>
Content-Type: application/json

{ "account_id": "acc_5521", "client_order_id": "0192f3c4-5b1e-7a42-9c3d-2f8e1a6b7c90" }

It returns the order if it exists, or "status": "VOIDED", which means "it doesn't exist, and now it never will". Step 1.2 explains why this call writes.

4. The stream. One WebSocket per app session (a WebSocket is a long-lived, two-way connection that starts as an HTTP request and then carries messages in both directions). The app says what it wants:

jsonCopy
{ "op": "subscribe", "symbols": ["AAPL", "MSFT", "NVDA"] }

The server answers with a snapshot of each symbol's current quote, then updates, then heartbeats (small "I'm alive" frames) when nothing else is sent:

jsonCopy
{ "type": "quote", "symbol": "AAPL", "bid": "227.48", "ask": "227.52", "bid_size": 300, "ask_size": 200,
  "last": "227.50", "volume": 18422310, "server_ts": "2026-09-28T13:30:01.004Z" }

jsonCopy
{ "type": "order", "account_event_seq": 1043, "order_id": "ord_8KZ2M4Q1", "status": "NEW" }

jsonCopy
{ "type": "heartbeat", "server_ts": "2026-09-28T13:30:02.004Z" }

5. Today's chart. GET /v1/charts/AAPL?interval=1m&day=today returns one-minute bars (open, high, low, close, volume).

Recap

• One order API whose key is born on the phone; one resolve call for orders in doubt.
• One WebSocket for quotes and order updates, with heartbeats.
• 201 means recorded and reserved, nothing more.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From Polling and Posting to Streams and Keys

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

The app polls GET /v1/quotes?symbols=... every second. The Buy button posts an order; the order service checks the balance, calls the partner broker, and returns what the broker said.

Synthesizing vector architecture diagram...

Everything is request and response. The app asks for prices; the order waits on the broker.

What's good about it: simple, and easy to reason about.

What it costs us: 50K phones × 1 poll a second is 50K requests a second, most returning the same prices; the prices are up to a second old before they even leave; and a retried order post is a second order.

Step 1.1: Polling Is Slow and Wasteful

Primitive: WebSocket, SSE & Long Polling

Step 1.2: The Network Dropped After I Tapped Buy, So I Tapped Again

Primitive: Distributed Unique ID Generators

Step 1.3: "Is My Order Done?"

Synthesizing vector architecture diagram...

Only RECEIVED, ROUTED, PENDING_CANCEL and a risk REJECTED are ours to set; every other state is a fact reported by the broker. A cancel can fail (the broker rejects it), which returns the order to where it was. Four states end the order.

Primitive: Change Data Capture & the Outbox Pattern

Step 1.4: The User Doesn't Have the Money

Primitive: Database Isolation Levels, ACID & Concurrency Anomalies · Drill: ACID isolation write skew anomaly (both of its questions are answered just above)

Step 1.5: Holdings Disagree With Orders

Round 1 Step Summary

Two costs stay open for Round 2: every quote goes to every watcher (which won't survive millions of users), and anyone holding a stolen session token can trade.

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Quotes flow in from the vendor, through the quote service, out through the gateways. Orders flow the other way: through the order API into one Aurora transaction, then out to the broker by the router. Everything the broker reports comes back through the router, into the database, and out to the phone as a numbered event.

The pieces:

• Quote service (2 tasks, active and standby) holds the vendor connection and the latest quote per symbol in memory. At the open, our users' ~5,000 watched symbols send it up to about 200K changes a second (R1.7).
• Stream gateways on ECS (Fargate) behind a Network Load Balancer (a layer-4 load balancer that passes TCP connections straight through). They hold the WebSockets, the subscriptions, and send each client one frame a second.
• Order API on ECS behind an Application Load Balancer: authenticates, validates, and runs the order transaction.
• Aurora PostgreSQL (a writer and a reader in another AZ) holds accounts, orders, keys, events, the ledger and an outbox.
• Outbox relay and router. The order transaction also writes an outbox row (the transactional outbox: a table of messages to send, written in the same transaction as the change, so "saved" and "will be sent" can't disagree). The router reads it, sends the order to the partner broker over FIX (the standard messaging protocol between brokers and markets) through a site-to-site VPN, and writes the broker's execution reports back. The outbox loop teaches the relay.

Schema (Aurora PostgreSQL; the ledger's own tables follow the digital wallet loop)

sqlCopy
CREATE TABLE accounts (
    account_id      TEXT PRIMARY KEY,
    user_id         TEXT NOT NULL,           -- the owner; checked against the token on every write
    cash_settled    NUMERIC(18,2) NOT NULL,
    cash_unsettled  NUMERIC(18,2) NOT NULL,
    cash_reserved   NUMERIC(18,2) NOT NULL DEFAULT 0,   -- promised to open buy orders
    event_seq       BIGINT NOT NULL DEFAULT 0           -- last order event number
);

CREATE TABLE positions (
    account_id    TEXT NOT NULL REFERENCES accounts,
    symbol        TEXT NOT NULL,
    quantity      NUMERIC(18,6) NOT NULL,
    qty_reserved  NUMERIC(18,6) NOT NULL DEFAULT 0,     -- promised to open sell orders
    PRIMARY KEY (account_id, symbol)
);

CREATE TABLE order_keys (
    account_id       TEXT NOT NULL,
    client_order_id  UUID NOT NULL,
    request_hash     BYTEA,                   -- NULL for a VOID row
    state            TEXT NOT NULL CHECK (state IN ('ORDER', 'VOID')),
    order_id         TEXT,
    PRIMARY KEY (account_id, client_order_id)
);

CREATE TABLE orders (
    order_id         TEXT PRIMARY KEY,
    account_id       TEXT NOT NULL,
    client_order_id  UUID NOT NULL,
    device_id        TEXT NOT NULL,
    symbol           TEXT NOT NULL,
    side             TEXT NOT NULL CHECK (side IN ('BUY', 'SELL')),
    type             TEXT NOT NULL CHECK (type IN ('MARKET', 'LIMIT')),
    quantity         NUMERIC(18,6) NOT NULL CHECK (quantity > 0),
    limit_price      NUMERIC(18,4),           -- the collar price for a market buy
    time_in_force    TEXT NOT NULL,
    status           TEXT NOT NULL,
    filled_qty       NUMERIC(18,6) NOT NULL DEFAULT 0,
    avg_fill_price   NUMERIC(18,6),
    reserved_amount  NUMERIC(18,2) NOT NULL,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX orders_by_account ON orders (account_id, created_at DESC);

CREATE TABLE executions (
    broker        TEXT NOT NULL,
    exec_id       TEXT NOT NULL,              -- the broker's execution ID
    order_id      TEXT NOT NULL REFERENCES orders,
    quantity      NUMERIC(18,6) NOT NULL,
    price         NUMERIC(18,6) NOT NULL,
    executed_at   TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (broker, exec_id)             -- a resent report is applied once
);

CREATE TABLE order_events (
    account_id  TEXT NOT NULL,
    event_seq   BIGINT NOT NULL,              -- dense per account, set under the row lock
    order_id    TEXT NOT NULL,
    payload     JSONB NOT NULL,
    PRIMARY KEY (account_id, event_seq)
);

CREATE TABLE device_last_order (
    device_id        TEXT PRIMARY KEY,
    account_id       TEXT NOT NULL,
    client_order_id  UUID NOT NULL,           -- the last one accepted from this device
    accepted_at      TIMESTAMPTZ NOT NULL     -- server time; "newer" is judged by this, not the UUIDv7
);

Trace 1: a quote update.

Synthesizing vector architecture diagram...

Two vendor changes, one frame to the phone: the second replaced the first before the gateway's next second came round.

Trace 2: an order with a lost response and a retry.

Synthesizing vector architecture diagram...

The retry creates nothing. It reads the order the first request made, and the user sees one order moving through its states.

R1.7 Numbers

Quotes (assumptions: 50K users online at the open, 10 symbols each, every watched symbol changes at least once a second at the open, which is the worst case; JSON frames)

Orders (assumptions: 100K orders a day; 30% of them in the first half hour; the busiest seconds are 5× that half hour's average)

Monthly egress (assumption: averaged over the 6.5-hour session, stream traffic is about a quarter of the open's peak: fewer users after the first hour, and quiet symbols send nothing): $64 \text{ MB/s} \times 25\% = 16 \text{ MB/s}$; $16 \text{ MB/s} \times 23{,}400 \text{ s} \times 21 \text{ trading days} \approx 7.9$ TB a month.

Fleet (per-task rates are assumptions to confirm by load test; every fleet must carry the peak with one AZ lost, rounded per AZ)

Monthly cost (us-east-1 on-demand list prices; rounded)

Say the headline: the AWS bill is small; the market-data licence probably isn't. Exchanges charge for real-time data per user (with different fees for professional and non-professional users) or through enterprise arrangements, and the vendor passes that on. Get the vendor's fee schedule before choosing between real-time and delayed data for each screen.

R1.8 Trade-Offs

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Streams quotes and explains the WebSocket vs SSE choice by what the client sends, not by fashion.
• Makes orders idempotent with a key born on the phone and claimed atomically on the server, and handles the order whose outcome is unknown.
• Refuses to queue orders offline, and refuses optimistic fills.
• Designs the order state machine, and knows ack from fill.
• Puts buying power on the server under a lock, and can name the anomaly the lock prevents.
• Derives traffic and cost, and knows that data licensing matters more than servers at this size.

Follow-up questions

1. "Why not let the app decide the order is a duplicate?" Answer: the app can't see what the server did with a request whose response was lost. It can only keep the same key and ask. The decision has to be one atomic write on the server.

2. "What if the user edits the price and resubmits while the first order is unknown?" Answer: the edited order is a new intent with a new key, so the app must first resolve the unknown one. While an order is UNKNOWN, the ticket for that symbol shows "Checking your last order…" and won't send a new one until the resolve call returns.

3. "The phone's clock is wrong by an hour. What breaks?" Answer: nothing that matters. Keys only need to be unique (a UUIDv7 has 74 random bits besides its timestamp). The stale timer uses the phone's monotonic clock (time since boot), not wall time. Times on screen come from server_ts.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "3M Traders at the Opening Bell"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 10M accounts, 3M online at 9:30 · 45M subscriptions, 90M quote deliveries/s at peak · 2M orders/day, 5K/s at the open · order in, to broker, P99 < 150 ms · 99.99% in market hours

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built a trading app for a small brokerage: 50K users online at the open, 10 symbols each, 100K orders a day. Quotes stream over one WebSocket per app session: a snapshot on subscribe, then at most one frame a second with every changed symbol, and heartbeats when nothing changes; after 3 seconds of silence, prices grey out and market orders are blocked. Orders carry a client_order_id born on the phone. The server claims it with a unique key in the same Aurora transaction that locks the account row, reserves buying power and inserts the order, so a retry returns the original. The server abandons requests after 5 seconds; the app waits 10 before asking, and its resolve call voids a key that doesn't exist yet, so a late original can't create a second order. We never queue orders offline and never show a fill we weren't told about. The server owns the state machine: received, routed, working, partly filled, filled, cancelled, rejected, expired, with a numbered event per account. An outbox and a router send orders to our partner broker over FIX; execution reports come back and are applied once by execution ID. Holdings come from a ledger, settled on T+1, reconciled nightly with the partner. About $2.4K a month on AWS. Two costs are open: every quote goes to every watcher, and a stolen session token can trade."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: quotes down one path, orders up another, meeting only in the app.

Round 1 step summary

Open costs: fan-out that grows with every watcher; session tokens alone authorize trades.

R2.1 The Scope Raise

Interviewer: "We're a popular retail app now: 10 million accounts, and on a meme-stock morning 3 million people are online at 9:30, each watching about 15 symbols. The hot symbols change a hundred times a second. People scroll and pinch charts on 120 Hz phones and expect it to feel like glass. Last month someone stole a session token and traded in a customer's account, so orders must be signed with the user's biometric. Nobody may ever trade on a stale price. And the first minutes bring 5,000 orders a second."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

1. A binary quote record, with its own sequence. Frames are now binary (a compact encoding such as Protocol Buffers or SBE; the choice doesn't matter here). One WebSocket message carries a frame header and one or more records.

The frame header carries the protocol version, the frame type (DATA, HEARTBEAT, ORDER_EVENT, TIME) and server_ts in milliseconds.

2. Snapshots. In-band, on the same WebSocket, ordered with the updates:

jsonCopy
{ "op": "snapshot", "symbols": ["NVDA"] }

And over HTTP, one URL per symbol, cacheable for 1 second at CloudFront, for the moments before the WebSocket is up (a cold launch, a widget, a notification tap):

httpCopy
GET /v1/quotes/NVDA HTTP/1.1
Host: api.example-broker.com

httpCopy
HTTP/1.1 200 OK
Cache-Control: public, max-age=1
Content-Type: application/json

{ "symbol": "NVDA", "epoch": 7, "cseq": 481203, "bid": "131.42", "ask": "131.45",
  "last": "131.44", "status": [], "server_ts": "2026-09-28T13:30:00.512Z" }

3. A signed order. The app signs the exact bytes it sends; the server parses what was signed, never a re-serialized copy (two JSON encoders rarely agree byte for byte).

httpCopy
POST /v1/orders HTTP/1.1
Host: api.example-broker.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "key_id": "dk_7f3a91c2",
  "payload": "eyJjbGllbnRfb3JkZXJfaWQiOiIwMTkyZjNjNC01YjFlLTdhNDIt...",
  "signature": "MEUCIQDs2x9...AiB4kQ"
}

payload is the base64url of these bytes, and signature is ECDSA P-256 over their SHA-256:

jsonCopy
{
  "client_order_id": "0192f3c4-5b1e-7a42-9c3d-2f8e1a6b7c90",
  "account_id": "acc_5521",
  "device_id": "dev_91c0",
  "symbol": "NVDA",
  "side": "BUY",
  "type": "MARKET",
  "quantity": 20,
  "time_in_force": "DAY",
  "quote_ref": { "symbol": "NVDA", "epoch": 7, "cseq": 481203 },
  "shown_price": "131.45",
  "signed_at_client": "2026-09-28T13:30:01.330Z"
}

• client_order_id is inside the signature. If it were only a header, a captured signed order could be replayed under a fresh key and accepted as a new order.
• quote_ref names the quote the user saw, by the server's own identifiers. The server checks its age with its own clock (step 2.4); signed_at_client is for support, never for decisions.

4. New error codes.

R2.4 Design Evolution: Volume, Frames, Freshness and Intent

Step 2.1: 100 Ticks a Second, Times Millions of Watchers

Synthesizing vector architecture diagram...

Volume shrinks 50× in the conflators (1M to 20K a second), then grows 4,500× in the gateways (20K to 90M), and only there.

Step 2.2: The UI Stutters During Volatility

Synthesizing vector architecture diagram...

Records arrive whenever the network delivers them; the screen reads them only at its own rhythm. Nothing the network does can make a frame late.

Step 2.3: A Record Arrived Out of Order, and a Gap Appeared

Primitive: Distributed Locks & Leases

Step 2.4: The User Traded on a 30-Second-Old Price in an Elevator

Synthesizing vector architecture diagram...

Market orders are allowed only in LIVE. RESYNC and STALE dim the price; HALTED shows a badge.

Step 2.5: A Stolen Session Token Placed Trades

Synthesizing vector architecture diagram...

The private key never leaves the secure hardware, and it signs nothing without the biometric. The server trusts the math, not a flag.

Primitive: OAuth2, OIDC & Distributed Token Authentication

Step 2.6: The Opening Bell Floods Us

Primitive: Distributed Rate Limiting · Drill: Rate limiting a partner API (both of its questions, shared counters and token bucket vs fixed window, are answered just above)

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Three streams. Market data: vendor → feed handlers → conflators → every gateway, order API task and alert engine. Orders: phone → ALB → order API → Aurora → routers → broker. Order events: routers → Valkey channel for that account → the one gateway holding that account's connection → phone.

What changed from Round 1:

• The market-data tier is split into feed handlers, 8 conflator shards with hot standbys, and a gateway fleet; the order API and alert engines subscribe to the same conflated stream.
• Order events go through per-account channels. When a connection authenticates, its gateway subscribes to the account's channel in Valkey (sharded pub/sub, so channels spread over the cluster's shards). Routers publish each numbered event to the account's channel. Pub/sub doesn't store messages, so an event published while a phone is reconnecting is simply missed; the per-account event_seq (step 1.3) makes the gap visible, and the app fetches the missing events from the order API. The numbered log in Aurora is the source of truth; the channel is only the fast path.
• Routers partition accounts over 4 FIX sessions to the broker, each an active router with a standby, over AWS Direct Connect. A router holds its session with a lease and an epoch, like the conflators, and the broker accepts only one logon per session, which fences a second router at the other end. After an order commits, the order API hands it to its router directly (the fast path); the outbox relay only sends orders the fast path missed. Before sending, the router claims the order with UPDATE orders SET status = 'ROUTED' WHERE order_id = $1 AND status = 'RECEIVED', so the fast path and the relay can't both send it.
• DynamoDB holds device keys, sessions, price alerts and the conflator leases.

Server tables added in Round 2

Price alerts ("tell me when NVDA goes above 140")

• An index fed from the change stream. Alert engines own symbols by hash, like conflators. Each keeps, per symbol, two sorted lists of armed thresholds (above and below). The table's DynamoDB Stream is consumed by a Lambda function that forwards each record to the engine that owns its symbol. At startup an engine notes its stream position first: it starts accepting forwarded records into a buffer, then loads its symbols' alerts from the table, then applies the buffered and later records, keeping a record only if its alert version is newer than the one it holds. Loading first and subscribing second would lose any change written in between. Streams order records per item only, and an alert's create, edit and delete are one item, so they arrive in order; different alerts don't need a shared order. The stream consumer is a Lambda function with bisect on error (a failing batch is split in half until the bad record is isolated) and an on-failure destination, and a nightly job compares each engine's index with the table and repairs differences.
• Evaluation. On each conflated record, the engine finds crossed thresholds by binary search in the sorted lists: cost grows with alerts that fire, not with alerts that exist.
• Firing once. The engine writes status = FIRED "only if still ARMED", then sends the push. A retry after a crash between the two re-sends the push, so each push carries a collapse ID (apns-collapse-id on iOS, the notification tag on Android) derived from the alert ID: a second copy replaces the first on the lock screen instead of adding another.
• Fill notifications work the same way, with the order ID and fill number as the collapse ID and tag.

Trace 1: the open.

Synthesizing vector architecture diagram...

Trace 2: the elevator.

Synthesizing vector architecture diagram...

The burst of old frames is received "now" but dated 30 seconds ago; the server-time rule keeps the price dimmed until a fresh frame arrives, and the user confirms at 135.00, not 120.00.

Trace 3: a signed order after a network drop.

Synthesizing vector architecture diagram...

No second signature and no second order. The resolve call either finds the order or makes sure it can never appear.

R2.6 Numbers and Cost

Market data (the interviewer's figures, derived; 40 B per record; the peak assumes every watched symbol changes in every 500 ms window, which is the worst case)

Synthesizing vector architecture diagram...

Conflation is the big cut (1.44 Tbps of payload to 28.8 Gbps); the binary format then cuts JSON's 86.4 Gbps to about 32.6 Gbps on the wire, headers included.

Orders

Order latency budget (P99, from the order reaching our load balancer to the broker receiving it; the steps depend on each other, so they add)

An order that misses the fast path waits for the outbox relay, which polls every second: rare, and outside the fast path's budget, so we alarm on how often it happens.

Fleet sizing (per-task rates are assumptions to confirm by load test; every fleet carries the peak with one AZ lost, rounded per AZ)

Quotas to raise before launch: Fargate vCPUs (225 + 76 tasks × 2 vCPU ≈ 600 vCPUs at the open); DynamoDB warm throughput on the session table; FCM's default of 600,000 messages a minute (10K a second) per project, which a mass alert firing at the open can reach, so push senders pace themselves under it.

Monthly cost (us-east-1 list prices; 21 trading days; stream traffic averaged over the 23,400-second session at a quarter of the open's peak, as in Round 1; rounded)

Say the headline: half the bill is bytes leaving AWS, and every byte saved per record is multiplied by 90M a second. The market-data licences for 3M real-time users are a separate bill, and likely the larger one.

The pricing trap: a managed WebSocket service. API Gateway WebSocket APIs charge per message, tiered ($1.00 per million for the first 1B messages a month, then $0.80 per million; each message up to 32 KB), plus $0.25 per million connection minutes. Our average of 1.5M frames a second over the session is $1.5\text{M} \times 23{,}400 \times 21 \approx 737\text{B}$ messages a month: 1\text{B} \times \1.00 + 736\text{B} \times $0.80 per million ≈ **\590K, about $0.6M**, before connection minutes. Per-message pricing is great for chat; it's the wrong model for a firehose.

R2.7 Trade-Offs

R2.8 Failure Modes

Drill: WebSocket reconnect thundering herd (the storm rows above and step 2.6 answer its first question; R1.8 and R2.7 answer why WebSocket over SSE)

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions (every user-facing rate counts distinct installs, not events: one phone in a reconnect loop can send thousands of stale reports)

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Separates ticks in (per symbol, bounded) from deliveries out (per subscriber) and puts conflation where it cuts the most.
• Handles slow clients by conflating per client instead of queueing.
• Puts the sequence on the conflated stream, with a fenced single writer, and knows why exchange sequence numbers don't work after conflation.
• Designs the render loop around the frame budget, not around message arrival.
• Catches the elevator burst: staleness by server time with a measured offset, and a server-side check by quote reference.
• Signs orders with hardware keys and verifies exact bytes; plans for key invalidation and recovery.
• Prepares for the open on a schedule, sheds before expensive work, and prices the managed alternative.

Follow-up questions

1. "A user wants every tick for one symbol, for day trading." Answer: a separate, opt-in subscription class: one or two symbols at a 100 ms window from the conflator (a second emit schedule for those symbols), licensed and priced for it. It multiplies deliveries only for the few who ask; the default stays at 500 ms.

2. "Why not skip the server check if the app already blocks stale prices?" Answer: the app is outside our control: an old version, a modified app, or a bug. The server's check uses only its own clock and its own records, so it holds even when the client is wrong.

3. "What if the Secure Enclave key is fine but the user's session was stolen on another device?" Answer: the thief has a token but no registered key, so they can't sign orders. They'd have to enroll a new device, which needs a login with a second factor, and Round 3 adds new-device risk checks and alerts to the user's other devices.

Interview gotchas from this round

---

Round 3 · Architect · "A Regulated Broker: Audit, DR and Fairness"

~45 min · Principal (L7) · us-east-1 primary for orders, us-east-2 warm for orders, market data active in both · 25M accounts, 6M online at the open · 90M subscriptions · 8M orders/day with options, 20K/s at the open · RPO 0 for accepted orders · order entry back within 15 min of a region loss

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run a retail trading app with 10 million accounts and 3 million people online at 9:30, watching 45 million symbol subscriptions. Feed handlers take up to a million ticks a second from the vendor; 8 conflator shards keep the latest state per symbol and emit each changed symbol every 500 ms, 20,000 records a second for the whole market. Every gateway gets that stream and flushes each client every 500 ms: 90 million deliveries a second, 28.8 Gbps of payload, about 32.6 on the wire. Slow clients get replaced records, never queues. Records carry (epoch, cseq) on our conflated stream, so the app drops old ones and asks for in-band snapshots on gaps; conflators hold a lease and a new owner bumps the epoch. The app draws once per frame from a latest-value table. Prices go stale after 2.5 s without a frame or when the newest frame is 2.5 s old by server time, using an offset measured on the connection; stale blocks market orders, and the server re-checks the quote reference at 5 s. Orders are signed per use by a P-256 key in the Secure Enclave or StrongBox, unlocked by a current biometric; the server verifies the exact bytes. We pre-scale for the open, shed before TLS, and admit orders with per-account token buckets, cancels first. Order in to broker is about 52 ms at P99, and AWS costs about $59K a month, half of it egress. Open costs: logs aren't records, one broker route, one region, a new device is trusted too easily, releases can break trading, and the risk checks only understand stocks."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: a conflated market-data tier and a signed, idempotent order path, in one region.

Steps so far

Open costs: no provable records; one route; one region; weak new-device trust; risky releases; stock-only risk checks.

R3.1 The Scope Raise

Interviewer: "We're a registered broker-dealer now: 25 million accounts, 6 million online at the open, and options trading. We route to several market centers ourselves. Regulators can ask us about any order from years ago, and we must be able to defend where we sent it. If our main region fails at 10 a.m., no accepted order may vanish and customers must not be left guessing about their positions. Account takeovers drained a few hundred accounts last quarter. And last spring an app release broke the order ticket on a trading morning."

About "five nines during market hours". 99.999% of 1,638 market hours a year is $1{,}638 \times 3{,}600 \times 0.00001 \approx 59$ seconds of downtime a year (the "5.26 minutes a year" often quoted is five nines of a 24/7 year). A single regional failover takes minutes, so no design meets five nines through a region loss. We say so and set two targets: 99.99% in market hours for normal operation (about 9.8 minutes a year), and a region loss handled as a disaster with RPO 0 for accepted orders and order entry back within 15 minutes.

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

1. The audit record. Every order event becomes one of these, written once:

jsonCopy
{
  "record_id": "ae_01J9ZQ4M2T8V",
  "account_id": "acc_5521",
  "account_event_seq": 88213,
  "epoch": 3,
  "order_id": "ord_9QH1",
  "client_order_id": "0192f3c4-5b1e-7a42-9c3d-2f8e1a6b7c90",
  "type": "ORDER_RECEIVED",
  "occurred_at": "2026-09-28T13:30:01.412318Z",
  "clock_source": "amazon-time-sync",
  "actor": { "user_id": "u_318", "device_id": "dev_91c0", "key_id": "dk_7f3a91c2", "app_version": "8.4.1", "ip": "203.0.113.24" },
  "evidence": { "payload_sha256": "9f2c…", "signature": "MEUCIQDs2x9…", "quote_ref": { "symbol": "NVDA", "epoch": 7, "cseq": 481203 },
                "shown_price": "131.45", "disclosures_version": "opt-2026-07" },
  "body": { "symbol": "NVDA", "side": "BUY", "type": "MARKET", "quantity": 20, "time_in_force": "DAY" }
}

Other types: RISK_PASSED or RISK_REJECTED (with the rule versions used), ROUTE_DECIDED, SENT, ACKED, FILL, CANCEL_REQUESTED, CANCELLED, REJECTED, EXPIRED, RECONCILED.

2. The routing decision record.

jsonCopy
{
  "type": "ROUTE_DECIDED",
  "order_id": "ord_9QH1",
  "decided_at": "2026-09-28T13:30:01.431907Z",
  "policy_version": "route-2026-09-15",
  "nbbo": { "bid": "131.42", "ask": "131.45", "source": "consolidated", "as_of": "2026-09-28T13:30:01.430Z" },
  "candidates": [
    { "venue": "WHOLESALER_A", "score": 0.91, "expected_price_improvement_bps": 1.8, "fill_rate": 0.998, "status": "UP" },
    { "venue": "EXCHANGE_X",   "score": 0.84, "expected_price_improvement_bps": 0.0, "fill_rate": 0.97, "fee_per_share": "0.0030", "status": "UP" },
    { "venue": "EXCHANGE_Y",   "score": null, "status": "CIRCUIT_OPEN" }
  ],
  "chosen": "WHOLESALER_A",
  "reason": "highest score for NVDA, market, 1-100 shares, this quarter's stats"
}

3. Order status "unknown", then reconciled. After a region failover, an order the new region can't yet confirm is shown honestly:

jsonCopy
{ "order_id": "ord_9QH1", "status": "UNKNOWN", "reason": "REGION_FAILOVER",
  "message": "We're confirming this order with the market. Its funds stay reserved until we do." }

It later moves to a real state, with a RECONCILED audit record saying how we learned it.

4. Device and session risk signals. Every sensitive call carries an integrity token (a Play Integrity verdict or an App Attest assertion), the device ID and the app version; the server adds network, location and behaviour signals and returns a decision:

jsonCopy
{ "decision": "STEP_UP", "methods": ["passkey", "trusted_device_approval"], "reason": "NEW_DEVICE_24H" }

R3.4 Design Evolution: Records, Routes, Regions and Releases

Step 3.1: Every Order Must Be Provable Years Later

Primitive: Event Sourcing & CQRS

Step 3.2: "Best Execution"

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance

Step 3.3: A Region Fails at 10 a.m.

Recovery objectives

Failover timing (every delay counted; the parallel steps take the max)

We plan capacity and residency before the day: the us-east-2 quotas (Fargate vCPUs, Aurora instances, NLB) already match production, and every Region in the design is in the US, so failover never moves data somewhere it isn't allowed to be.

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active · Drill: Multi-region replication consistency (its first question is the async-replication wrong answer above; its second is why synchronous replication stays narrow)

Step 3.4: Account Takeover Drains Accounts

Primitive: Bot Defense, Sybil Resistance & Registration Abuse · Drill: Credential stuffing surge (per-IP limits and SMS-on-every-login, both answered above) · Drill: JWT revocation token blacklist (point 5 answers both)

Step 3.5: A Release Broke Order Entry on a Trading Day

Step 3.6: Options and Margin Need Complex Risk Checks

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Market data runs in both regions all the time. Orders run in one: the journal is the only synchronous cross-region write, and the venues' drop copies keep the standby informed of every fill. The routing control decides where orders go, and it moves only after the old region is fenced.

Scaling the order path to 20K a second: account cells. An order touches one account, so we split accounts over 4 cells, each a complete Round 2 order path (an Aurora cluster and its share of order API, risk and router tasks) handling up to 5K orders a second. A small cell directory (account → cell, a DynamoDB global table cached in every task) routes each request. A bad deploy or a hot account affects one cell. Sharding a ledger is covered in depth in the digital wallet loop, step 2.1.

Order latency budget, Round 3 (P99, dependent steps add): Round 2's 52 ms, plus the MRSC journal write (~20 ms: a round trip to us-east-2 of roughly 10–15 ms, plus the write), plus options risk (up to 10 ms instead of 3), plus the routing decision (2 ms): about 81 ms, still well under 150 ms.

Trace 1: a region failure during an order.

Synthesizing vector architecture diagram...

The resolve reads the journal, not us-east-2's lagging Aurora copy, so it can't wrongly void a key the lost region accepted. The journal is why the order isn't lost; the fence is why "unknown to the venue" is final; and the rule "never send it late" is why the customer isn't surprised by a trade at 10:07.

Trace 2: a routing decision, audited later.

Synthesizing vector architecture diagram...

Trace 3: a takeover attempt, blocked.

Synthesizing vector architecture diagram...

The attacker had everything the old design asked for. What they didn't have was a device the account already trusted.

R3.6 Numbers and Cost

Traffic (25M accounts; 6M online at the open; 15 symbols each; the same record and frame sizes as Round 2)

Fan-out capacity in both regions. Each region must carry all 6M connections if the other is lost: $6\text{M} \div 20\text{K} = 300$ gateways per region, 100 per AZ. Normally each carries about 3M at 50% use, which also covers losing an AZ inside a region ($200 \times 20\text{K} = 4\text{M} \ge 3\text{M}$). 600 gateways in market hours.

Order path (per task 200 orders/s, lower than Round 2's 250 because of options risk; AZ-loss sizing): $20{,}000 \div 200 = 100$, so 50 per AZ, 150 tasks in us-east-1; 30 warm in us-east-2, scaled to 150 during a failover.

Audit records (assumptions: ~10 events per order, ~1 KB each, compressing about 4×)

Firehose bills each record in 5 KB increments, so we pack about four 1 KB events per record: 20M records a day billed at 5 KB is 100 GB a day, instead of 400 GB if each event were its own record.

Journal (MRSC): 2 writes per order (received, terminal) of ~2 KB (2 write units) in 2 replica Regions: $8\text{M} \times 2 \times 2 \times 2 = 64\text{M}$ write units a trading day; stored 30 calendar days, about 21–22 trading days: $8\text{M} \times 2 \text{ KB} \times 21 \approx 340$ GB per replica.

Monthly cost (us-east-1 and us-east-2 list prices; 21 trading days; stream traffic averaged at a quarter of the peak over the session; rounded)

What DR costs: the us-east-2 Aurora secondaries (4 × $2.87 × 730 ≈ $8.4K), the warm order API and routers (≈ $2.3K), the gateways beyond what one region already needs (a single region sized to lose an AZ would run 450, so the extra is ~150: 150 × $0.079 × 210 ≈ $2.5K), us-east-2's own feed handlers and conflators (26 tasks × $0.079 × 730 ≈ $1.5K), ARC ($1.8K), and a second set of connections and a journal replica (≈ $1.2K): about $17.7K a month, roughly 12% of the bill (a rough split: some of it also serves nearby users every day), for RPO 0 on accepted orders and 15 minutes of RTO. Retention storage for six years of records is tiny next to it. Egress is still the largest line, and market-data licences for 6M real-time users are still outside this table.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we show prices that are fresh enough to trust, and make every order happen exactly once, only if the user meant it? The answer is now:

• Fresh enough to trust: the market's firehose is conflated to what a phone can use, sequenced on our own stream, drawn once per frame, and marked stale by two clocks the phone can trust, with the server checking again (Rounds 1 and 2).
• Exactly once: a key born on the phone, claimed atomically and never forgotten; a resolve call for doubts; a journal that survives a region; the market's own reports as the final word (Rounds 1 and 3).
• Only if the user meant it: a signature from a key only their current biometric unlocks, a fresh price, a device the account trusts, and never an order sent later than they meant it (Rounds 2 and 3).
• And provably so: every step recorded once, kept write-once, and reviewable years later (Round 3).

R3.8 Failure Modes

Drill: Circuit breaker cascading thread stall (the venue row answers both of its questions: thread exhaustion from a slow dependency, and why aggressive timeouts alone are wrong)

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6 (user-facing rates count distinct installs or accounts)

Opening-bell procedure OPS 10 · REL 7

Synthesizing vector architecture diagram...

1. 8:30: both feed sources healthy in both regions; drop copies flowing to both.
2. 8:50: FIX sessions logged on; each venue's circuit closed.
3. 9:00: scheduled scaling has run; check desired and running counts (CLI 2) and that Fargate quotas leave headroom.
4. 9:20–9:30: watch connections ramp (CLI 1) and snapshot load.
5. 9:30–10:00: watch order P99, 429 rates, stale installs; the freeze is in force.
6. 10:30: let scale-in bring gateways down as connections fall.

Vendor feed failover procedure REL 11

1. Confirm it's the source: one source silent or diverging while the other is healthy, in both regions.
2. Switch the feed handlers to the healthy source (automatic after 1 s of silence; manual for "wrong but not silent" data).
3. Watch stale installs fall and conflators keep cseq continuous (no client gap spike).
4. Tell compliance if displayed data was wrong for any period: it may matter for customer complaints and trade reviews.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the names with real ones.

textCopy
# 1. Active WebSocket connections through a region's NLB
aws cloudwatch get-metric-statistics --namespace AWS/NetworkELB --metric-name ActiveFlowCount --dimensions Name=LoadBalancer,Value=net/stream-nlb/0123456789abcdef --statistics Maximum --period 60 --start-time 2026-09-28T13:00:00Z --end-time 2026-09-28T14:00:00Z

# 2. Is the gateway service at open size?
aws ecs describe-services --cluster market-data --services stream-gateway --query "services[0].[desiredCount,runningCount]"

# 3. The scheduled pre-scale, in Eastern time
aws application-autoscaling put-scheduled-action --service-namespace ecs --scalable-dimension ecs:service:DesiredCount --resource-id service/market-data/stream-gateway --scheduled-action-name open-prescale --schedule "cron(0 9 ? * MON-FRI *)" --timezone "America/New_York" --scalable-target-action MinCapacity=300,MaxCapacity=360

# 4. Aurora global replication lag for one cell's secondary
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name AuroraGlobalDBReplicationLag --dimensions Name=DBClusterIdentifier,Value=orders-cell-1-use2 --statistics Maximum --period 60 --start-time 2026-09-28T13:00:00Z --end-time 2026-09-28T14:00:00Z --region us-east-2

# 5. Failover: promote one cell's secondary (unplanned; accepts replication lag as loss)
aws rds failover-global-cluster --global-cluster-identifier orders-cell-1 --target-db-cluster-identifier arn:aws:rds:us-east-2:111122223333:cluster:orders-cell-1-use2 --allow-data-loss

# 6. Failover: send order traffic to us-east-2 (run against one of the cluster's endpoints)
aws route53-recovery-cluster update-routing-control-state --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/abc/routingcontrol/use2-orders --routing-control-state On --region us-west-2 --endpoint-url https://host-example.us-west-2.example.aws

The failover commands run only in the gated order of step 3.3: fence first, then promote (5), then the epoch bump and FIX logons, then traffic (6).

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Defines availability in market hours honestly, and separates normal operation from disaster with explicit RPO and RTO.
• Knows why asynchronous replication breaks dedup guards, and keeps synchronous replication narrow: a journal of what we promised.
• Fences the old region at the market (the FIX sessions), not only in our own code, and refuses to send stale orders late.
• Builds records from the same events that drive the system, write-once, provably complete, and keeps legal claims general unless verified.
• Treats routing as a measured, versioned, reviewable decision.
• Puts friction where attackers profit, and makes revocation fast without a lookup per request.
• Makes releases boring on trading days, and keeps a fallback path for the one screen that must never break.
• Prices DR and says what it buys.

Follow-up questions

1. "Why not make us-east-2 a full active-active order region for users closer to it?" Answer: the gain is a few milliseconds for some users; the cost is two writers for the same accounts, which needs either cross-region locking on every order or accepting double spending during lag. An account could have a home region, with cells assigned per region, but then each home region needs its own standby, and we've built two of this design. We'd do it for residency, not for latency.

2. "The venue says it never received an order, but its message is still in flight on the old session." Answer: that's why we fence first. Once the venue has ended the old session, nothing more from it can arrive, and "never received" becomes final. Before the fence, the answer is "not yet", and the order stays UNKNOWN.

3. "A regulator asks for everything about one account over three years." Answer: a query over the archived records by account_id (Athena over the Parquet files for recent years; a restore from Deep Archive for older ones, which takes hours, fine for a request with notice), checked for completeness by the dense sequence, and verified against the signed chain heads.

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: how fresh must prices be, who executes the orders, and what exactly does the user see between tapping Buy and owning the shares?"
• When the scope is raised: "Here's what breaks, and I'll fix it in this order: anything that can create an order the user didn't mean or lose one we accepted, then anything that shows a price that isn't true, then smoothness, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels