Design a Mobile Chat Client Architecture

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

In this loop the phone is part of the system. It has its own database, a radio that wants to sleep, an operating system that freezes and kills apps, and (in Round 3) the only copy of the encryption keys. Every round designs both halves: the app on the device and the backend it talks to. How the server orders messages and fans them out to group members is the subject of the chat and instant messaging loop; here we link to it instead of teaching it again. The local database basics (WAL mode, background schedulers, schema migrations) are taught in the offline-first news feed loop.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Makes Mobile Chat Hard?

You Already Use One: a Walkie-Talkie That Keeps Falling Asleep

Imagine chatting over a walkie-talkie that switches itself off every few seconds to save its battery. When it's off, it can't hear anything. When you pick it up, it takes a moment to come back, and it may have moved to a different channel. Yet the people you talk to expect every word to arrive, in order, and nothing said twice.

A phone is that walkie-talkie. Its radio sleeps to save battery, the operating system freezes apps you aren't looking at and kills them when it needs memory, and you walk from Wi-Fi to 5G in the middle of a sentence. A chat app still has to feel instant.

What Makes It Hard

• Keeping a connection costs battery; dropping it costs latency. A connection held open in the background wakes the radio for every keep-alive packet. A closed connection means the next message has to find another way in.
• The OS is in charge. iOS and Android decide when a backgrounded app may run, for how long, and whether a "wake up" push is delivered at all.
• Messages must never be lost or shown twice, even when the app is killed halfway through sending one, or a retry races the original.
• The device may be the only place the data exists. An unsent message in the outbox, and (in Round 3) the keys that decrypt everything, live only on the phone.

The Question the Whole Loop Answers

How do we make chat feel live on a device that keeps going to sleep, losing its network and getting killed?

The answer gets sharper every round:

• Round 1: the local database is the truth for the screen. A message is written locally first and sent from an outbox; the network only fills and drains that database.
• Round 2: a live socket only while the app is in the foreground, OS push notifications for everything else, media on a separate resumable path, and receipts as cursors, all inside a battery budget.
• Round 3: the keys live on the devices, not on our servers. Every device is a separate encrypted endpoint, the server carries sealed envelopes, and restoring a new phone becomes a key-management problem.

---

Round 1 · Mid-level · "A Chat Screen That Works Offline"

~35 min · SDE II (L5) · 1 region, 3 AZs · 1M DAU · 20M messages/day, ≈ 810/s peak · ~100K open sockets · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Our second-hand marketplace has an in-app chat between buyers and sellers. Users complain that messages take seconds to appear, disappear when they're on the train, and sometimes show up twice. Design the chat on the phone, and what it needs from the backend." We ask before we draw, and we say what each answer changes.

Out of scope for this round:

• Receipts beyond "sent".
• Push notifications when the app is in the background.
• Media.
• End-to-end encryption and multiple devices.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into a requirement:

Not yet: delivered and read receipts, push notifications, media, end-to-end encryption.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Instant local display. How long from tapping send to the message on screen? It can't depend on the network at all, so it must come from a local write.
• No lost messages. If the app is killed between "tap" and "sent", is the message still there on the next launch?
• No duplicates. If the network drops after the server stored a message but before the phone heard back, the phone will retry. Does the other person see it twice?
• Order. Do both people see the conversation in the same order, even if their phones' clocks disagree?
• Battery. How long do we keep a connection open, and what does it cost the phone when the app is in the background?

R1.4 The API

The app speaks two protocols: a WebSocket for live traffic while the app is open, and plain HTTPS for catch-up and history.

A WebSocket is a connection that starts as an HTTP request and then "upgrades" into a two-way channel over the same TCP connection. After the upgrade, either side can send a message (a frame) at any time.

The backend protocol is the one in the chat loop's Round 1 API: the app gets a short-lived connect ticket over HTTPS, then opens the socket with it. The frames that matter to the phone are three:

SEND_MESSAGE (phone → server), JSON in Round 1:

ACK (server → phone), sent only after the message is stored:

RECEIVE_MESSAGE (server → phone): conversation_id, server_msg_id, client_msg_id (the sender's), seq, sender_id, body, sent_at.

Catch-up and history over HTTPS:

httpCopy
GET /v1/conversations/c_7Hk2/messages?after_seq=10477&limit=50 HTTP/1.1
Host: chat.example-market.com
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "messages": [
    { "seq": 10478, "server_msg_id": "718293847561029300", "client_msg_id": "1f0c…",
      "sender_id": "u_802", "body": { "text": "Yes, still available" },
      "sent_at": "2026-09-26T18:01:02.410Z" }
  ],
  "has_more": false
}

after_seq reads forward from where the phone is (catch-up); before_seq reads backward (scrolling up into older history). And one call to learn which conversations changed:

httpCopy
GET /v1/inbox?changed_since=inb_8Kq2xW HTTP/1.1
Authorization: Bearer <access_token>

It returns each changed conversation with its newest seq and a new sync_token to send next time. The token is opaque: the server builds it from its own change positions, not from a millisecond timestamp, because two changes written at nearly the same moment can become visible out of order and a pure time cursor would skip the later-committed one. (If a timestamp is used, the server re-reads with an overlap margin of a few seconds, and the phone deduplicates.)

The server never trusts identity from a frame: sender_id is taken from the authenticated connection, and every send and read checks that the caller is a member of conversation_id.

Recap

• A WebSocket while the app is open; HTTPS for catch-up and history.
• SEND_MESSAGE carries a phone-generated client_msg_id; ACK maps it to the server's seq.
• Catch-up by a per-conversation cursor, after_seq, plus an inbox call for "what changed".

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From "Send and Wait" to a Local-First Chat

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

The chat screen calls POST /v1/messages when the user taps send, waits for the response, and then shows the message. To see new messages, it asks the server again when the screen opens.

Synthesizing vector architecture diagram...

Everything the screen shows comes from a network call. If the call is slow or fails, the screen has nothing.

What's good about it: it's simple, and the server is the only copy of anything. What it costs us: on a weak signal the user stares at a spinner after every send; with no signal the message is simply gone; and the screen is blank on the train.

Step 1.1: The Message Only Appears After the Server Replies

Primitive: Change Data Capture & the Outbox Pattern · Primitive: Write-Ahead Log & LSM Trees

Step 1.2: A Retry Sent the Message Twice

Primitive: Distributed Unique ID Generators

Step 1.3: Replies Arrive, but Slowly

Primitive: WebSocket, SSE & Long Polling

Step 1.4: I Opened the App After a Day: What Did I Miss?

Step 1.5: Messages Show Out of Order

Primitive: Distributed Unique ID Generators

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get names on both sides.

Synthesizing vector architecture diagram...

On the phone, every arrow into the database goes through one writer, and the screen only ever reads. The outbox worker sends through the socket manager, and runs catch-up over HTTPS before it drains. On the server, a message is stored with its sequence number before it is acknowledged; the backend details are in the chat loop's Architecture v1.

The local schema (SQLite, SQL shown as a data model):

sqlCopy
CREATE TABLE conversation (
  conversation_id  TEXT PRIMARY KEY,
  peer_user_id     TEXT NOT NULL,
  title            TEXT,
  newest_seq       INTEGER NOT NULL DEFAULT 0,  -- newest seq the server has told us about
  synced_seq       INTEGER NOT NULL DEFAULT 0,  -- we hold every message up to here
  last_read_seq    INTEGER NOT NULL DEFAULT 0,  -- unread = newest_seq - last_read_seq; raised on own sends
  last_activity_ms INTEGER NOT NULL             -- server time, for the chat list order
);

CREATE TABLE message (
  local_id         INTEGER PRIMARY KEY,          -- order the user wrote pending messages
  conversation_id  TEXT NOT NULL REFERENCES conversation(conversation_id),
  client_msg_id    TEXT NOT NULL,                -- created before the first send attempt
  server_msg_id    TEXT UNIQUE,                  -- NULL until acknowledged
  seq              INTEGER,                      -- NULL until acknowledged
  sender_id        TEXT NOT NULL,
  body             TEXT NOT NULL,
  state            TEXT NOT NULL CHECK (state IN ('SENDING','SENT','FAILED','RECEIVED')),
  created_local_ms INTEGER NOT NULL,             -- phone clock: display of pending rows only
  sent_at_ms       INTEGER,                      -- server clock
  attempts         INTEGER NOT NULL DEFAULT 0,
  next_attempt_ms  INTEGER
);

CREATE UNIQUE INDEX idx_message_order ON message(conversation_id, seq);
CREATE UNIQUE INDEX idx_message_client ON message(conversation_id, sender_id, client_msg_id);
CREATE INDEX idx_outbox ON message(conversation_id, local_id) WHERE state = 'SENDING';

CREATE TABLE sync_state (
  id              INTEGER PRIMARY KEY CHECK (id = 1),
  inbox_token     TEXT,                          -- changed_since for GET /v1/inbox
  clock_offset_ms INTEGER NOT NULL DEFAULT 0     -- server time minus phone time
);

A unique index in SQLite allows many rows with a NULL seq, which is exactly what pending messages need. The partial index makes "next pending message in this conversation" a cheap lookup.

The message's life on the phone:

Synthesizing vector architecture diagram...

A message leaves SENDING only when the server has proved it has the message: an ACK, or the message appearing in catch-up. Retry keeps the same client_msg_id, so a retry can never make a second copy.

Trace 1: sent offline, then reconnect

Synthesizing vector architecture diagram...

Catch-up runs before the outbox drains, so a message the server already had would have been matched instead of resent.

Trace 2: a lost ACK and a retry

Synthesizing vector architecture diagram...

The server stores the message once; the phone gets the same answer twice.

R1.7 Numbers

Traffic (assumptions: each daily user sends 20 messages a day; the evening peak is 3.5× the daily average; 10% of daily users have the app open at peak)

On the phone (a row is about 350 bytes: a ~60-byte text, three 36-character IDs, a numeric server ID, a few integers, SQLite's record overhead, and two index entries)

Older messages are deleted locally in a background job and fetched again with before_seq when the user scrolls that far. Pending (SENDING, FAILED) rows are never evicted.

Monthly cost. The backend is the chat loop's Round 1 system at exactly this traffic (1M DAU, 20M messages a day, 100K sockets), so we reuse its derivation, minus the push notifications we don't send yet (us-east-1 list prices, 730 hours a month; rounded):

The phone adds almost nothing to the server's bill: catch-up is a few small reads per app open. What the phone costs is paid on the phone: storage (a few MB at most) and battery (a socket only in the foreground).

R1.8 Trade-Offs

How the app hears about new messages in the foreground.

SSE over HTTP/2 is a real option: it rides ordinary HTTP, multiplexes with other requests, and reconnects on its own. But chat sends constantly in both directions (messages, and in Round 2 receipts and typing), and with SSE every send is a separate HTTP request with its own headers, while a WebSocket frame costs a few bytes of header. Both need a stateful connection tier on the server, so SSE doesn't save us that. We choose WebSockets and keep HTTPS polling as a fallback for networks that block them.

Where the local data lives.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Makes the local database the source of what the screen shows, and writes before showing.
• Explains why the outbox survives the app being killed, and that there's one writer.
• Generates the message ID on the phone before the first attempt, and explains why text or time windows can't deduplicate.
• Opens a socket only in the foreground, and knows the OS suspends backgrounded apps.
• Catches up by cursor in transactions, and reconciles pending messages against catch-up.
• Orders by the server's sequence, not the phone's clock.

Follow-up questions

1. "Why not let the server assign the message ID and have the phone wait for it?" Answer: then the phone can't name a message until the server has answered, which is exactly the answer that gets lost. If the first attempt is stored but the reply is lost, a retry has nothing in common with the first attempt, and the server stores a second copy. The ID has to exist before the first attempt.

2. "The user edits a pending message before it's sent. What happens?" Answer: if the row is still SENDING and not in flight, the writer updates its body in place; the next attempt carries the new text with the same client_msg_id. If it's in flight, we treat the edit as a separate "edit" message that follows it, because we can't know whether the server already stored the old text.

3. "How does the app delete old messages without deleting something important?" Answer: a background job deletes SENT and RECEIVED rows beyond the cap per chat, oldest first, in small transactions through the same writer. It never touches SENDING or FAILED rows, and it never moves synced_seq: the cursor says what the phone has received, and older messages can be fetched again with before_seq.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "30M Daily Users, Receipts, Media and Battery"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 30M DAU · 10M open sockets at peak · 500M messages/day, ≈ 20K/s peak · 99.99% · send-to-display < 100 ms and reconnect < 300 ms P99 on good networks · < 1.5% battery a day for messaging

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built buyer–seller chat for a marketplace: 1M daily users, 20M messages a day, about 810 a second at peak, 100K open sockets. The rule on the phone is that the screen reads only from a local SQLite database in WAL mode, and every write goes through one serial writer. Tapping send writes the message as SENDING with a client message ID generated before the first attempt, and shows it at once; those rows are the outbox. An outbox worker sends them over a WebSocket that is open only while the app is in the foreground, one in flight per conversation, retrying with the same ID; the server stores once and answers retries with the original sequence number. When the app opens, it catches up per conversation from a cursor, one transaction per page, and reconciles pending rows against catch-up by client ID before sending anything. Messages sort by the server's sequence, pending ones last, and times come from the server. The backend is the chat loop's Round 1 system, about $7,300 a month. Open costs: nothing reaches a phone whose app is in the background, there are no receipts or media, and we haven't counted the battery."

Architecture v1, compact

Synthesizing vector architecture diagram...

The screen reads, one writer writes, the outbox drains through a foreground socket after catch-up.

Round 1 step summary

Open costs: no delivery while the app is backgrounded or closed; no delivered or read receipts; no media; no numbers for battery or data use.

R2.1 The Scope Raise

Interviewer: "We're now a consumer messenger: 30 million people use it daily, and at the evening peak 10 million have it open. People expect a message to arrive when their phone is in their pocket, and to see sent, delivered and read ticks. They send photos and videos up to a few hundred MB, often on a bad connection. They walk from Wi-Fi to cellular mid-conversation. And our top support complaint is battery drain."

We ask back, and say what each answer changes.

Scope change

The "Not yet" list from R1.2 is now mandatory: receipts, push and media.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

The socket speaks binary now. Round 1's frames were JSON. From Round 2 the socket carries compact binary frames: each field has a number and a type, written as a tag byte, then the value (a Protocol Buffers-style encoding). R2.6 does the byte count and R2.7 the trade-off. Here are the frames as field tables; the "bytes" column is our estimate for a typical frame.

Connecting and resuming in the upgrade request. Under the WebSocket protocol (RFC 6455) the phone can't send frames until the server's 101 Switching Protocols arrives, so anything the server needs for a resume goes into the upgrade request itself:

httpCopy
GET /v1/ws?inbox_token=inb_8Kq2xW&app_version=8.14.0&platform=ios&network=cellular&caps=1f HTTP/1.1
Host: chat.example.com
Authorization: Bearer <access_token>
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

The server answers 101 and, as its first frame in the same flight, WELCOME: the recommended heartbeat interval and the conversations that changed since inbox_token. There is no separate hello frame, so resuming costs no extra round trip.

RECEIVE_MESSAGE (server → phone), the frame that dominates traffic:

Heartbeats:

Receipts are cursors, one frame per conversation (step 2.5):

The server checks that the caller is a member of the conversation, and clamps up_to_seq to the member's own range: from the seq at which they joined to the smaller of the conversation's newest seq and the seq at which they left (if they did). A DELIVERED cursor is also capped at the highest seq the server has actually sent or served to that device. A buggy or hostile client can't mark messages it hasn't received.

Push payloads. On iOS we send through APNs (Apple Push Notification service); on Android through FCM (Firebase Cloud Messaging). Both limit the payload to 4 KB.

An APNs alert (headers apns-push-type: alert, apns-priority: 10, apns-collapse-id: c_9Jq1):

jsonCopy
{
  "aps": {
    "alert": { "title": "Priya", "body": "See you at 7!" },
    "mutable-content": 1,
    "thread-id": "c_9Jq1",
    "sound": "default"
  },
  "conv": "c_9Jq1",
  "up_to_seq": 20418
}

An FCM data message (HTTP v1 API; data values must be strings):

jsonCopy
{
  "message": {
    "token": "<device registration token>",
    "android": { "priority": "HIGH", "ttl": "86400s" },
    "data": { "conv": "c_9Jq1", "up_to_seq": "20418", "sender": "Priya", "text": "See you at 7!" }
  }
}

Resumable media upload:

httpCopy
POST /v1/media/uploads HTTP/1.1
Authorization: Bearer <access_token>
Content-Type: application/json

{ "conversation_id": "c_9Jq1", "mime": "video/mp4", "size_bytes": 209715200,
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" }

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{ "media_id": "m_5Qz8", "part_size_bytes": 8388608, "part_count": 25,
  "part_urls": [ { "part": 1, "url": "https://chat-media-prod.s3.us-east-1.amazonaws.com/…&X-Amz-Signature=…" } ],
  "urls_expire_in_s": 3600 }

Then: PUT each part to its pre-signed URL; POST /v1/media/uploads/m_5Qz8/part-urls with { "parts": [7, 8, 9] } to get fresh URLs for parts not yet uploaded; and POST /v1/media/uploads/m_5Qz8/complete with { "parts": [ { "part": 1, "etag": "…" }, … ] }. Every call checks that the caller created m_5Qz8. A file of 8 MiB or less is a single pre-signed PUT.

R2.4 Design Evolution: Push, Radios, Media and Receipts

Step 2.1: Messages Don't Arrive When the App Is in the Background

Step 2.2: Silent Pushes Get Dropped

Step 2.3: Heartbeats Drain the Battery; Long Silence Kills the Connection

Step 2.4: A 200 MB Video Over the Socket Stalls the Chat

Loop: Design Google Drive (file sync and storage) · Loop: Design S3-Like Object Storage

Step 2.5: Read Receipts Flood the Network in Groups

Step 2.6: The Message State Is Confusing

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The phone has three ways in: the socket in the foreground, OS push in the background, and HTTPS for catch-up and media. All three write through the same single writer. On the server, the chat core is the chat loop's Round 2 pipeline; what's new here is the deadline set in ElastiCache, push workers talking to APNs and FCM directly, and a media path that never touches the gateways.

The pieces added in this round:

• Push workers (ECS) hold HTTP/2 connections to APNs (token-based auth) and use FCM's HTTP v1 API. They read from a queue fed by the delivery workers and the deadline sweeper, coalesce per recipient and conversation, and delete device tokens that APNs or FCM report as no longer valid. Every push payload is built from the stored message, so a replayed or retried push carries the same up_to_seq and the same collapse ID.
• The deadline set in the routing ElastiCache cluster, and a sweeper in the delivery service.
• The media service (ECS behind the ALB), the S3 media bucket (lifecycle: abort incomplete multipart uploads after 2 days; expire objects after 30 days; blocked public access, reachable only through CloudFront with origin access control), and CloudFront with signed URLs.
• Consumers of the ingest and delivery logs follow the usual rules for stream consumers: a record that keeps failing is retried a bounded number of times, then parked in a dead-letter topic with an alarm, so one poison record can't stall a partition; and anything parked is repaired from the stored message, not by replaying the raw log.

The server-side media record (DynamoDB):

Trace 1: a backgrounded delivery with escalation (Android)

Synthesizing vector architecture diagram...

The phone's own DELIVERED cursor is the only thing that clears the deadline.

Trace 2: a network handoff (Wi-Fi to 5G, RTT 50 ms)

Synthesizing vector architecture diagram...

Three round trips from "the OS told us" to "resumed". The old socket's registry entry is overwritten by the new connection, and its cleanup deletes the entry only if it still names the old connection.

Trace 3: a large media send

Synthesizing vector architecture diagram...

The socket carries one small frame for a 200 MB video, and nothing is re-uploaded after the tunnel.

R2.6 Numbers and Cost

Traffic (the Round 2 figures, derived; peak is 3.5× the average)

A correction to a common receipt figure. A common shortcut counts 2 receipts per message (1B a day, ≈ 40.5K/s at peak). That is right only when every message has one recipient. With groups, receipts are per recipient: 1.9B a day before watermarks.

Frame sizes: our byte budget, estimates.

A second correction: the often-quoted "≈ 800 B JSON vs ≈ 160 B binary, 400 vs 80 GB a day" describes an end-to-end encrypted frame (a ratchet public key, an authentication tag, ciphertext) and counts only the upload direction. Round 2 has no end-to-end encryption, so its frame is smaller; Round 3 re-derives the encrypted one.

Heartbeats: blend the rates, not the intervals. Assume half the open sockets are on Wi-Fi (60 s) and half on cellular (180 s). The average ping rate is $(1/60 + 1/180) \div 2 = 0.0111$ per second, an effective interval of 90 s, not the 120 s you get by averaging the intervals.

Heartbeats move about 5.7 times the bytes of all binary message frames (1.15 TB vs 203 GB a day). The interval matters more than the encoding.

Gateways: round per AZ, and survive an AZ. A naive 200 tasks (50,000 sockets each) holds exactly 10M, with no headroom, and "67 per AZ" is 201.

50,000 sockets per task is an assumption to load-test (TLS buffers per connection decide it).

The routing cache (ElastiCache for Valkey; operations per second at peak):

At a planning figure of 80K operations a second per shard (to load-test), 4 shards would run at 99% of that figure with no headroom, so we use 5 shards (≈ 63K each at peak), each a primary and a replica in another AZ: 10 × cache.r7g.xlarge.

Push volume. Assume 55% of recipient deliveries find the app not in the foreground, and per-conversation coalescing keeps 57% of those: $950\text{M} \times 0.55 \times 0.57 \approx$ 300M pushes a day, ≈ 3,472/s on average and ≈ 12.2K/s at peak. With 60% of phones on Android, FCM sees ≈ 7.3K/s at peak, ≈ 437K a minute against FCM's default quota of 600,000 messages a minute per project: inside it, but at 73%, so we alarm at 80% and request an increase before growth. (FCM also limits a single Android device to 240 messages a minute and 5,000 an hour; our coalescing stays far below that.) APNs publishes no fixed per-app rate; we watch its throttling responses.

DynamoDB writes, counted honestly (write units per message, average):

$$5{,}787 \times 5.87 \approx 34.0\text{K write units/s on average} \qquad 20{,}255 \times 5.87 \approx 119\text{K at peak}$$

The peak is far above DynamoDB's default quotas (40,000 write units a second per table; 80,000 per account for provisioned tables), so we request increases before launch.

Reads. 450M app opens a day × ≈ 4 units (an inbox query and a page or two of catch-up) = 1.8B, plus history scrolling ≈ 0.3B: ≈ 2.1B a day ≈ 24.3K units/s on average. For an AZ loss, 3.33M phones reconnect through 220 surviving gateways admitting 300 a second each (66,000/s, about 50 s); at ≈ 0.8 read units per reconnect that's ≈ 53K extra read units a second, which we provision as a floor during the 4 evening peak hours with scheduled scaling.

Media (step 2.4; 50M files a day, 500 KB average):

Monthly cost (us-east-1 list prices, 730 hours and 30.4 days a month; rounded)

About 0.56 cents per daily user a month. Media is 47% of the bill ($78.3K of ≈ $168.3K), and most of that is CloudFront delivering photos and videos. Gateways are 21%, DynamoDB 20%. Two things people expect to be expensive aren't: heartbeats (their bytes cost a few hundred dollars) and binary-vs-JSON (the saving is about 319 GB a day, under $1K a month). What they cost is paid on the phone, in battery.

Sending pushes through Amazon SNS instead would add about 300\text{M} \times 30.4 \times \1.00 per million (publish plus delivery) ≈ **\9.1K a month**.

R2.7 Trade-Offs

Socket or push, by app state.

Heartbeat interval vs battery vs detection time.

We don't pick detection time by heartbeat alone: network-change callbacks, send timeouts and the server's delivery deadline all shorten it.

Binary vs JSON frames.

The money is small. We choose binary for the socket because it is typed, versioned by field numbers, and cheap to parse; REST APIs stay JSON.

Push through SNS vs directly. SNS mobile push hides the provider protocols and token handling for ≈ $9.1K a month at our volume. We send directly: at 300M pushes a day the saving pays for the code, and we control collapse IDs, priorities and push types per message. A smaller team should use SNS (it supports these headers too).

Media retention. 30 days on the server keeps storage at 750 TB (≈ $16.3K a month); keeping everything would grow by 760 TB a month, forever. The price is "no longer available" on old media for users who lost their phone, which Round 3's backups address.

R2.8 Failure Modes

Drill: WebSocket reconnect thundering herd (the storm is answered in the "Reconnect storm" row above; WebSockets vs SSE over HTTP/2 in R1.8)

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Uses the OS push service as the background path and knows each platform's rules: iOS alerts vs background pushes, Android high priority vs deprioritization, force-quit and force-stop.
• Defines "delivered" as the phone's acknowledgement, and escalates on a deadline rather than sending more pushes.
• Explains why sockets die (NAT timeouts, the NLB idle timeout), treats interval values as tunable assumptions, and reacts to network-change callbacks.
• Does the battery arithmetic in wake-ups, and shows that a background socket alone would break the budget.
• Moves media to resumable direct-to-S3 uploads run by the OS, and cleans up orphans without racing attaches.
• Turns receipts into forward-only cursors and counts them per recipient.
• Re-checks carried numbers: receipts per recipient, heartbeat rates blended correctly, gateway headroom per AZ, and finds that media dominates the bill.

Follow-up questions

1. "Typing indicators?" Answer: foreground only, over the socket, never stored and never pushed. At most one every few seconds per user, sent only to members who have the chat open, and the indicator expires on the receiving side after about 5 seconds unless refreshed.

2. "The user sends a photo with no signal. What does the recipient see, and when?" Answer: the sender sees the photo at once, from the local file, marked sending. The upload manager queues the upload; the message waits in the outbox, behind its upload, because a media reference is only valid once the upload is COMPLETE. When the network returns, the OS runs the upload, then the outbox sends the message. The recipient sees the message with its inline thumbnail immediately, and the full image when their phone downloads it.

3. "Why not have the Notification Service Extension fetch everything the user missed?" Answer: it has about 30 seconds and a tight memory limit, and it runs once per notification. It should do one small, bounded job (write the message in the payload, or fetch one), and leave a full catch-up to the app when the user opens it.

Interview gotchas from this round

---

Round 3 · Architect · "End-to-End Encrypted, Multi-Device, and Restorable"

~45 min · Principal (L7) · priced as one home region, 3 AZs (geography as in the chat loop's Round 3) · 100M DAU, ~2.2 active devices each · 1.7B messages/day, ≈ 69K/s peak · ≈ 7.5 encrypted envelopes per message · 40M open sockets at peak · 99.99%

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run a messenger for 30 million daily users: 10 million open sockets at peak, 500 million messages a day, about 20,000 a second at peak, 1.9 recipients per message. On the phone, the screen reads a local SQLite database through one writer; sends go through an outbox with client-generated IDs. The socket is open only in the foreground, with adaptive heartbeats (60 s Wi-Fi, 180 s cellular as tunable starting points) and reconnects on network-change callbacks in three round trips. In the background, OS push is the path: iOS alerts with a Notification Service Extension, Android high-priority data messages that always end in a notification. Delivery counts only when the phone acknowledges it; a deadline escalates to a notification the OS shows by itself, collapsed per conversation. Media goes straight to S3 in resumable 8 MiB parts through the OS's background transfers, and the message carries only a checked reference. Receipts are forward-only cursors, and each message has one ordered delivery state. The battery cost is about 0.3% a day. About $168K a month, almost half of it media delivery. Open costs: the server can read every message, a user has exactly one device, a lost or replaced phone loses its history, and a lost phone's database is protected only by the OS."

Architecture v2, compact

Synthesizing vector architecture diagram...

Socket in the foreground, push in the background, media on its own path, everything landing in one local database.

Steps so far

Open costs: we can read every message; one device per user; no restore; keys and data protected only by the OS.

R3.1 The Scope Raise

Interviewer: "We're growing to 100 million daily users, and we're making every conversation end-to-end encrypted: we must be unable to read messages, even if ordered to or breached. People use a phone, a tablet and a desktop, all in sync. A new phone must restore history. During outages, messages will arrive out of order and late. A lost phone must not expose chats. And notifications must still show who wrote and what, without our servers seeing the text."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Device registration and keys. Each device creates its own key pairs and uploads only the public halves:

httpCopy
POST /v1/devices HTTP/1.1
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "device_name": "Priya's iPad",
  "identity_key": "base64…",
  "signed_prekey": { "id": 17, "public_key": "base64…", "signature": "base64…" },
  "signed_pq_prekey": { "id": 4, "public_key": "base64…", "signature": "base64…" },
  "one_time_prekeys": [ { "id": 5001, "public_key": "base64…" } ],
  "one_time_pq_prekeys": [ { "id": 901, "public_key": "base64…", "signature": "base64…" } ],
  "device_list_update": { "version": 8, "devices": ["d_1", "d_2", "d_3"], "signature": "base64…" }
}

The device list update is signed by an existing device of the same account, which approved the new one (the user scanned a QR code on it). The server checks the signature and the version before storing the list, and never accepts a device list for another user.

Fetching keys to start a session (one bundle per device of the target user; each fetched one-time prekey is removed so it's used once). Because every fetch consumes keys, bundle fetches are rate-limited per requesting account and device (a few dozen new users an hour, with bursts for contact syncs), so nobody can drain a victim's one-time prekeys on purpose. Only the device that owns a key set can upload or replace its prekeys; the server checks the authenticated device, not a field in the body:

httpCopy
GET /v1/keys/u_802/devices HTTP/1.1
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "device_list": { "version": 8, "devices": ["d_1", "d_2", "d_3"], "signature": "base64…" },
  "bundles": [ { "device_id": "d_1", "identity_key": "…", "signed_prekey": { … },
                 "signed_pq_prekey": { … }, "one_time_prekey": { "id": 5001, … },
                 "one_time_pq_prekey": { "id": 901, … } } ] }

A sealed send: one message, one envelope per device (field table; bytes are estimates for a short text):

Each envelope as delivered, with the routing header the server adds (seq, server_msg_id, sender user and device, sent_at), is about 300 bytes in binary. The same thing in JSON, with UUID strings and the sealed part base64-encoded, is about 590 bytes. (The often-quoted "800 B vs 160 B" counts this kind of frame, but without padding the ciphertext; our estimate pads, which hides message length at the cost of bytes.)

If the envelope set doesn't match the recipient's current devices, the server refuses the whole send:

httpCopy
HTTP/1.1 409 Conflict
Content-Type: application/json

{ "error": "DEVICE_MISMATCH", "user_id": "u_802", "device_list_version": 9,
  "missing_devices": ["d_4"], "extra_devices": ["d_2"] }

The sender's device fetches bundles for d_4, drops d_2, re-encrypts and retries with the same client_msg_id.

Encrypted backups: POST /v1/backups/snapshots returns pre-signed URLs for an encrypted snapshot; POST /v1/backups/increments for a daily increment; GET /v1/backups/latest lists what exists. The server stores opaque bytes per account and checks that the caller owns the account. If the user chose a PIN-protected backup key, the key vault has two calls: PUT /v1/vault/entry (store the wrapped backup key with a PIN-derived verifier) and POST /v1/vault/recover (guess-limited).

Revoking a device: DELETE /v1/devices/d_2 from another signed-in device, carrying a device list update without d_2, signed by that device. The server verifies that d_2 belongs to the caller's account before doing anything.

Safety numbers are computed on the devices, and must cover every device identity key of both users (so adding a device changes the number), unless the account uses one identity key shared by its linked devices, as Signal does. The server only serves the public keys.

R3.4 Design Evolution: Keys, Devices, Backups and Lost Phones

Step 3.1: We Must Not Be Able to Read Messages

Step 3.2: Messages Arrived Out of Order and Some Won't Decrypt

Step 3.3: Phone, Tablet, Desktop

Step 3.4: Restore History on a New Phone

Step 3.5: A Lost Phone

Primitive: OAuth2, OIDC & Distributed Token Authentication · Drill: JWT revocation token blacklist

Step 3.6: Notification Previews Without Us Reading the Text

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Keys are created and used only on the left. The right side stores public keys, routes ciphertext, and keeps backups and media it can't read. The only secret the server side guards is the vault's wrapped backup keys, and those are useless without the user's PIN and the enclave's guess limit.

The server's tables (DynamoDB):

Envelopes are deleted when their device acknowledges them. Envelopes for a device that never comes back carry a 30-day ttl attribute for cleanup. DynamoDB deletes expired items only eventually (typically within a few days), so readers also skip any envelope whose expires_at is in the past by our servers' clocks.

Trace 1: the first message to a new contact (Alice → Bob, who has 3 devices)

Synthesizing vector architecture diagram...

One network trip to the key directory per new contact, then normal sends.

Trace 2: a restore on a new phone

Synthesizing vector architecture diagram...

History comes from the backup; sessions are always new.

R3.6 Numbers and Cost

Traffic (100M DAU × 17 messages; peak 3.5× average)

Gateways (Round 2's rule: two AZs hold everything, plus 10%): $40\text{M} \div 2 \div 50{,}000 = 400$ per AZ, × 1.1 = 440 per AZ, 1,320 tasks.

DynamoDB writes per message. Every envelope is written once and deleted once when acknowledged (1 unit each, as envelopes are about 400 bytes with attributes):

$$19{,}676 \times 15.66 \approx 308\text{K units/s on average} \qquad 68{,}866 \times 15.66 \approx 1.08\text{M at peak}$$

Each mailbox is its own partition key, so the per-partition limit (1,000 write units a second) only matters for a single device, far above what one device receives. The table and account quotas must be raised far ahead of launch.

Key directory. Assume each daily user starts sessions with 0.5 new contacts' users a day (new contacts, new devices, reinstalls): $100\text{M} \times 0.5 \times 2.2$ devices = 110M bundle fetches a day (≈ 1,270/s). Each takes one classical and one post-quantum one-time prekey (conditional deletes; the post-quantum key is about 1.6 KB, so 2 units), and the owner later uploads replacements: ≈ 6 write units per fetch, ≈ 660M a day ≈ 7.6K units/s on average. Storage: 220M devices × (50 classical one-time keys × ~100 B + 50 post-quantum ones × ~1.7 KB) ≈ 19.8 TB, most of it post-quantum keys.

Device-list checks. Every send is checked against the recipients' current device lists, cached by version in ElastiCache: the recipients' lists and the sender's own (for its other devices), ≈ 68,866 × (1.9 + 1) ≈ 200K lookups/s at peak.

The routing cache (per second at peak): registry refreshes 40M ÷ 60 ≈ 667K; device-list lookups ≈ 200K; delivery deadlines ≈ (259K live envelope deliveries + 50.8K Android pushes) × 2 (added and removed) ≈ 620K; connects and disconnects ≈ 122K (15 opens per user a day); cursors ≈ 60K. ≈ 1.67M operations a second → at 80K per shard that is 21 shards with no headroom, so we run 24 shards (≈ 70K each at peak), 48 × cache.r7g.xlarge.

Pushes. Phones and tablets receive 5.38 × (1.6 ÷ 2.2) ≈ 3.91 message envelopes per message; with Round 2's 55% not in the foreground and 57% kept after coalescing: $1.7\text{B} \times 3.91 \times 0.55 \times 0.57 \approx$ 2.09B pushes a day (≈ 24.2K/s average, ≈ 84.7K/s peak). FCM's share (60%) is ≈ 50.8K/s ≈ 3.05M a minute: five times FCM's default project quota of 600,000 a minute. We must have the increase approved before launch. Through SNS these would cost ≈ $63.5K a month; we send directly.

Media (10% of messages, 500 KB average, encrypted on the device with a random per-file key that travels inside the message):

Encrypted media can't be recompressed, thumbnailed or deduplicated by us: the sender's device makes the thumbnail and puts it inside the message.

Backups (assume 60% of daily users turn them on; text-only backups average 15 MB encrypted; media backup is out of scope):

Monthly cost (us-east-1 list prices; rounded)

About 0.79 cents per daily user a month. Media is 41% of the bill ($327.4K), envelope writes 26%, gateways 18%. The server can't read anything, yet it does more work than before: every message becomes 7.5 stored and deleted envelopes.

Levers, in order:

1. Reserved capacity for the steady part of the provisioned DynamoDB capacity (a one-year commitment; this table is single-Region, so it qualifies): a large cut on the $209K write line at list prices.
2. Media auto-download policy: don't auto-download video on tablets and desktops. Cutting downloads from 3.54 to 2.5 per file saves roughly a quarter of the CloudFront line.
3. Mailbox cursors instead of deletes (R3.7): about $100K of writes, traded for about $38–44K of storage, a race to handle, and ciphertext kept longer.
4. Fewer post-quantum one-time prekeys per device: most of the key directory's storage.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we make chat feel live on a device that keeps going to sleep, losing its network and getting killed? The answer is now:

• Live: the local database is the truth for the screen; a message is written before it's shown, and the network only fills and drains the database.
• Asleep: a socket only in the foreground; the OS's push connection in the background; delivery counts only when the device says so, and deadlines escalate to notifications the OS shows by itself.
• Losing its network: an outbox with IDs made before the first attempt, catch-up by cursor, resumable uploads the OS runs, and reconnects triggered by the OS's network callbacks.
• Killed, lost or replaced: everything important is durable on the device, encrypted with a key that never leaves it; a new device is a new endpoint with new sessions, restored from a backup whose key the user holds.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6

Counting decrypt failures by distinct device pairs matters: one broken session retrying a hundred times is one problem, not a hundred, and a spike in pairs is what a real bug looks like.

Procedure: a push-provider outage REL 11 · OPS 10

1. Confirm it's the provider. Send failures by provider and error code (CLI 1), and the provider's status page. If only one of our push workers' connections fails, recycle the workers instead.
2. Don't retry harder. Push workers already back off per provider; raising concurrency adds to the provider's load. Check that the queue is growing and coalescing (its size should be bounded by users × conversations, not messages).
3. Tell users what still works. Foreground apps and desktops still receive over sockets. Mailboxes hold everything.
4. When the provider recovers, the queue drains coalesced: one push per conversation with the newest up_to_seq, never a burst of stale notifications. Watch FCM quota use during the drain.
5. Afterwards: compare time-to-DELIVERED during the outage with the alarm, and review whether escalations behaved.

Procedure: a decrypt-failure spike SEC 10 · OPS 10

1. Slice it. By app version, platform, message type (prekey or normal), and whether the recipient restored or reinstalled recently (CLI 2).
2. One app version? Halt its rollout; if it's live, disable the new feature by remote flag. A client bug that corrupts session state can't be fixed server-side, but the decryption-error repair recovers each session as the sender's device retries.
3. Prekey messages only? Check the key directory: bundles served with a wrong signature, a one-time key served twice (the conditional delete must make that impossible), or a stale device list cache. Invalidate the device-list cache (versions make stale entries detectable).
4. Treat it as a possible attack if failures cluster on key changes for particular users: check device-list change rates and signatures, and involve the security team.
5. Afterwards: a post-incident review; add the failing sequence to the client's protocol test suite.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace names and times with real ones.

textCopy
# 1. Push send failures by provider (custom metric from the push workers)
aws cloudwatch get-metric-statistics --namespace Chat/Push --metric-name SendFailures --dimensions Name=Provider,Value=FCM --statistics Sum --period 60 --start-time 2026-09-28T18:00:00Z --end-time 2026-09-28T19:00:00Z

# 2. Decrypt failures by app version (custom metric from client telemetry)
aws cloudwatch get-metric-statistics --namespace Chat/Client --metric-name DecryptFailurePairs --dimensions Name=AppVersion,Value=8.14.0 --statistics Sum --period 300 --start-time 2026-09-28T18:00:00Z --end-time 2026-09-28T19:00:00Z

# 3. Mailbox table write throttling
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions Name=TableName,Value=Mailboxes --statistics Sum --period 60 --start-time 2026-09-28T18:00:00Z --end-time 2026-09-28T19:00:00Z

# 4. Scale push workers
aws ecs update-service --cluster chat --service push-workers --desired-count 60

# 5. Confirm the media bucket's lifecycle rules (abort incomplete uploads, 30-day expiry)
aws s3api get-bucket-lifecycle-configuration --bucket chat-media-prod

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Distinguishes "encrypted" from "end-to-end", and explains session setup and the Double Ratchet at the level of what each key does, without hand-waving.
• Bounds skipped keys and explains why both "drop" and "keep everything" are wrong; handles duplicate delivery paths and cross-process state.
• Treats every device as an endpoint, verifies device lists, and counts the per-device fan-out.
• Makes backup recovery an explicit key-custody decision, keeps sessions out of backups, and names the history that can be lost.
• Knows what secure hardware can and can't hold, and the lock-state trade-off notifications force.
• Prices the system and sees that the server does more work, not less, after it stops reading messages.

Follow-up questions

1. "Can we detect spam if we can't read messages?" Answer: partly. Metadata signals (new accounts sending to many strangers, fan-out rates, reports) work without content, and a reported message arrives decrypted from the reporter's device. What we can't do is scan every message; that's the product decision end-to-end encryption makes.

2. "A user has 5 devices and joins a 256-member group. What happens?" Answer: each of their devices needs the group's sender keys from every member device, and every member device needs theirs: one sender-key message from each of their 5 devices to up to $255 \times 2.2 \approx 560$ member devices, over pairwise sessions, plus fetching bundles for any devices they haven't talked to. That's a burst of thousands of small envelopes on a join, which is why very large groups favour MLS.

3. "Why not store message history on the server, encrypted with a key all the user's devices share?" Answer: that's a legitimate design (some apps do it): a per-user history key, distributed to each new device by an existing one. It gives new devices history without a backup. The costs: one key that decrypts everything a user ever had, held on every device, and server-side storage of all history. We chose device-held history with backups; a product that needs seamless history on new devices might choose the other.

Interview gotchas from this round

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: what must work offline, what happens when the app is in the background or killed, and who is allowed to read the messages?"
• When the scope is raised: "Here's what breaks, and I'll fix it in this order: anything that can lose or duplicate a message, then anything that drains the battery or depends on the OS behaving, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels