Design a Chat and Instant Messaging System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Chat System?

You Already Use One: a Phone Line That Stays Open

When you make a phone call, the line stays open. Either side can talk at any moment, and nobody has to redial to hear the answer. A chat app works the same way. Each app keeps a connection open to our servers, so when someone sends you a message, we can push it to you straight away instead of waiting for your phone to ask.

Every loop before this one was request and response: the client asks, the server answers. Chat is the first loop where the server speaks first.

Four things make up the whole product:

What Makes It Hard

• Millions of open connections spread over hundreds of servers. Bob's phone is connected to exactly one of them. To deliver to Bob, we have to know which one, right now, and that answer changes every time his phone switches from Wi-Fi to 4G.
• Order with no gaps. If Bob sees message 1 and message 3, he must be able to tell that message 2 exists and fetch it. If he sees them in the wrong order, the conversation reads as nonsense.
• People are offline most of the time. A message to an offline person must wait safely, trigger a notification on their phone, and appear in order when they return.

The Question the Whole Loop Answers

How do we get a message to every device of the right people, in order, instantly if they're online and reliably if they're not?

The answer gets sharper every round:

• Round 1: one open connection per user, a registry that says which server holds it, a durable log per conversation with a gap-free sequence number, and catch-up by cursor.
• Round 2: everything multiplies: group members, devices, receipts, presence. The fix is to store once and deliver many times, to turn receipts into cursors, and to survive a million phones reconnecting at once.
• Round 3: geography, encryption and giant channels change what the server is even allowed to know. Each conversation gets a home region, the server carries sealed envelopes, and huge channels switch from push to pull.

---

Round 1 · Mid-level · "1-on-1 Chat for a Small App"

~35 min · SDE II (L5) · 1 region, 3 AZs · 1M DAU · ~100K concurrent connections · ~700 messages/s peak · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Our second-hand marketplace wants buyers and sellers to chat inside the app. Design it." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Groups.
• Presence ("online", "last seen").
• Read receipts ("seen").
• More than one device per user.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into a requirement:

Not yet: groups, presence, receipts, multiple devices.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Latency. How long from Alice pressing send to the message leaving our servers towards Bob? We measure it inside our system, from the moment Alice's frame reaches us to the moment Bob's frame leaves us, because a phone on a weak mobile network can add hundreds of milliseconds we don't control.
• No lost messages once "sent" is shown. When Alice's app shows the single tick, the message must survive any single server crash. This is the rule that shapes step 1.3.
• Order within a conversation. Everyone sees the messages of a conversation in the same order, and can tell if one is missing.
• Connection scale. How many open connections at peak, and how many can one server hold? This sizes the fleet.
• Availability. If chat is down, buyers and sellers can't close deals. 99.9% allows about 43.8 minutes of downtime a month.

R1.4 The API

The app speaks two protocols: a WebSocket for live traffic, and plain HTTPS for everything that is a normal request (creating a chat, reading history).

A WebSocket is a connection that starts as an HTTP request and then "upgrades" into a two-way channel over the same TCP connection. After the upgrade, either side can send a message (a frame) at any time.

1. Connect. Browsers can't add an Authorization header to a WebSocket request, so the app first gets a short-lived, single-use connect ticket over normal HTTPS:

httpCopy
POST /v1/ws-ticket HTTP/1.1
Host: chat.example-market.com
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "ticket": "wst_4Kq9x2", "expires_in_s": 30 }

Then it opens the socket with that ticket:

httpCopy
GET /v1/ws?ticket=wst_4Kq9x2 HTTP/1.1
Host: chat.example-market.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

The gateway checks the ticket once and throws it away. A ticket in a URL can end up in a log, which is why it lives 30 seconds and works once.

2. Frames over the socket (JSON; IDs are strings because 64-bit numbers don't fit exactly in JavaScript):

Alice sends:

jsonCopy
{
  "type": "SEND_MESSAGE",
  "client_msg_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "conversation_id": "c_7Hk2",
  "body": { "text": "Is the bike still for sale?", "image_url": null }
}

The server replies to Alice once the message is stored:

jsonCopy
{
  "type": "ACK",
  "client_msg_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "conversation_id": "c_7Hk2",
  "server_msg_id": "718293847561029384",
  "seq": 10482,
  "sent_at": "2026-09-26T18:04:11.135Z"
}

Bob receives:

jsonCopy
{
  "type": "RECEIVE_MESSAGE",
  "conversation_id": "c_7Hk2",
  "server_msg_id": "718293847561029384",
  "seq": 10482,
  "sender_id": "u_401",
  "body": { "text": "Is the bike still for sale?", "image_url": null },
  "sent_at": "2026-09-26T18:04:11.135Z"
}

If the message can't be accepted, Alice gets a NACK with the same client_msg_id and a reason (NOT_A_MEMBER, TOO_LARGE, RATE_LIMITED).

Why a client message ID? Alice's phone sends a message, then goes into a lift. The ACK never arrives. Did the server store it or not? The phone can't know, so it sends the message again. The client_msg_id is a random ID the phone generated before the first attempt, and it stays the same on every retry. The server uses it to recognise the second copy and answer with the first copy's seq instead of storing it twice. Without it, every weak signal would produce duplicate messages.

3. Create a conversation (idempotent: the ID of a buyer-seller chat about one listing is derived from the three IDs, so a second tap returns the same chat):

httpCopy
POST /v1/conversations HTTP/1.1
Authorization: Bearer <access_token>
Content-Type: application/json

{ "listing_id": "l_88", "participant_id": "u_802" }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "conversation_id": "c_7Hk2", "members": ["u_401", "u_802"], "created": false }

4. Read history and catch up.

httpCopy
GET /v1/conversations/c_7Hk2/messages?after_seq=10481&limit=50 HTTP/1.1
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "messages": [
    { "seq": 10482, "server_msg_id": "718293847561029384", "sender_id": "u_401",
      "body": { "text": "Is the bike still for sale?" }, "sent_at": "2026-09-26T18:04:11.135Z" }
  ],
  "has_more": false
}

after_seq reads forwards (catch-up); before_seq reads backwards (scrolling up through history). Both are cursors: they say where you are in the conversation, so a page never shifts when new messages arrive.

Recap

• A WebSocket per user for live traffic; HTTPS for creating chats and reading history.
• Three frames: SEND_MESSAGE (with a client message ID), ACK (server ID and sequence), RECEIVE_MESSAGE.
• History and catch-up by sequence cursor.
• "Sent" means stored. No gaps, same order for both people.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From Polling to a Pushed, Ordered Log

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

Store messages in a database. Every few seconds, each app calls GET /v1/conversations/{id}/messages?after_seq=... for its open chats and shows anything new.

Synthesizing vector architecture diagram...

What's good about it: it is plain HTTP, every server is stateless, and nothing new is needed. What it costs us: a message waits on average half a polling interval before Bob sees it, almost every poll comes back empty, and a phone that wakes its radio every few seconds drains its battery.

Step 1.1: Polling Is Slow and Costly

Primitive: WebSocket, SSE & Long Polling

Step 1.2: Alice Is on Gateway 3, Bob Is on Gateway 7

Primitive: Distributed Cache Patterns & Eviction

Step 1.3: A Message Was Shown as Sent, Then Lost

Primitive: Distributed Unique ID Generators · Loop: Design a Distributed Unique ID Generator

Step 1.4: Messages Appear in the Wrong Order

Step 1.5: Bob Was Offline for a Day

Step 1.6: A Gateway Restarted and Dropped 11,000 Connections

Drill: WebSocket reconnect thundering herd

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Follow Alice's message. It enters through the NLB to her gateway, which hands it to the message service. The service claims the next sequence number and stores the message and its idempotency record in one transaction, acknowledges Alice, looks Bob up in the registry and calls his gateway directly. Independently, the table's change stream triggers a Lambda function that moves the conversation to the top of both inboxes and, if Bob has no registry entry, sends a push notification through SNS. If the call to Bob's gateway fails (NOT_HERE or an error), the message service deletes the stale entry and requests the push itself. Pushes from either path are deduplicated per (recipient, conversation, seq) with a short-lived create-if-absent key in the registry cache, so Bob gets one.

The pieces:

• Network Load Balancer (NLB) with a plain TCP listener. It passes the encrypted bytes through, and each gateway terminates TLS itself. (An NLB TLS listener would decrypt for us, but it bills far more capacity units per open connection: R2.6 shows why that matters.) The NLB's TCP idle timeout (350 seconds by default) is far longer than our 30-second heartbeat, so live sockets are never cut for being idle.
• Gateways on ECS (EC2 launch type, Graviton), spread over three AZs.
• Message service on ECS: stateless.
• ElastiCache (Valkey) for the registry: one primary and one replica in another AZ.
• DynamoDB table ChatTable, on-demand capacity, with one global secondary index for the inbox.
• DynamoDB Streams → Lambda for work that must happen after every stored message but isn't on Alice's critical path: updating inboxes and sending push notifications. A stream here is DynamoDB's ordered log of every change to the table; Lambda reads it and retries on failure, so each change is processed at least once.
• SNS mobile push to APNs and FCM.

The table design (one table, ChatTable; one GSI, a second copy of chosen items that DynamoDB keeps in sync asynchronously, keyed differently):

GSI1SK is a Number, so it sorts as a number. Note that CMSG#, MEMBER# and META all sort below MSG# (C < M, and ME < MS), which is why the message range reads use BETWEEN MSG#<from> AND MSG#999999999999.

Access patterns:

• "Newest message" (to claim the next seq) → Query PK = CONV#c, SK BETWEEN MSG#000000000000 AND MSG#999999999999, newest first, Limit 1, strongly consistent.
• "Catch up after 10477" → the same range starting after MSG#000000010477, oldest first.
• "Scroll up" → SK < MSG#<before>, newest first.
• "My inbox, most recent first" → Query GSI1 where GSI1PK = USER#me, descending. Unread count = last_seq − last_read_seq, computed, never counted.
• "Is Alice a member?" → GetItem CONV#c / MEMBER#u_401 (cached briefly in the service).

The registry layout:

Why inbox updates are "set to the larger value", never "add one". The Lambda function may process a change twice (Lambda retries, and streams are read at least once). If it did unread_count = unread_count + 1, a retry would count one message twice, and the badge would say 5 when there are 4. Instead it writes last_seq = 10482 and last_activity_ms = … only if the stored last_seq is smaller. On the sender's inbox entry, the same update also sets last_read_seq = 10482 (again only if larger), so Alice never sees her own message as unread. A replay writes the same values, or is refused because they're already there. Unread is computed from two cursors, so it can't drift.

Trace 1: an online delivery

Synthesizing vector architecture diagram...

Alice's tick waits on one read and one transactional write. Bob's delivery happens right after, on the same service instance.

Trace 2: an offline delivery, then catch-up

Synthesizing vector architecture diagram...

Nothing was queued for Bob. His cursor and the log together say exactly what he's missing.

One gap we accept in Round 1. If the message service crashes after the transaction commits but before it calls Bob's gateway, an online Bob doesn't get the message in real time. It's stored and acknowledged, so it isn't lost: Bob sees it on his next catch-up, or as soon as the next message arrives with a higher seq and his phone notices the hole. Round 2 closes this by driving delivery from a durable log.

R1.7 Numbers

Traffic (assumptions: each daily user sends 20 messages a day; peak is 3× the daily average; 10% of daily users are connected at peak)

Gateways. Assumption, to confirm with a load test: one gateway task on a c7g.large (2 vCPU, 4 GiB) holds 25,000 connections comfortably. At this size the limit isn't CPU (700 messages a second over the whole fleet is nothing); we cap connections per task to keep a crash small. We want the fleet to survive losing an AZ: two AZs must hold 100K, so each AZ needs at least $100{,}000 \div 2 \div 25{,}000 = 2$ tasks. We run 3 per AZ, 9 tasks: about 11,100 connections each normally, 16,700 each after an AZ loss.

Storage. A message item averages about 600 bytes (a short text, an optional photo link, two IDs, a timestamp and attribute names); the idempotency record about 150 bytes.

$$20\text{M/day} \times 600 \text{ B} = 12 \text{ GB/day} \qquad 12 \times 365 \approx 4.4 \text{ TB after a year}$$

The idempotency records (20M × 150 B = 3 GB a day) expire after 7 days, so they hold steady at about 21 GB.

Write capacity, honestly counted. DynamoDB bills writes in write request units: 1 unit per started KB for a normal write, 2 per started KB inside a transaction. A GSI is billed separately for every change to its copy, and an update that changes the index sort key costs two index writes (remove the old entry, add the new one).

The message itself is 2 of those 12 units. The rest is bookkeeping, and that ratio will get worse in Round 2.

Availability budget. 99.9% of a 30.4-day month: $43{,}776 \text{ min} \times 0.001 \approx 43.8$ minutes.

Monthly cost (us-east-1 list prices, 730 hours a month; rounded)

Say the headline: the biggest line is DynamoDB writes, and most of that is bookkeeping, not messages. The gateways, the part everyone draws first, are about 6% of the bill.

R1.8 Trade-Offs

WebSocket vs SSE vs long polling. Covered in step 1.1: chat is two-way and chatty, so WebSockets. SSE would work for receiving, but every send would be a separate HTTPS request. Long polling stays as a fallback for networks that block WebSockets.

API Gateway WebSocket APIs vs our own gateway fleet. At this size, the managed option deserves an honest look. API Gateway WebSocket APIs hold the connections for us, route each incoming frame to a backend (a Lambda function or an HTTP endpoint), and give each connection an ID we can post frames to.

At 1M users, API Gateway is a legitimate choice: about $1,800 a month more, and nobody to page when a gateway misbehaves. We choose our own fleet because the business expects to grow into Round 2, where the 2-hour reconnect and per-message pricing start to dominate, and because owning the gateway lets us drain gracefully and resume by cursor exactly as we want. It is a judgment call; say so in the interview.

DynamoDB vs a Cassandra-style store for message logs. Both fit the shape "partition by conversation, sort by sequence, range-read the tail". Cassandra (or ScyllaDB) is cheaper per write at very large scale if you run it well, and Discord famously stores messages that way. DynamoDB gives us conditional writes (which our gap-free sequence relies on), three-AZ durability, and no cluster to operate. Cassandra's equivalent, a lightweight transaction, is a Paxos round and much slower than a normal write. At Round 1's size, the managed store wins easily.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Starts from polling, names its cost, and moves to a persistent connection with a reason.
• Explains how a message finds the recipient's server (a registry with expiry), and why broadcasting to every server doesn't scale.
• Stores before acknowledging, and makes retries idempotent with a client-generated ID.
• Separates a global message ID from a per-conversation sequence, and can say whether the sequence has gaps and why.
• Uses the log as the mailbox: catch-up by cursor, push as a nudge.
• Plans for reconnect storms with backoff and jitter.
• Derives traffic, gateway count and a monthly cost, and sees that writes, not connections, dominate.

Follow-up questions

1. "Why not have the gateway write to DynamoDB directly and skip the message service?" Answer: we could. We keep gateways as thin as possible because they hold state: every deploy of a gateway disconnects users, so we want to deploy them rarely. Business logic (membership checks, limits, sequencing) changes often and lives in a stateless service we can deploy any time without dropping a single socket.

2. "How does Alice's app show the message before the ACK arrives?" Answer: it shows it at once as "sending" (a clock icon), at the bottom of the conversation, keyed by client_msg_id. When the ACK arrives, it attaches the seq and moves the message to its place in the order (almost always where it already is). If no ACK comes after a timeout, it retries with the same ID; after several failures it shows "not sent, tap to retry".

3. "A photo is 3 MB. Does it go over the WebSocket?" Answer: no. The app uploads it straight to S3 through a pre-signed URL, then sends a normal message carrying the photo's URL. The socket carries small frames only, and photos are served to the other side through a CDN.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Groups, Presence, Receipts, Many Devices, 100M DAU"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 100M DAU · 20M concurrent sockets · ≈ 174K messages/s and ≈ 660K deliveries/s peak · 99.99% · delivery P99 < 100 ms

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built 1-on-1 chat for a marketplace: 1M daily users, about 100K open connections and 700 messages a second at peak, one region, three AZs, 99.9%. Each app holds a WebSocket to a gateway fleet behind an NLB, with a heartbeat every 30 seconds. A registry in ElastiCache maps each user to the gateway holding their socket, with a 90-second expiry refreshed by heartbeats, and deliveries are a direct call to that gateway. A stateless message service stores every message before acknowledging it: it claims the next per-conversation sequence number by writing MSG#<seq> with a 'must not exist' condition, in one DynamoDB transaction with an idempotency record keyed by the client's message ID. So the sequence is gap-free, and retries never duplicate. Offline users get a push through SNS to APNs or FCM, and catch up by cursor with one range read. A stream-triggered Lambda keeps each user's inbox sorted by activity, setting values to the larger one so replays can't double-count. Apps reconnect with jittered backoff and resume by cursor. About $7,600 a month, and DynamoDB writes, mostly bookkeeping, are the biggest line. Open costs: one delivery per message, one device per user, the read-then-claim loop won't survive a busy group, and if the service crashes after storing but before delivering, an online recipient only sees the message on their next catch-up."

Architecture v1, compact

textCopy
 app ──WebSocket──► NLB ──► gateways (9) ──► message service ──► DynamoDB ChatTable
                              │  ▲                 │   1. Query newest MSG# (strong)
          conn:<user> 90 s ◄──┘  │                 │   2. Transact: MSG#<seq> + CMSG#<client id>, "must not exist"
          (ElastiCache)          └── deliver ◄─────┘   3. ACK, then look up recipient, call its gateway
 DynamoDB Streams ──► Lambda: inbox "set if larger" (GSI1 by last activity); push via SNS if no registry entry
 catch-up: GET /messages?after_seq=… → Query SK BETWEEN MSG#<after+1> AND MSG#999999999999

Round 1 step summary

Open costs: groups, devices, receipts and presence don't exist yet; the sequence claim won't survive a busy group; delivery isn't driven by a durable log.

R2.1 The Scope Raise

Interviewer: "We're now a standalone messaging app. 100 million people use it daily, and at the evening peak 20 million sockets are open. People chat in groups of up to 1,000. They want to see who's online, and whether their message was delivered and read. Everyone has a phone, many also have a desktop app or a browser tab, and all of them must stay in sync. Losing an AZ must not become an outage. Delivery must feel instant: under 100 milliseconds at P99 inside our system."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

The "Not yet" list from R1.2 is now mandatory: groups, presence, receipts and multiple devices.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

Groups.

httpCopy
POST /v1/conversations HTTP/1.1
Authorization: Bearer <access_token>
Content-Type: application/json

{ "type": "GROUP", "name": "Bike club", "member_ids": ["u_401", "u_802", "u_913"] }

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{ "conversation_id": "c_9Jq1", "type": "GROUP", "member_count": 3, "membership_version": 1 }

POST /v1/conversations/{id}/members adds members and DELETE /v1/conversations/{id}/members/{user_id} removes one. Each change bumps membership_version, and the change is also stored as a system message in the conversation's log (so every member sees "Sam added Priya" in order with the chat).

Resume, per device. On connect, each device says what it has:

jsonCopy
{ "type": "RESUME", "device_id": "d_phone_1", "inbox_sync_token": "1767225600135", "open_conversation": "c_9Jq1" }

The server answers with the conversations that changed after the token (from the inbox index), and the device fetches after_seq for each.

Receipts are cursors, one frame per conversation, not per message:

jsonCopy
{ "type": "DELIVERED", "conversation_id": "c_9Jq1", "up_to_seq": 20417 }

jsonCopy
{ "type": "READ", "conversation_id": "c_9Jq1", "up_to_seq": 20417 }

The server tells the user's other devices, so their badges clear:

jsonCopy
{ "type": "READ_SYNC", "conversation_id": "c_9Jq1", "up_to_seq": 20417 }

and tells senders what happened to their messages (1-on-1 per recipient; groups as a summary):

jsonCopy
{ "type": "RECEIPT_UPDATE", "conversation_id": "c_9Jq1", "seq": 20415, "delivered_count": 21, "read_count": 12, "member_count": 25 }

Presence, only for people currently on screen:

jsonCopy
{ "type": "PRESENCE_SUBSCRIBE", "user_ids": ["u_802", "u_913"] }

jsonCopy
{ "type": "PRESENCE_UPDATE", "user_id": "u_802", "status": "OFFLINE", "last_seen": "2026-09-26T18:22:40Z" }

A new subscribe replaces the previous set, and the app sends one whenever the visible contacts change (at most 50 users, at most once a second).

Large-group activity nudge, and gap repair:

jsonCopy
{ "type": "NEW_ACTIVITY", "conversation_id": "c_big7", "latest_seq": 88210 }

jsonCopy
{ "type": "SYNC_GAP", "conversation_id": "c_9Jq1", "missing_seq": [20416] }

The inbox (a user's conversation list, most recent first):

httpCopy
GET /v1/inbox?limit=50 HTTP/1.1
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "conversations": [
    { "conversation_id": "c_9Jq1", "type": "GROUP", "name": "Bike club", "last_seq": 20417,
      "last_read_seq": 20409, "unread": 8, "last_activity": "2026-09-26T18:22:41Z" }
  ],
  "sync_token": "1767226961000"
}

R2.4 Design Evolution: Fan-Out, Ownership, Presence and Recovery

Step 2.1: A Message to a 1,000-Member Group

Primitive: Distributed Cache Patterns & Eviction

Step 2.2: The Busy Group's Sequence Claim Throttles

Primitive: Message Queues vs Event Streams · Primitive: Distributed Locks & Leases · Drill: Distributed lock fencing token · Loop: Design a Distributed Message Queue

Step 2.3: Presence Flaps and Floods Everyone

Primitive: Gossip Protocol & Failure Detection

Step 2.4: Sent, Delivered, Read, on Every Device

Step 2.5: An AZ Died and 6.7 Million Phones Reconnected at Once

Drill: WebSocket reconnect thundering herd · Primitive: Circuit Breaker, Bulkhead & Fault-Tolerance Patterns

Step 2.6: A Phone Got Message 20416 Before 20415

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Follow a message. The gateway publishes it to ingest, keyed by conversation, so one sequencer owns each conversation. The sequencer numbers it, stores it with a conditional put, acknowledges the sender and publishes it to sequenced. From there, independent consumers deliver it (per member for small groups, one publish for large ones), update inboxes and cursors, send pushes to offline members, and archive it to S3. The routing cluster holds everything short-lived: the per-device registry, presence, delivery cursors and the pub/sub channels.

The pieces added in this round:

• MSK: 12 brokers, 4 per AZ (sized in R2.6), replication factor 3, min.insync.replicas = 2. Topic ingest and topic sequenced, each with 600 partitions, both keyed by conversation ID. Retention 24 hours: the log is a pipeline, not the store.
• Sequencers, delivery workers, inbox updaters, archiver, presence service, push workers: ECS services on Graviton, all in three AZs.
• Two ElastiCache clusters, on purpose. The routing cluster (registry, presence, cursors, pub/sub) runs with maxmemory-policy noeviction and lots of spare memory: silently evicting a registry entry would make a user look offline. An eviction policy like volatile-ttl would be exactly wrong here, because it evicts the keys closest to expiring first, which are our 90-second registry entries. The recent-messages cache (the last 200 messages of each active large group) is disposable and runs allkeys-lru. When a reader finds a group's window missing and rebuilds it from DynamoDB, it creates the key only if it's still absent (SET … NX), so a slightly older snapshot can never overwrite a window the delivery worker has just extended with newer messages.
• Member lists cached in delivery workers, keyed by membership_version. The membership service writes the new list into the cache on every change, and a worker that refills after a miss only creates the entry if absent, for the same reason. A removed member stops receiving within about a second.

Trace 1: a message to a 25-member group

textCopy
Priya's phone ─► gw-112 ─► ingest (key c_9Jq1) ─► sequencer p-311:
    known client_msg_id? no → seq = 20418 → PutItem MSG#000000020418 (must not exist) ✓
    ─► ACK to gw-112 (seq 20418)            ─► publish to sequenced (key c_9Jq1)
    (later, off the critical path: META.last_seq ← 20418 "set if larger", at most hourly)
delivery worker (owns sequenced partition of c_9Jq1):
    members (25, cached, version 7) → pipelined registry read of 25 conn:<user> sets → 11 live devices
    ─► direct calls to 9 gateways ─► RECEIVE_MESSAGE on 11 devices (not Priya's phone; yes Priya's laptop)
    ─► 14 members with no live device → push queue (unless muted)
inbox updater: last_seq / last_activity "set if larger" on members' inbox items (coalesced; see R2.6)

Trace 2: presence debounce in a tunnel

textCopy
t = 0 s     Bob's last heartbeat reaches gw-40
t = 45 s    gw-40 closes the silent socket; Bob has no other device
            presence: pres:u_802 = OFFLINE_PENDING (last_seen = t0); offdue:<b> add u_802 at t0 + 45 s = now
t = 45.2 s  sweeper: u_802 due and still pending → OFFLINE → watchers' gateways (watch:u_802 = {gw-7, gw-301})
alternative: t = 20 s   Bob reconnects on gw-9 → pres:u_802 = ONLINE, due entry removed, nobody notified

Trace 3: an AZ loss

textCopy
t = 0       us-east-1b fails: 220 gateways and ~6.7M sockets gone
t = 0–2 s   apps notice (socket error or missed heartbeat) and wait random(0, 1 s) … random(0, 2 s) …
t = 1–60 s  440 gateways admit ≤ 300/s each (132K/s); the excess are closed at TCP accept, before TLS → back off
            each admitted app: RESUME → inbox changes since token → fetch after_seq for 0–2 conversations
t ≈ 60–120 s the last apps are back; presence never flickered for most (they reconnected inside 45 s)
meanwhile   MSK keeps 2 of 3 replicas; ElastiCache promotes replicas in the surviving AZs; DynamoDB unaffected

The key schema, complete for this round (still one table, one GSI):

The CMSG# idempotency item from Round 1 is gone: the sequencer's memory of recent client_msg_ids replaced it, which saves 2 write units per message and a transaction.

Primitive: Database Sharding & Partition Keys

R2.6 Numbers and Cost

Traffic (the current track's figures, re-derived)

(Multiple devices add to the deliveries: if an online recipient has 1.2 connected devices on average, that's about 790K frames a second at peak. We size on the carried 660K and keep the difference as margin.)

Gateways: the track's 400 has no headroom. The track assumes about 10 KB of memory per idle socket and caps each task at 50,000 sockets. Both are assumptions to load-test: 10 KB is optimistic for a TLS connection (a TLS library's buffers alone can be tens of KB per connection unless it releases them when idle). And 400 × 50,000 is exactly 20M: no room for a single failure, let alone an AZ.

Each task is a c7g.xlarge (4 vCPU, 8 GiB). Messages aren't the constraint: even after an AZ loss, 660K deliveries ÷ 440 ≈ 1,500 frames a second per task. The 50K cap is a choice about blast radius: a crashed task drops 0.25% of users, and a storm of 50K reconnects is easy to absorb.

The reconnect storm, sized. One AZ ≈ 20M ÷ 3 ≈ 6.67M sockets. 440 surviving tasks × 300 admissions/s = 132,000/s → at least $6.67\text{M} \div 132{,}000 \approx 51$ s. Catch-up reads while admitting at that rate: one inbox query each (≈ 0.5 read units) plus, for about 30% of apps, 1–2 conversation fetches (≈ 1 unit each): $132{,}000 \times (0.5 + 0.3 \times 1.5 \times 1) \approx 125{,}000$ read units a second, for about a minute. We provision a floor of about 125K spare read units during peak hours.

Presence traffic.

About a thousand times less work, and the half-second target only applies to 21K updates a second.

DynamoDB writes: the track's 174,000 WCU is too low. It counts one 1 KB write per message. It misses the item size rounding (a 1 KB body plus IDs and attribute names is just over 1 KB, and a write is billed per started KB, so most message items cost 2 units) and every index and bookkeeping write:

$$173{,}611 \times 7.5 \approx 1.30\text{M write units/s at peak} \qquad 57{,}870 \times 7.5 \approx 434\text{K on average}$$

So the honest peak is about 7.5 times the track's figure. Two consequences: the default table quota (40,000 write units a second per table) must be raised far ahead of launch, and the capacity mode matters for cost:

• On-demand: $434{,}000 \times 2.63\text{M s} \approx 1.14$ trillion write units a month × $0.625 per million ≈ $713K/month.
• Provisioned with auto scaling (target 70% utilization, so on average ≈ 620K units provisioned): 620{,}000 \times \0.00065 \times 730 \approx **\294K/month**. We choose this; the traffic is a smooth daily curve, which auto scaling follows well.

DynamoDB reads (assumptions: 20 conversation opens per daily user per day, ~30 KB each ≈ 4 read units eventually consistent; 10 inbox reads per user per day ≈ 1 unit): $(2\text{B} \times 4 + 1\text{B} \times 1) \div 86{,}400 \approx 104\text{K}$ units/s on average. Provisioned at 70% ≈ 150K, plus the 125K storm floor ≈ 275K units × $0.00013 × 730 ≈ $26K/month.

Storage and tiering. 5 TB a day (5B × 1 KB); over 5 years $5 \times 365.25 \times 5 \approx 9{,}131$ TB ≈ 9.13 PB. We keep 90 days hot in DynamoDB, and the archiver writes everything to S3 as it happens:

• The archiver (a consumer of sequenced) packs messages into compressed segment files of several MB, grouped by a hash of the conversation ID, and writes them to S3 every few minutes. It reads the log at least once, so a segment can contain a message twice; the monthly compaction job keeps one copy per (conversation, seq) and writes each conversation's archive manifest item (ARCH#<yyyy-mm>) pointing at byte ranges.
• Hot items expire through DynamoDB TTL at 90 days. Here TTL is the right tool: it's cleanup, its lateness only costs a few days of storage, and TTL deletes consume no write capacity. A daily job checks that every conversation-day is in the archive before its items' TTL date. The conversation's numbering survives the expiry because META.last_seq never expires (step 2.2).
• History older than 90 days is read with an S3 byte-range GET using the manifest. We store segments in S3 Glacier Instant Retrieval: millisecond access, $0.004 per GB-month, a retrieval fee ($0.03 per GB) and a 128 KB minimum billable object size and 90-day minimum, which our multi-MB, never-deleted segments don't mind. Text and JSON compress well; we assume 3×.

At the 5-year mark the gap is wider: 8.68 PB of old history is ≈ 2.9 PB compressed, ≈ $11.6K a month in Glacier IR, against about $2.28M a month for all 9.13 PB in DynamoDB. The "eleven nines" durability figure belongs to S3 (it's S3's design target for objects); DynamoDB doesn't publish a durability number in nines. It keeps three copies in three AZs and answers only after a majority has durably logged a write, and our promise ("no acknowledged message lost") comes from storing before acknowledging, not from a number of nines.

The routing cluster. Mostly operations, not memory (20M users' registry sets and presence entries are a few GB):

Planning figure (to load-test): ~100K pipelined operations a second per shard primary, kept under 80% → 36 shards, each a primary and a replica in another AZ: 72 × cache.r7g.xlarge.

MSK. Two topics each carrying ≈ 174 MB/s at peak, replication factor 3: ≈ 1 GB/s of broker writes at peak. We assume 12 brokers of kafka.m7g.2xlarge (4 per AZ) and confirm with MSK's sizing guidance and a load test. Replication traffic between brokers isn't charged by MSK, but client traffic across AZs is; we turn on rack-aware consumers (fetch from the replica in the same AZ) to cut most of it.

Monthly cost (us-east-1 list prices, 730 hours a month; rounded)

About 0.65 cents per daily user a month. Read the table from the top: DynamoDB is two thirds of the bill, and about half of that is bookkeeping (inbox bumps and cursors), not messages. The levers are coalescing harder and reserved capacity for the steady part of the write curve, not cheaper gateways.

Why a TCP listener and not TLS on the NLB? An NLB bills by capacity units, taking the largest of three dimensions each hour. For TLS listeners, one unit covers only 3,000 active connections (TCP: 100,000). 20M TLS connections would be about 6,700 units ≈ $29K a month in that dimension alone, against 200 units for TCP. Terminating TLS in our gateways costs some CPU, which they have spare.

R2.7 Trade-Offs

Per million connection-minutes, our fleet costs about \77{,}000 \div 526{,}000 \approx $0.15, against API Gateway's \0.25 before a single message is sent.

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Stores once and fans out delivery, with a size threshold and a reason for it.
• Knows DynamoDB limits are per partition, and why a growing sort key can't be split around; replaces a contended claim with a single owner and a conditional-write fence.
• Picks a log for latency, not by habit, and budgets the P99 by stage.
• Designs presence around churn: a debounce, change-driven writes, viewport subscriptions, and no reliance on best-effort expiry events.
• Turns receipts into monotonic cursors and handles multiple devices without races.
• Sizes for an AZ loss and its reconnect storm, and notices that the carried fleet size has no headroom.
• Recounts the carried write figure (item rounding, index writes, bookkeeping) and finds the real cost driver.

Follow-up questions

1. "Typing indicators?" Answer: they're presence's little brother: ephemeral, never stored, and only useful to people looking at the chat. The gateway forwards TYPING to the conversation's members who have it open (the same viewport idea), rate-limited to one every few seconds per user, and the indicator expires on the receiving side after about 5 seconds unless refreshed. Nothing goes through the sequencer or the table.

2. "A user is in 300 groups and comes back after a week." Answer: RESUME returns the conversations that changed since the device's token, from the inbox index, a page at a time. The app fetches the newest page of each conversation as it's opened, not all 300 up front. Badges come from last_seq − last_read_seq, so the list is right before any messages are downloaded.

3. "Why not assign the sequence number in the gateway and skip the ingest log?" Answer: the gateway that receives a message is whichever one holds the sender's socket, so many gateways would number the same conversation: that's Round 1's race again. The ingest log's partitioning is what gives each conversation one owner without us building a coordination service.

Interview gotchas from this round

---

Round 3 · Architect · "Global, Encrypted, and Huge Channels"

~45 min · Principal (L7) · 4 home regions, each with a standby region · ~500M DAU · ~100M sockets · ≈ 290K messages/s average · 99.99% per region · no acknowledged message lost to a region failure

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run messaging for 100 million daily users in one region: 20 million sockets at peak, 5 billion messages a day (about 174,000 a second at peak, 660,000 deliveries), 99.99%, delivery P99 under 100 ms. 660 gateways, sized so two AZs hold every socket, accept at most 300 new connections a second each, and apps reconnect with full jitter and resume by cursor. Gateways publish sends to a Kafka topic on MSK, keyed by conversation, so one sequencer owns each conversation: it numbers the message, stores it with a 'must not exist' conditional put (which also fences a stale owner), acknowledges, and publishes it to a second topic. Delivery workers push full messages to groups of up to 100 and a single pub/sub nudge for bigger ones; offline members get coalesced pushes. Presence is debounced 45 seconds and sent only to people who have you on screen. Receipts are cursors that only move forward. 90 days of history live in DynamoDB, older history in compressed segments in S3 Glacier Instant Retrieval. About $650K a month, two thirds of it DynamoDB, about half of that bookkeeping. Open costs: one region; the server can read every message; the largest group is 1,000; and nothing handles retention or residency."

Architecture v2, compact

textCopy
 app ─► NLB ─► gateways (660, ≤ 50K each; admission ≤ 300/s) ─► MSK ingest (key = conversation)
      ─► sequencer (one owner per partition): seq = last + 1 → PutItem MSG#<seq> "must not exist" → ACK
      ─► MSK sequenced ─► delivery (≤ 100: per-member lookups + direct calls | > 100: SPUBLISH conv:<id> nudge)
                       ─► inbox/cursor updaters ("set if larger")  ─► archiver → S3 Glacier IR (> 90 days)
 routing cluster (noeviction): conn:<user> per-device sorted set, presence + 45 s debounce, delivery cursors

Steps so far

Open costs: one region; the server reads everything; groups stop at 1,000; no residency or retention.

R3.1 The Scope Raise

Interviewer: "We're global: about 500 million daily users on every continent, chatting across continents. Direct messages and small groups must be end-to-end encrypted: we should be unable to read them. We're adding community channels with 100,000 or more members, mostly readers, like Discord. EU users' data has to stay in the EU. Workspace admins set retention, and legal can put a hold on data. And we must survive losing a whole region without losing a single message we've acknowledged."

We ask back before we design:

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Device keys for end-to-end encryption. Each device publishes a public identity key and a batch of one-time key packages (MLS's name for "the public material someone needs to add this device to a group"):

httpCopy
POST /v1/devices/d_phone_1/key-packages HTTP/1.1
Authorization: Bearer <access_token>
Content-Type: application/json

{ "identity_key": "base64…", "key_packages": ["base64…", "base64…"], "count": 100 }

httpCopy
GET /v1/users/u_802/key-packages HTTP/1.1
Authorization: Bearer <access_token>

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "devices": [ { "device_id": "d_phone_1", "key_package": "base64…" }, { "device_id": "d_laptop_2", "key_package": "base64…" } ] }

Each fetched key package is used once and removed.

A sealed message. The server sees who, where and when; never what:

jsonCopy
{
  "type": "SEND_MESSAGE",
  "client_msg_id": "4e0c1a7e-51f2-4b7e-9f3d-0d6a1c2b9e77",
  "conversation_id": "c_9Jq1",
  "sealed": { "protocol": "MLS", "epoch": 42, "ciphertext": "base64…" },
  "notify": { "mentions": ["u_913"] }
}

Channels.

httpCopy
GET /v1/channels/ch_42/messages?after_seq=551000&limit=50 HTTP/1.1
Authorization: Bearer <access_token>

jsonCopy
{ "type": "CHANNEL_OPEN", "channel_id": "ch_42" }

CHANNEL_OPEN and CHANNEL_CLOSE tell the gateway which channel is on screen, so live posts are sent only while it is.

Retention, holds and residency.

httpCopy
PUT /v1/workspaces/w_17/retention HTTP/1.1
Authorization: Bearer <admin_token>
Content-Type: application/json

{ "messages_days": 30 }

httpCopy
POST /v1/legal-holds HTTP/1.1
Authorization: Bearer <legal_token>
Content-Type: application/json

{ "scope": { "workspace_id": "w_17" }, "reason": "case-2031-044" }

Every user and workspace carries a residency attribute: "data_region": "EU" or "data_region": "ANY".

R3.4 Design Evolution: Home Regions, Sealed Envelopes, Channels and Disasters

Step 3.1: Alice in Tokyo Chats with Bob in Paris

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active

Step 3.2: Messages Must Be Unreadable by Us

Step 3.3: A Channel Has 150,000 Members, Mostly Readers

Drill: Caching hot product page

Step 3.4: EU Data Must Stay in the EU; Admins Set Retention

Step 3.5: A Region Is Gone

Drill: Replication multi-region consistency · Drill: Raft consensus split brain

Step 3.6: Should We Run Our Own Gateways?

Loop: Design a Distributed Rate Limiter

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Every home region is a Round 2 system plus three things: a forwarder that sends each frame to its conversation's home, message tables that don't acknowledge until a second region has the write, and data keys that let us make whole months unreadable. The global layer is thin: an anycast entry point and a directory of IDs.

Trace 1: a Tokyo–Paris message (conversation homed in eu-central-1)

textCopy
Alice (Tokyo app) ─► GA edge ─► ap-northeast-1 gateway ─► directory: c_9Jq1 home = eu-central-1
  ─► forwarder ─► (≈ 115 ms one way, AWS backbone) ─► eu-central-1 ingest (key c_9Jq1)
  ─► sequencer: epoch 7 active here? yes → seq 20419 → PutItem MSG#…20419 into ChatMessages-2031-03 (MRSC)
       └─ returns after Ireland or the Paris witness has it (≈ 10–25 ms)
  ─► ACK ─► (≈ 115 ms) ─► Alice: tick after ≈ 250–270 ms
  ─► sequenced ─► delivery: Bob's devices in Frankfurt (direct), Alice's laptop via the Tokyo relay

Trace 2: an E2E group message

Synthesizing vector architecture diagram...

Trace 3: a region failover (eu-central-1 lost)

textCopy
t = 0        eu-central-1 unreachable; its MRSC writes stop (no second region can confirm them, so none succeed)
t ≈ 1–5 min  alarms: health checks, delivery P99, sockets per region; the on-call confirms (R3.9)
             ownership: HOMESET#eu epoch 7 → 8, active = eu-west-1 (conditional update, MRSC)
t + seconds  eu-west-1 sequencers take the EU partitions; last seq = max(META.last_seq, newest MSG#), strong reads → continue at the next seq
             Global Accelerator already sends new connections to healthy endpoints (Ireland for most EU users)
t → +30 min  warm gateways scale out; admission paces reconnects; REST history and push already work
return       failback is planned: drain, epoch 8 → 9 with active = eu-central-1, rebuild caches

R3.6 Numbers and Cost

Traffic per home region (a planning split; each region planned for its own peak at 3× its average; messages counted by the conversation's home)

Gateways use Round 2's rule: $1.1 \times \text{sockets} \div 50{,}000 \times \tfrac{3}{2}$ (two AZs hold everything, plus 10%). ap-northeast-1 is exactly Round 2's system. Each standby region keeps a warm quarter of its home's gateways: 825 more.

Cross-region forwarding. Assume 10% of messages are in conversations with members in another region: 2.5B a day ≈ 28,900 a second on average. Each costs one forwarded send (~1 KB) plus remote deliveries (~1.5 KB on average): $2.5\text{B} \times 2.5 \text{ KB} \approx 6.25$ TB a day ≈ 190 TB a month × $0.02 per GB (the US and EU inter-region rate; Asia is higher) ≈ $3.8K.

Synchronous replication. Every stored message is written to the standby replica too (about 25 TB a day). DynamoDB doesn't charge cross-Region data transfer for global-table replication, in either mode; we pay for the replicated writes and the storage (below), not the bytes. The witness adds no storage or replicated-write charges (as AWS describes it; confirm on the pricing page). The same holds for the asynchronously replicated bookkeeping (inbox items, cursors, directory).

Channel read load. Assume 200M daily users open channels 5 times a day: 1B opens ≈ 11,600 a second on average, ≈ 35,000 at peak, each ≈ 20 KB from a regional recent cache: ≈ 700 MB/s (≈ 5.6 Gbps) at peak, ≈ 609 TB a month out. Live posts to open screens are small by comparison (step 3.3).

Storage per region and tier (hot = current + previous month, ~45 days on average, two MRSC replicas; archive a year in, compressed 3× for channel posts and not at all for E2E messages; DMs and groups are E2E by default (step 3.2), so we assume ~90% of messages are E2E and ~10% are channel posts; two copies, in the same legal area where one exists)

(The factor 0.93 is 0.1 × ⅓ for channel text + 0.9 × 1 for ciphertext.)

DynamoDB writes. Round 2's $294K a month was for 5B messages a day at 7.5 units each; 25B is 5× that, ≈ $1.47M. Every write is then applied again in a second region (the MRSC replica for messages, the MREC standby for the rest). We assume replicated writes cost about the same as local ones (confirm on the current price list): another ≈ $1.47M, ≈ $2.94M in total.

Monthly cost (list prices, us-east-1 rates where regional prices differ; other regions cost somewhat more; rounded; storage a year in)

About 1 cent per daily user a month. DynamoDB is about 72% of the bill (writes, storage and reads), and half of the write line exists only because every write lands in two regions. That's the price of RPO 0 and of standby state. The levers, in order:

1. Replicate only what can't be rebuilt. Inbox bumps (3 of the 7.5 units) can be recomputed from memberships and each conversation's latest seq. Keeping them regional saves 0.4 \times \1.47\text{M} \approx $590\text{K}$ a month; after a failover, inboxes are re-sorted in the background.
2. Reserved capacity (a one-year commitment on provisioned capacity) cuts the committed part roughly in half at list prices, but it can't be bought for replicated write capacity. It applies only to single-Region tables: for example the inbox tables kept regional after lever 1, not the MRSC message logs or the MREC standby copies.
3. Coalesce harder: read cursors once every few seconds per member instead of once a second.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we get a message to every device of the right people, in order, instantly if they're online and reliably if they're not? The answer is now:

• In order: one owner per conversation numbers every message, and a number exists only if its message does. In one region that owner is a log partition; across regions it's the conversation's home, fenced by a conditional write that every region checks against the latest version.
• Every device of the right people: store once; deliver per device through a registry that knows each device's gateway; for big audiences, publish once and let screens that are open pull.
• Instantly if online: a persistent socket, a budget for every stage, and presence and receipts that don't flood the path.
• Reliably if not: the log is the mailbox. A cursor says what's missing, a push says something is, and an acknowledgment is only given once the message would survive losing the region that took it.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per region OPS 8 · REL 6

Reconnect storm procedure REL 5 · OPS 10

1. Confirm the cause. Sockets per AZ (a whole AZ?), a gateway deploy in progress (stop it), or a client release with broken backoff (check the reconnect rate by app version).
2. Don't raise admission limits first. Check what's behind the gateways: DynamoDB read throttles, routing-cluster CPU, MSK. The admission cap protects them.
3. Add capacity to the surviving AZs if headroom is below 20% (CLI 3). New gateways take reconnects as they come up.
4. Watch the plateau. Admissions should run flat at the cap and the backlog should shrink; the last users are usually back within a couple of minutes.
5. Afterwards: check presence (a wave of false "offline" means the debounce was too short), and write the post-incident review.

Region failover procedure REL 13 · OPS 7

1. Confirm the region is impaired (health checks, MRSC errors, sockets), and decide to fail over. This is a human decision with a written threshold; automation prepares every step.
2. Dial the region's Global Accelerator endpoint group to 0% (CLI 5), so no new connections go there even if it half-recovers.
3. Flip ownership with the conditional update (CLI 6). Confirm standby sequencers report the new epoch.
4. Scale the standby's gateways and workers from warm pools (CLI 3); admission paces reconnects.
5. Verify: sends acknowledged, gap requests normal, push flowing, history readable.
6. Failback later, planned, in the reverse order.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the IDs and ARNs with real ones.

textCopy
# 1. Active connections on a gateway NLB (per minute)
aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/NetworkELB --metric-name ActiveFlowCount --dimensions Name=LoadBalancer,Value=net/chat-gw/0123456789abcdef --statistics Average --period 60 --start-time 2031-03-01T18:00:00Z --end-time 2031-03-01T19:00:00Z

# 2. Fan-out lag: delivery consumer group on the sequenced topic
aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/Kafka --metric-name MaxOffsetLag --dimensions "Name=Cluster Name,Value=chat-msk" "Name=Consumer Group,Value=delivery" "Name=Topic,Value=sequenced" --statistics Maximum --period 60 --start-time 2031-03-01T18:00:00Z --end-time 2031-03-01T19:00:00Z

# 3. Scale gateways (here: the standby region)
aws ecs update-service --region eu-west-1 --cluster chat --service gateways --desired-count 825

# 4. DynamoDB write throttling on this month's message table
aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions Name=TableName,Value=ChatMessages-2031-03 --statistics Sum --period 60 --start-time 2031-03-01T18:00:00Z --end-time 2031-03-01T19:00:00Z

# 5. Stop sending new connections to the impaired region (Global Accelerator's API is called in us-west-2)
aws globalaccelerator update-endpoint-group --region us-west-2 --endpoint-group-arn arn:aws:globalaccelerator::111122223333:accelerator/1234abcd-abcd-1234-abcd-1234abcdefgh/listener/0123abcd/endpoint-group/eu01 --traffic-dial-percentage 0

# 6. Flip conversation ownership for the EU home set (conditional on the current epoch)
aws dynamodb update-item --region eu-west-1 --table-name ChatControl --key '{"PK": {"S": "HOMESET#eu"}, "SK": {"S": "OWNER"}}' --update-expression "SET active_region = :r, epoch = :new" --condition-expression "epoch = :old" --expression-attribute-values '{":r": {"S": "eu-west-1"}, ":new": {"N": "8"}, ":old": {"N": "7"}}'

# 7. Confirm a message table's replicas and consistency mode
aws dynamodb describe-table --region eu-west-1 --table-name ChatMessages-2031-03

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Gives every conversation one home for ordering, and explains what goes wrong with multi-writer (last writer wins losing messages).
• Distinguishes "encrypted" from "end-to-end", picks a group protocol for a reason (MLS: one ciphertext per message and an ordered commit log we already have), and lists what the product loses.
• Changes the delivery model for huge audiences instead of scaling the old one.
• Treats residency and retention as placement and key management, including archives and backups, with holds that override.
• Turns "no acknowledged message lost" into RPO 0 and a concrete mechanism (synchronous replication before the ack), and knows its costs: latency, no TTL, no transactions, three regions.
• Prices the system per region, finds where the money goes, and makes build vs buy a numbers question.

Follow-up questions

1. "A user moves from Tokyo to Paris. Do their conversations move?" Answer: their connections move at once (Global Accelerator picks the nearest region). Their conversations' homes don't: moving a conversation means changing the region that numbers it. For a conversation that should move (say, all active members now live in Europe), a background job can migrate it: pause its sequencer, copy its log to the new home, flip its directory entry with a new epoch, resume. We only do it when it pays off in latency, never automatically on a single trip.

2. "Can we offer search in end-to-end encrypted chats?" Answer: on the device, yes: each device indexes what it can decrypt. Server-side search would need the server to read or index the text, which E2E rules out. The honest product answer is "search what's on this device", plus encrypted backups that restore a device's history and index.

3. "Why not put channels on MRSC too, and push to every member?" Answer: channel posts do go to the MRSC log (an acknowledged post shouldn't be lost either). What we don't do is per-member work: 150K deliveries and bookkeeping writes per post would cost more than the whole channel is worth, for members who mostly won't look today.

Interview gotchas from this round

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: what happens when the recipient is offline, how long do we keep history, and how many devices does a user have?"
• When the scope is raised: "Here's what breaks in the current design, and I'll fix it in this order: anything that can lose or misorder an acknowledged message, then anything that can take delivery down, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels