Design a Distributed Email Service

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is an Email Service?

You Already Know One: a Post Office, a Filing Cabinet and a Librarian

Think of three jobs in one building.

• The post office accepts letters from anyone in the world, checks they aren't dangerous, and hands them to the right person. It also sends your letters out, and when an address doesn't exist, it brings the letter back with a note.
• The filing cabinet keeps every letter a person ever received, sorted into folders: Inbox, Sent, Spam, Trash, and whatever folders they made.
• The librarian can find any old letter in a second: "the invoice from Acme last spring."

An email service is those three jobs, run for millions of people, on a network where anyone can send you anything. A few words we'll use all page:

What Makes It Hard

• The internet is hostile. Much of the mail that arrives at a big provider is spam, phishing or malware. Some of it is built to crash parsers: a zip file that unpacks to petabytes, or MIME nested a thousand levels deep.
• An accepted message may never be lost. Once our server says "250 OK" in SMTP, the sender deletes its copy. From that moment we are the only holder.
• Every mailbox must feel instant. Opening the inbox, the unread count, a thread, a search: all have to answer in tens of milliseconds, over a pile of mail that only grows.
• Sending is judged by others. Whether our mail reaches an inbox or a spam folder is decided by Gmail, Microsoft and Yahoo, based on our authentication and our reputation.

The Question the Whole Loop Answers

How do we accept, store, find and send mail reliably, for millions of mailboxes, when much of the internet is hostile?

The answer grows every round:

• Round 1: point MX at a managed receiver, store the raw message first and parse it later, keep a small metadata record per message apart from the bytes, and authenticate what we send with SPF, DKIM and DMARC.
• Round 2: run our own mail servers, store each attachment once, keep counters and threads exact, feed a search index from the database's change stream, parse hostile mail safely, and send to Gmail and Microsoft the way their rules require.
• Round 3: serve thousands of companies: tenant routing, retention and legal holds, anti-phishing, customer-held keys, EU residency, and importing ten years of old mail.

The notification loop covered the sending side of email as a customer of a provider (templates, sender reputation basics, bounces from SES). See Design a Notification System. Here we are the mail system: we receive from the internet, keep mailboxes, search them, and send as a first-class mail server.

---

Round 1 · Mid-level · "Hosted Email for One Company"

~35 min · SDE II (L5) · 1 region · 5,000 employees · 500K messages/day, 17.4/s at peak · 50 GB/day · inbox view < 500 ms · never lose accepted mail · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Acme Corp has 5,000 employees and wants to stop paying for a hosted mail product. Build them email on their own domain, acme.example." Before drawing anything, we ask.

Out of scope for this round: search at scale, conversation threading, many companies.

R1.2 Functional Requirements, Derived Step by Step

Not yet: search at scale, threading, many companies.

R1.3 Non-Functional Requirements: the Questions

We name each quality in words first; the numbers come in R1.7.

• Never lose an accepted message. SMTP makes this concrete: once a receiver answers 250, the sender forgets the message. So we answer 250 only when the message is safe on durable storage. This is the correctness bar of the round.
• Deliverability of what we send. A message that lands in a customer's spam folder has failed, even though no error appears anywhere. Our domain must prove who it is.
• Fast mailbox views. Opening the inbox should take well under half a second, no matter how many years of mail the folder holds.
• Security. Mail carries contracts and passwords. Encrypt it at rest and in transit, and keep one employee from reading another's mailbox.
• Availability: 99.9%. Mail tolerates short outages better than most systems: senders retry for days (R1.9). 99.9% is about 44 minutes a month.

R1.4 The API

Three interfaces touch the mailbox: SMTP from the internet, a REST API for the web app, and IMAP for mail apps.

1. SMTP: how a message arrives. Another company's mail server looks up our MX record, connects on port 25 and talks like this (C: is the sender, S: is our server; a typical exchange, and the exact greeting and extension lines vary by server):

textCopy
S: 220 inbound-smtp.us-east-1.amazonaws.com ESMTP
C: EHLO mail.partner.example
S: 250-SIZE 41943040
S: 250 STARTTLS
C: STARTTLS
S: 220 Ready to start TLS
   (TLS handshake; the client sends EHLO again inside TLS)
C: MAIL FROM:<bob@partner.example> SIZE=2641920
S: 250 2.1.0 Ok
C: RCPT TO:<alice@acme.example>
S: 250 2.1.5 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: (headers and MIME body, about 2.6 MB)
C: .
S: 250 2.0.0 Ok: queued as 0100019a2c1e...
C: QUIT
S: 221 Bye

Three things to notice. The recipient is checked at RCPT TO, before any bytes arrive, so mail for an unknown user can be refused cheaply. The three-digit code carries the meaning (2xx done, 4xx try later, 5xx give up), and the 2.1.5 style enhanced status code (RFC 3463) adds detail. And the final 250 after the dot is the moment responsibility passes to us.

2. What the message looks like inside (MIME, shortened)

textCopy
From: Bob Lee <bob@partner.example>
To: Alice Chen <alice@acme.example>
Subject: Q3 pricing
Date: Mon, 28 Sep 2026 09:14:02 +0000
Message-ID: <7f3a.1759050842@partner.example>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="b1"

--b1
Content-Type: multipart/alternative; boundary="b2"

--b2
Content-Type: text/plain; charset=utf-8
Hi Alice, the price sheet is attached.
--b2
Content-Type: text/html; charset=utf-8
<p>Hi Alice, the price sheet is attached.</p>
--b2--
--b1
Content-Type: application/pdf; name="q3.pdf"
Content-Disposition: attachment; filename="q3.pdf"
Content-Transfer-Encoding: base64
JVBERi0xLjcKJeLjz9MKMSAwIG9iago8PC9UeXBlL0NhdGFsb2cv...
--b1--

A message is a tree. Here the top part is multipart/mixed with two children: a multipart/alternative (the same text as plain and HTML) and a PDF. Binary attachments travel as base64, which turns every 3 bytes into 4 characters: a 2 MB PDF becomes about 2.7 MB of text.

3. Web API

httpCopy
GET /v1/folders/INBOX/messages?limit=50&cursor=eyJ0cyI6MTc1OTA1MDg0Mn0 HTTP/1.1
Authorization: Bearer <session token>

jsonCopy
{ "folder": "INBOX", "unread": 14, "total": 1250,
  "messages": [
    { "id": "01K6F3Q9M2ZC4T8RVKJ0W5H7NA", "from": { "name": "Bob Lee", "email": "bob@partner.example" },
      "subject": "Q3 pricing", "snippet": "Hi Alice, the price sheet is attached.",
      "received_at": "2026-09-28T09:14:02Z", "unread": true, "has_attachments": true, "size": 2641920 } ],
  "next_cursor": "eyJ0cyI6MTc1OTA1MDExMH0" }

• GET /v1/messages/{id} returns headers and body parts; GET /v1/messages/{id}/attachments/{n} streams one attachment.
• PATCH /v1/messages/{id} with { "unread": false } or { "folder": "Archive" } marks or moves it.
• POST /v1/messages/send sends:

httpCopy
POST /v1/messages/send HTTP/1.1
Authorization: Bearer <session token>
Idempotency-Key: 5b0d2f8e-3c41-4c55-9a1e-7f0e2b6d9c10
Content-Type: application/json

{ "to": ["dan@customer.example"], "cc": [], "subject": "Re: Q3 pricing",
  "text": "Thanks, Dan. Details below.", "in_reply_to": "<a91c@customer.example>",
  "attachment_upload_ids": ["up_3xk9"] }

The answer is 202 Accepted with the new message's id and its Message-ID, because delivery happens later (step 1.6). The Idempotency-Key is made by the web app once per click of "Send" and reused on retries, so a retried request can't send twice.

4. IMAP. Mail apps speak IMAP4rev2 (RFC 9051) to our IMAP server on port 993 (TLS). IMAP needs three things from our data model, and we build them in from the start:

• UIDs: every message in a folder has a number that only grows (UID 4812). An app asks "anything above UID 4812?" to find new mail.
• UIDVALIDITY: a number per folder that changes only if UIDs had to be reassigned; if it changes, the app throws away its cache for that folder.
• MODSEQ: with the CONDSTORE and QRESYNC extensions (RFC 7162), every change (a new message, a flag, a delete) gets a higher modification sequence, so an app can ask "what changed since modseq 90,211?" instead of re-reading every flag.

And IDLE (RFC 2177) lets an app keep a connection open and be told when new mail arrives.

Recap

• SMTP decides the contract: 250 after the dot means the message is ours; 4xx means "try later"; 5xx means "give up".
• A message is a MIME tree; attachments are base64, a third bigger on the wire.
• The web API lists folders by cursor and sends with an idempotency key; IMAP needs UIDs, UIDVALIDITY and MODSEQ from the data model.

R1.5 Design Evolution: From One Mail Server to a Managed Pipeline

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

One virtual machine runs an open-source mail server. It accepts SMTP on port 25, writes each message as a file on its disk (one directory per user, one file per message, the classic Maildir layout), serves IMAP from the same disk, and sends outbound mail straight to the internet.

It works for a demo. One disk holds the company's entire mail history, and one machine failure stops mail for everyone.

Step 1.1: How Does Mail Reach Us at All?

Step 1.2: Listing a Folder Reads Every Message File

Step 1.3: Parsing Slows Down Accepting Mail

Primitive: Message Queues vs Event Streams · Drill: The Warehouse System That Fell a Day Behind. Its two questions are the two above: a consumer that throws halfway (the visibility timeout returns the message, retries are idempotent, the DLQ catches poison messages) and why a queue instead of a direct call (it decouples the SMTP accept rate from the parse rate and absorbs the burst).

Step 1.4: Our Mail Lands in Spam Elsewhere

Step 1.5: Spam and Fake Senders Reach Our Inboxes

Step 1.6: The Recipient's Server Is Down

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

Inbound: SES owns the SMTP conversation, stores the raw bytes, and announces them. Everything after that is our idempotent parser, driven by a queue.

Synthesizing vector architecture diagram...

Reading and sending: both clients read the same metadata and bytes; sending goes through a queue to SES, and the results come back as events.

Tables

The mail table (partition key mailbox_id, sort key sk):

The changes table (partition key mailbox_id, sort key modseq, a Number so it sorts numerically): one item per change, {op: NEW | FLAGS | MOVE | DELETE, ids, folder}, with a TTL of 30 days (TTL deletes are eventual, typically within a few days; a reader never trusts an item's presence to mean it's recent, it compares modseq). IMAP's "what changed since modseq 90,211" is one query on this table.

The sends table (partition key idempotency_key): message_id, state, created_at.

The delivery transaction (one TransactWriteItems per new message, 5 items):

1. Put M#<ulid> if it doesn't exist (this makes a retried parse a no-op: the whole transaction is cancelled, even if the user has since moved the message or deleted it, because the locator outlives both).
2. Put F#INBOX#<ulid>.
3. Update S#INBOX: unread + 1, total + 1, uid_next + 1 (the old uid_next is the message's UID; the parser read it just before and makes the update conditional on it being unchanged, retrying on conflict).
4. Update HEAD: modseq + 1, conditional on the value the parser read.
5. Put the changes item for that modseq.

Tracing an inbound message

Synthesizing vector architecture diagram...

Everything after SES's 250 is ours. Alice's app hears about the message within about 30 seconds this round, because the IMAP server polls; Round 2 pushes.

Tracing a send that hits a temporary failure (numbered steps)

1. 10:02:00: Alice clicks Send. The API writes the MIME to S3, the Sent item (delivery_status: QUEUED), the sends record, and a message on send-queue: 202.
2. 10:02:01: the sender worker calls SES; SES accepts. Status SENT_TO_SES.
3. 10:02:03: customer.example's MX answers 421 4.3.2. SES reports DeliveryDelay; the Sent item shows "Delayed, still trying."
4. SES retries on its own schedule. At 10:19 the server is back and accepts: a Delivery event; the Sent item shows "Delivered."
5. Had the server stayed down for 14 hours, SES would have reported a Bounce, and Alice would find a non-delivery report in her inbox.

R1.7 Numbers

Targets

Traffic (assumptions: each employee receives 80 messages a day, 60 from outside and 20 from colleagues, and sends 20, half of them outside)

Storage (assumption: 100 KB per mailbox entry on average; each entry has its own raw object this round)

Inbox view chain (dependent steps add): TLS to the ALB and routing 5 ms, session check 2 ms, one DynamoDB query of 50 items 15 ms, building the JSON 8 ms, the internet round trip to the browser 80 ms: about 110 ms.

IMAP polling, sized. 5,000 mailboxes × about 2 connected apps each = 10,000 connections. Each IMAP server polls HEAD for its connections every 30 s: 10,000 ÷ 30 ≈ 333 reads a second, eventually consistent, 0.5 read units each: 14.4M read units a day. Cheap now; Round 2 has 20 million active users, and this becomes push.

Monthly cost at the end of year 1 (us-east-1 list prices, 730 hours a month, ignoring free tiers; check the AWS Pricing Calculator before quoting):

About $0.74 per employee a month. SES is 62% of the bill ($2,312), and receiving alone is 53%. Remember that ratio; it decides a build-or-buy question in Round 2.

R1.8 Trade-Offs

SES vs our own MTA

Where metadata lives

A relational database would also work at 17 messages a second. We pick DynamoDB because the access pattern is simple (by mailbox, sorted by time) and Round 2's 35,000 a second doesn't force a migration.

Parsing synchronously vs asynchronously

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Knows how mail finds a domain (MX with preferences) and what the SMTP reply codes mean.
• Says out loud that 250 transfers responsibility, and stores the message durably before answering.
• Separates small metadata from large immutable bytes, and writes bytes first.
• Parses asynchronously from a queue, idempotently, with a DLQ.
• Explains SPF, DKIM and DMARC as three different questions, and rolls DMARC out gradually.
• Separates temporary (4xx) from permanent (5xx) failures, and knows who retries what.

Follow-up questions

1. "Why not store messages in a relational database with the body in a column?" Answer: a message averages about 100 KB and can be 25 MB. Bodies in rows bloat the buffer cache, backups and replication, while the thing we query (folder, date, flags) is about 1 KB. S3 stores bytes for about $0.023 per GB-month; database storage costs several times that and is sized for random access we don't need.

2. "A colleague sends to 200 people inside Acme. How many copies?" Answer: this round, 200 metadata items and 200 raw objects, one per mailbox, written by our API without SMTP. At 50 GB a day nobody minds. Round 2 stores an attachment once and points to it.

3. "How does Alice's phone learn about new mail without polling?" Answer: this round it partly does poll: the IMAP server checks the mailbox's modseq every 30 seconds and tells apps in IDLE. For phones that aren't connected, a push through APNs or FCM. Round 2 replaces polling with a notification sent right after the transaction.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "100M Mailboxes, a Billion Messages a Day"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 100M mailboxes, 20M active each day · 1B messages/day, 34.7K/s at peak · about 83 PB stored after 3 years · new mail visible < 2 s P95 · folder view < 40 ms P99 · search < 300 ms P95 on recent mail · 99.99%

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built email for one 5,000-person company: 500,000 messages a day, 17 a second at peak, 50 GB a day. Mail arrives through an MX record at SES receiving, which stores the raw message in S3 and announces it; an idempotent parser behind an SQS queue writes about 1 KB of metadata per message to DynamoDB, with the folder's counters, the next IMAP UID and the mailbox's change number in the same transaction. Bytes first, metadata second. The web app and an IMAP server read the same data; IMAP apps sync with UIDs and modseqs, and learn about new mail from a 30-second poll. Outbound mail goes through a queue to SES, signed with DKIM, with SPF on a custom MAIL FROM domain and DMARC rolled out slowly; SES retries temporary failures for up to 14 hours and reports bounces as events. Inbound, we act on SES's authentication and malware verdicts and quarantine forgeries instead of bouncing them. About $3.7K a month, two thirds of it SES. Open costs: SES's per-message price at scale, one raw copy per recipient, no search or threads, and polling."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: a managed receiver, raw bytes in S3, metadata in DynamoDB, and a managed sender.

Round 1 step summary

Open costs: SES pricing at scale; a copy per recipient; no search, threads or push.

R2.1 The Scope Raise

Interviewer: "Congratulations, we're now a consumer webmail service: 100 million mailboxes and a billion messages a day, with a big morning surge. Attachments up to 25 MB, and the same newsletter PDF goes to hundreds of thousands of people. Users want conversations grouped, and search over years of mail. Last month an attacker sent a zip bomb that took our parsers down. And Gmail has started deferring our outbound mail."

Scope change

The "Not yet" list from R1.2 comes back: search at scale and threading are in scope now. Many companies wait for Round 3.

A note on the availability target: we promise 99.99% (4.4 minutes a month), not "five nines". 99.999% allows 5.26 minutes a year.

R2.2 What Breaks in the Round 1 Design

The order we fix it in: attachments first (2.1), because they decide how bytes are stored; then counters (2.2) and threads (2.3), which shape the metadata; then search (2.4), which reads everything before it; then the two hostile edges: inbound content (2.5) and outbound reputation (2.6).

R2.3 New Requirements and API Additions

Threads

httpCopy
GET /v1/threads/th_01K6F3Q9?limit=100 HTTP/1.1
Authorization: Bearer <session token>

jsonCopy
{ "thread_id": "th_01K6F3Q9", "subject": "Q3 pricing", "message_count": 4, "unread": 1,
  "messages": [ { "id": "01K6F3Q9M2ZC...", "from": "bob@partner.example", "unread": false },
                { "id": "01K6G1B7Y4TA...", "from": "alice@ourmail.example", "unread": false } ] }

The folder list now returns one row per thread with its newest message, count and unread count.

Search

httpCopy
GET /v1/search?q=from:acme%20invoice%20has:attachment&scope=all&limit=20 HTTP/1.1
Authorization: Bearer <session token>

jsonCopy
{ "results": [ { "id": "01JZ8...", "thread_id": "th_01JZ8...", "subject": "Invoice 4471",
                 "received_at": "2025-03-02T10:11:00Z", "highlight": "...your <em>invoice</em> for March..." } ],
  "searched": "recent", "older_pending": true, "next": "c2VhcmNoOjI" }

searched: "recent" with older_pending: true means the first page came from the hot index (the last 90 days) and the older tiers are still being searched; the client asks for next to get them (step 2.4).

Attachment download by signed URL

GET /v1/messages/{id}/attachments/2 checks that the message is in the caller's mailbox and that the attachment's malware verdict is clean, then answers 302 Found with a pre-signed S3 GET URL for the attachment's object, valid for 5 minutes, with response-content-disposition set to the attachment's original file name. The bytes never pass through our API servers. A verdict still pending gets 409 SCAN_PENDING and the client retries in a second.

Delivery status of a sent message

jsonCopy
{ "id": "01K6G1B7Y4TA...", "recipients": [
    { "address": "dan@customer.example", "status": "DEFERRED", "attempts": 3,
      "last_reply": "451 4.7.1 Greylisted, try again later", "next_attempt_at": "2026-09-28T10:52:00Z" },
    { "address": "eve@other.example", "status": "DELIVERED", "at": "2026-09-28T10:02:04Z" } ] }

Sync with change tokens (JMAP-style)

Every change to a mailbox already gets the next modseq (Round 1). We expose it to web and mobile clients the way JMAP (RFC 8620 and RFC 8621) does: the mailbox's current modseq is its state string, and a client with an old state asks what changed.

httpCopy
GET /v1/changes?since=90211&max=500 HTTP/1.1

jsonCopy
{ "old_state": "90211", "new_state": "90219", "has_more": false,
  "created": ["01K6H..."], "updated": ["01K6F3Q9M2ZC..."], "destroyed": ["01K5Z..."] }

If since is older than the 30 days of change log we keep, the answer is the JMAP error cannotCalculateChanges, and the client does a full resync. IMAP clients get the same data through CONDSTORE/QRESYNC: HIGHESTMODSEQ, CHANGEDSINCE and VANISHED.

Push. Web and mobile clients keep one connection open to our gateways (WebSocket or EventSource) and receive a tiny frame, {"state":"90219"}, when their mailbox changes; they then call /v1/changes. Phones that aren't connected get an APNs or FCM push. IMAP apps get EXISTS and FETCH updates in IDLE; RFC 2177 has clients re-issue IDLE at least every 29 minutes so servers don't time them out.

Recap

• Threads and search are new read APIs; search says which tiers it has covered.
• Attachments download by 5-minute signed URLs, only after a clean verdict.
• Clients sync by a change token (the mailbox modseq), pushed as a tiny "something changed" frame.

R2.4 Design Evolution: Hash-Addressed Attachments, Exact Counts, Threads and Search

Step 2.1: The Same 10 MB Attachment, Stored 300,000 Times

Step 2.2: Unread Counts Drift, or Need Scans

Primitive: Distributed Cache Patterns and Eviction for the cached counters in R2.5.

Step 2.3: Group Replies into Conversations

Step 2.4: Search Years of Mail in 300 ms

Primitive: Trie Data Structure and Inverted Index · Change Data Capture and the Outbox Pattern · Drill: The Dual-Write That Broke Search Consistency. Its first question is the dual write above (the database committed, the publish failed, the index drifted forever); the stream fixes it because the change and its record are the same write. Its second asks why tail a log instead of polling an outbox table every 500 ms: DynamoDB has no single, ordered outbox to poll unless every change also goes into one global index partition, which would become the hottest key in the system; polling also adds up to 500 ms of delay and costs a read each time, mostly finding nothing. The stream's records are read by Lambda at no charge and arrive in order per item.

Step 2.5: A Zip Bomb Crashed the Parsers

Step 2.6: Gmail Is Deferring Our Mail

Primitive: Distributed Rate Limiting · Drill: The Partner Whose Retry Loop Took Down Everyone Else. Its two questions are the ones in point 2: local counters on many instances let through many times the limit, so the buckets are shared in Valkey; and a token bucket beats a fixed window because a window allows a double burst at its boundary.

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The inbound path: our MTAs answer 250 only after the spool write and the queue entry; everything after is asynchronous, and the search index follows the database's own stream.

Synthesizing vector architecture diagram...

The read path: metadata from DynamoDB with a cache checked against modseq, bodies by range from packs, attachments straight from S3.

Synthesizing vector architecture diagram...

The outbound path: the MIME is written once to the spool; the MTAs pace themselves per destination, park 4xx failures in minute buckets, and send failure notices back through the normal inbound pipeline.

The inbound MTAs, in order of what they check. Each check sits as early as it can, so the cheap refusals happen before the expensive work.

1. At connect, before TLS. Per-IP connection limits, and a DNS blocklist lookup on the connecting IP (cached). A listed IP gets 554 and the connection closes before any TLS handshake: the handshake is the most expensive thing a spammer can make us do, so we shed load in front of it.
2. At MAIL FROM. SPF for the envelope domain.
3. At RCPT TO. Is the recipient one of our 100M mailboxes? A cached directory answers; unknown users get 550 5.1.1. Over-quota mailboxes get 452 4.2.2, which tells the sender to retry later.
4. At the end of DATA, before answering. DKIM and DMARC. A message that fails DMARC for a domain publishing p=reject gets 550 5.7.1: rejected inside the conversation, so the sender's own server tells its user, and we send no backscatter. A message over 35 MiB was already refused during DATA.
5. Store, then answer. Each MTA collects the messages that finish in a 50 ms window and writes them as one spool object (a group commit), then sends one SQS entry per message, pointing at its byte range. Only then does each sender get 250. If the S3 write fails, every message in that window gets 451 4.3.0; if a queue entry fails, that message gets 451 and its bytes in the spool are simply never parsed (the sender's retry brings a fresh copy, and the spool expires after 7 days).

We also publish an MTA-STS policy (RFC 8461) so that senders who support it refuse to deliver to us without valid TLS, and our outbound MTAs honor other domains' MTA-STS policies.

Duplicates the internet sends us. If our 250 is lost on the network, the sender retries and we accept the message twice. The parser writes a dedup item D#<SHA-256 of the body>#<Message-ID> in the mailbox, only if it doesn't exist, in the same transaction as the message; the second copy's transaction fails, and the parser just re-sends the push for the first. The dedup items live 7 days (a TTL, deleted eventually), which covers any realistic retry by a sender.

Tables (DynamoDB, provisioned with auto scaling; storage-heavy tables in the Standard-IA table class)

The delivery transaction for one new message (one TransactWriteItems, 7 items plus two per attachment):

1. Put the locator M#<ulid> if it doesn't exist (it outlives moves and deletes as a tombstone, so a redelivered queue message can never bring a message back), and put F#<folder>#<ulid>.
2. Put the dedup item D#... if it doesn't exist.
3. For each attachment: put its refs item, and a condition check that its blobs item is LIVE.
4. Update S#<folder>: unread + 1, total + 1, uid_next + 1, conditional on the uid_next the parser read.
5. Update HEAD: modseq + 1, conditional on the value read; put the changes item.
6. Update T#<thread>: count, unread, newest date.

Before it, the parser claims the ID-map items (step 2.3). After it: the notify router, and nothing else. The search index learns from the stream.

The body packs. Parsers don't write one object per message body: each parser task collects the bodies it produced in the last 500 ms into one pack object (about 660 KB on average), writes it, and only then commits the metadata that points into it (bytes first). Reading a body is an S3 ranged GET of offset to offset + length. Section R2.6 shows why: at this scale, the number of objects costs more than their bytes.

Compacting packs safely. Deleted mail leaves dead bytes in packs, so a monthly job rewrites packs whose live fraction has fallen under 70%. A pack's bytes are referenced from many messages, so the job follows the same rules as the attachment collector (step 2.1):

1. Every pack has a manifest: the (mailbox, ulid) of each body in it, written with the pack.
2. The compactor resolves each manifest entry through its M# locator to the current F# item. Only entries whose item still exists and still points into this pack are live; everything else is dead bytes.
3. It writes the live bodies into a new pack (registered LIVE in packs), then moves each pointer in a transaction: update the F# item's pointer only if it still points at the old pack, with a condition check that the new pack is LIVE.
4. It sets the old pack DOOMED (only if LIVE). Anything that needs the old pack again (a point-in-time restore of the tables, for example) sets it back to LIVE first.
5. Just before the physical delete, it resolves the whole manifest again. If any item still points at the old pack, it returns the pack to LIVE. If none does, it sets DELETED (only if still DOOMED with the doomed_at it wrote), and deletes the object.
6. That delete happens 35 days after doom, the same window as point-in-time recovery, so a restored table never points at a pack we've already removed. About 30% of packs are rewritten once, so retired packs waiting out their 35 days add about 210 TB (30% × 20 TB a day × 35 days), counted in R2.6.

The recent-headers cache. Valkey keeps the first page (50 rows) of each active folder and the folder counters, each stamped with the mailbox modseq it was built from. The API reads HEAD.modseq from DynamoDB (one small, strongly consistent read) and uses the cached page only if the stamps match; otherwise it queries DynamoDB and refills. A stale cache entry can never be served, and a slow refill that writes an older page is simply ignored by the next reader.

Pushing new mail. After the transaction, the parser calls the notify router with (mailbox, modseq). The router looks up which gateway tasks hold that mailbox's connections in a Valkey registry and sends each a frame; it also asks SNS mobile push to wake phones without a connection. Each gateway refreshes its registry entries every 5 minutes (a 12-minute TTL): 15M connections ÷ 300 s = 50,000 registry writes a second, sized into the Valkey cluster. A parser that retries after a crash re-sends the push; clients ignore a state they've already seen, and phone notifications carry the message's ULID as their collapse ID (apns-collapse-id on Apple, collapse_key on Android), so a re-sent push replaces the first instead of showing twice.

Outbound deliveries hold a lease. An MTA worker claims a delivery by setting lease_owner and lease_until = now + 15 min only if the lease is free or lease_until < now − 30 s, where now is the worker's own clock and the 30 s absorbs clock differences between workers (DynamoDB conditions have no server time). A delivery to a slow server can take minutes (RFC 5321 lets a client wait up to 10 minutes for the reply after the final dot), so the worker extends the lease every 5 minutes; at 10,416 deliveries a second lasting about 2 seconds, about 21,000 are in flight at peak, and only the rare slow ones ever extend. If the worker times out after sending the final dot but before reading the reply, it can't know whether the message was delivered; retrying may deliver it twice. We retry: a duplicate beats a lost message.

The send API's idempotency is the durable item in sends from Round 1 (conditional on the key not existing, kept 7 days), not SQS: a FIFO queue's deduplication window is only 5 minutes. A retried request that finds its record in state ACCEPTED (the first attempt crashed before enqueueing) enqueues it now; a delivery worker's attempt claim makes an extra enqueue harmless.

Tracing an inbound message, end to end

Synthesizing vector architecture diagram...

About 1.2 seconds from the final dot to the phone showing the message (R2.6), most of it the body-pack window. The scanner's verdict on 9c1f arrives separately and unlocks the download.

Tracing a search

1. Alice types invoice acme. The API sends one query to the hot indices for the last 90 days (4 monthly indices), with routing set to her mailbox: 4 shards, one per index, searched in parallel.
2. The top 20 message IDs come back in about 120 ms. The API reads their metadata with one BatchGetItem and their text by ranged GETs, in parallel, and builds highlights.
3. It answers with searched: "recent" and older_pending: true, about 170 ms after the request arrived.
4. Alice scrolls. The next request searches the 33 older monthly indices in UltraWarm, one shard each; segments not in the warm cache come from S3, so this page takes about 1 to 2 seconds.

Tracing a greylisted send (numbered steps)

1. 10:02:00: Alice sends to dan@customer.example. The API writes the MIME to the spool, the Sent item, the sends record and a deliveries item, and enqueues it: 202.
2. 10:02:01: an MTA claims the delivery (attempt 1), takes a token from the customer.example bucket for the established pool, connects from 198.51.100.23, and gets 451 4.7.1 Greylisted, try again later.
3. It writes the deferred item due#10:07#17 (5 minutes, with jitter) and sets the delivery to DEFERRED, attempt 2.
4. 10:07: the scheduler reads bucket 10:07 and enqueues it. An MTA claims attempt 2 and connects from 198.51.100.41, another IP in the same /24 pool, which the greylister tracks as one sender. 250. The Sent item shows "Delivered".

Losing an AZ. MTAs, parsers, scanners and gateways run in three AZs; DynamoDB, S3, SQS and Lambda are regional. The NLB stops sending connections to the lost AZ's MTAs; senders whose connections broke retry. Sections R2.6 and R2.8 size the survivors.

R2.6 Numbers and Cost

Traffic

Message size and bandwidth

Storage after 3 years

Where it sits (lifecycle rules by age):

Two rules to know. Lifecycle transitions skip objects smaller than 128 KB by default (since September 2024), and Standard-IA and Glacier Instant Retrieval bill any object as at least 128 KB. That's why bodies go into packs: a 20 KB body alone would stay in Standard forever. Attachment objects average about 263 KB (55.3 KB per message ÷ 0.21 new attachment objects per message), so most of them move; the ones under 128 KB stay in Standard, which makes our transition count below an upper bound.

Search index

The default quota is 80 data nodes per domain; 87 needs an increase (the limit for this instance family is 400 on OpenSearch 2.17 and later). The 102 UltraWarm nodes fit the default of 150. Weekly indices would keep the hot tier close to 90 days and within 80 nodes, at the price of about four times as many indices for an old-mail search to touch; we keep monthly indices and raise the quota.

Metadata writes (transactions cost 2 write units per item)

29.7 × 10⁹ = 29.7B write units a day = 343,750 a second on average, about 1.03M at peak. That's far above the default 40,000 per table, so the quotas are raised (they're adjustable) and the tables pre-warmed. A single DynamoDB partition serves up to 1,000 write units a second; a mailbox receiving more than about 60 messages a second (a mailing-list archive under attack, say) gets 451 at RCPT TO until it slows.

Fleets, with per-AZ rounding

The parsers are sized for peak across three AZs, not for surviving an AZ at peak. Losing one leaves 116 tasks: 464 vCPUs, 23,200 messages a second. At peak the queue then grows by about 11,500 a second until replacement tasks start (a few minutes), so new mail runs a few minutes late while none is refused. The MTAs, which decide whether mail is refused, are the fleets sized for two AZs.

Cache sizes. Recent headers: 20M active mailboxes × 50 rows × 1 KB = 1 TB. Folder counters for every mailbox: 100M × 4 folders × 16 bytes = 6.4 GB.

Latency budgets (P95 unless noted; dependent steps add, parallel steps count once as their maximum)

A message becomes searchable about 1.5 to 2.5 seconds after it's visible: the stream's own delay (AWS publishes no bound; we measure it), up to 250 ms for Lambda's poll, a 40 ms GET, a 100 ms bulk request, and up to 1 s for OpenSearch's refresh.

Availability. 99.99% of a 30.4-day month is 43,776 min × 0.0001 ≈ 4.4 minutes.

Monthly cost at the 3-year point (us-east-1 list prices, decimal GB, 730 hours; at this size real contracts are negotiated, so read these as proportions):

About $0.024 per mailbox a month. Keeping data is most of the bill: S3 storage, DynamoDB storage and backups, and the search index together are about 60%. Object counts are the next surprise: request charges are $295K a month, 71% as much as all the bytes S3 stores. The levers we already pulled: dedup and lifecycle tiers ($2.28M → $0.42M for S3 bytes), packs and group commits instead of one object per message (R2.7), and indexing 2 KB per message instead of 10 KB with a replica.

R2.7 Trade-Offs

Search index layout

At a much larger scale, per-mailbox files on object storage become attractive (the index lives next to the mail and costs S3 prices), but they mean building a search service. We start managed.

Counters: exact vs reconciled

Our own MTAs vs SES, at this scale

The instances are almost free; the real cost of our own MTAs is the team and the reputation work. Against $5.6M a month of SES, that's easily paid for. SES still has a place: new, unwarmed pools could overflow to it, and Round 3 revisits it per company.

One object per message vs packs

The cost of packs is a delete that isn't immediate: a deleted message's body bytes stay in its pack until the monthly compaction. We say so in the privacy policy, and Round 3's legal requirements build on it.

Point-in-time recovery. It's 42% of the DynamoDB bill ($327K). We keep it: flags, folders, thread membership and references exist nowhere else, and a bad deploy that corrupts them needs a restore to a minute. The alternative, rebuilding metadata by re-parsing 83 PB, would take weeks and still lose every flag and folder move.

R2.8 Failure Modes

Primitive: Circuit Breaker, Bulkhead and Fault Tolerance Patterns · Drill: The Slow Recommendation Service That Took Down Checkout. Its two questions are the OpenSearch rows above: why a slow optional dependency exhausts the caller's threads, and why aggressive timeouts alone are worse than a breaker.

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer shows

• Runs its own MTAs at this scale and knows why (cost, control), and puts cheap checks before expensive ones in the SMTP conversation.
• Stores attachments by hash, builds a collector that can't race a new reference, and checks that every reference belongs to the caller.
• Keeps counters exact with conditional transactions and repairs them only when nothing changed.
• Threads by headers with placeholders for missing parents, and knows cycles are a read-time problem.
• Feeds search from the change stream, never by dual writes, and sizes the index honestly (what's indexed, hot vs warm).
• Parses hostile input as a stream with limits, and isolates scanning.
• Treats 4xx as "later" with a schedule, handles greylisting, paces per destination with a shared limiter, separates IP pools, and warms IPs up.

Follow-up questions

1. "A user deletes a message with a 20 MB attachment that 50,000 others also received. What happens to the bytes?" Answer: only that user's reference item goes. The hash lands on the candidate queue; the collector finds 49,999 references and leaves the file LIVE. The user's body bytes stay in their pack until the next compaction. When the last reference goes, the file is doomed, re-checked an hour later, deleted by generation, and kept 35 days as a non-current version.

2. "Why not deliver to Gmail as fast as it will accept?" Answer: Gmail decides its acceptance rate partly on our history; bursts raise deferrals and spam placement for everyone in the pool. We keep a steady pace per destination, raise it slowly while signals are clean, and cut it quickly when deferrals rise.

3. "A message arrives with References to a thread in someone else's mailbox." Answer: it can't join it: the ID map lives in each mailbox's own partition. At most it joins a thread in the recipient's own mailbox, which is what a legitimate reply does.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Business Email for Many Companies Worldwide"

~45 min · Principal (L7) · 4 regions: a home and a standby in the US and in the EU · 10M seats in 25,000 companies · 1B messages/day · retention, eDiscovery and legal holds · anti-phishing · EU residency · customer-held keys · imports · survive a region loss

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run consumer webmail for 100 million mailboxes: a billion messages a day, 34,700 a second at peak, 83 PB stored after three years. Our own inbound MTAs check cheap things first (blocklists before TLS, recipients at RCPT TO, DMARC before answering), group-commit each 50 ms of mail into one S3 spool object, queue one entry per message, and only then answer 250. Parsers stream each message with hard limits, write bodies into 660 KB packs and attachments once per SHA-256, and commit one DynamoDB transaction per message: the message, its counters, its IMAP UID, the mailbox's change number, the thread summary and a reference item per attachment. A collector deletes an attachment only after it's been doomed, re-checked, and found unreferenced. Threads come from Message-ID and References through a per-mailbox ID map. Search is OpenSearch fed from DynamoDB Streams, routed by mailbox: 90 days hot, the rest in UltraWarm. Clients sync by change tokens and get pushed a tiny frame. Outbound, our MTAs retry 4xx for 5 days on a greylist-friendly schedule, pace each destination with shared token buckets, and keep reputations apart in IP pools. About $2.39M a month, most of it keeping data. Open costs: one tenant, deletion with no legal holds, filters tuned for consumers, one region, and no way to bring in old mail."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: store, answer, parse; metadata transactions; an index that follows the stream; our own senders.

Round 2 step summary

Open costs: tenants; legal deletion rules; phishing aimed at businesses; residency and keys; imports.

R3.1 The Scope Raise

Interviewer: "We're launching business email. Companies bring their own domains and their own admins. Their legal teams need retention rules, eDiscovery and legal holds. Their executives are being phished. European customers need their mail kept in the EU, and some want to hold their own encryption keys. And our first big deal is a 5,000-person company moving over with ten years of mail."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Adding a domain

httpCopy
POST /v1/admin/domains HTTP/1.1
Authorization: Bearer <tenant admin token>
Content-Type: application/json

{ "domain": "acme.example" }

jsonCopy
{ "domain": "acme.example", "status": "PENDING_VERIFICATION",
  "records": [
    { "type": "TXT", "name": "acme.example", "value": "mailcloud-verify=5b1f0c2e9a" },
    { "type": "MX",  "name": "acme.example", "value": "10 mx.eu-central-1.mailcloud.example" },
    { "type": "MX",  "name": "acme.example", "value": "20 mx.eu-west-1.mailcloud.example" },
    { "type": "TXT", "name": "acme.example", "value": "v=spf1 include:_spf.mailcloud.example -all" },
    { "type": "TXT", "name": "mc2026a._domainkey.acme.example", "value": "v=DKIM1; k=rsa; p=MIIBIjANBg..." },
    { "type": "TXT", "name": "_dmarc.acme.example", "value": "v=DMARC1; p=none; rua=mailto:dmarc@acme.example" },
    { "type": "TXT", "name": "_mta-sts.acme.example", "value": "v=STSv1; id=20260928" } ] }

The domain stays PENDING until our checker sees the TXT token in public DNS; only then do we accept or send mail for it. A domain already verified by another tenant can't be claimed without the old tenant releasing it (or a support process with proof of ownership). The two MX records point at the tenant's home region first and its standby second (step 3.4).

Retention and legal holds

jsonCopy
{ "policy_id": "p_finance7", "scope": { "groups": ["finance@acme.example"] },
  "keep_for_days": 2555, "then": "PURGE", "user_deletes": "HIDE_ONLY" }

httpCopy
POST /v1/admin/holds HTTP/1.1
Content-Type: application/json

{ "hold_id": "h_2291", "matter": "Case 2026-114",
  "scope": { "mailboxes": ["dana@acme.example", "raj@acme.example"],
             "received_between": ["2024-01-01", "2026-09-28"] } }

DELETE /v1/admin/holds/h_2291 releases it. A hold has no expiry; only a release ends it.

eDiscovery

POST /v1/admin/ediscovery/searches with { "query": "\"project falcon\" AND from:*@rival.example", "custodians": [...], "dates": [...], "include_deleted": true } returns a search ID; results arrive in minutes, not milliseconds. POST /v1/admin/ediscovery/exports packages the results as EML files with a manifest into the tenant's own S3 bucket. Both need the ediscovery-manager role and are written to the audit log.

Import

jsonCopy
{ "import_id": "im_7Q2", "source": { "type": "IMAP", "host": "imap.oldprovider.example",
    "auth": "OAUTH2", "credential_ref": "secret:im_7Q2" },
  "mapping": [ { "from": "dana@acme-old.example", "to": "dana@acme.example" } ],
  "options": { "keep_dates": true, "keep_flags": true, "folders": "ALL" } }

POST /v1/admin/imports with this body starts a job; GET /v1/admin/imports/im_7Q2 reports progress per mailbox. An archive upload (MBOX or EML files into a staging bucket in the tenant's home region) is the other source type.

Recap

• Domains are verified by DNS before use; MX points at the home region, then the standby.
• Retention is policy; holds are separate and never expire on their own.
• eDiscovery is asynchronous and exported to the tenant; imports run as jobs, never through SMTP.

R3.4 Design Evolution: Tenants, Law, Attackers and Borders

Step 3.1: Thousands of Tenants, Each with Domains and Admins

Primitive: Circuit Breaker, Bulkhead and Fault Tolerance Patterns (cells are bulkheads at the scale of a whole stack) · Drill: One Customer, One Shard, One Outage. Its first question (the biggest tenant grows tenfold) is the dedicated cell above; its second (a query across all tenants) is the usage warehouse fed by the cells' streams.

Step 3.2: Retention, eDiscovery and Legal Hold

Step 3.3: Executives Are Being Phished

Primitive: Bot Defense, Sybil Resistance and Registration Abuse, for the same idea applied to accounts: judge the sender by its history and its signals, not by a single check.

Step 3.4: EU Mail Stays in the EU; Some Customers Want Their Own Keys

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active · Drill: The Booking That Existed in Frankfurt but Not in Virginia. Its first question is the collision above: two regions accepting writes to the same record during replication lag, which one owner region per tenant prevents. Its second (why not replicate synchronously everywhere) is answered by a cross-region round trip on every delivery, by deliveries failing whenever any region is down, and by multi-Region strong consistency, which runs in exactly three regions and supports no transactions, while our delivery is a transaction.

Step 3.5: Import Ten Years of Mail

Primitive: OAuth2, OIDC and Distributed Token Authentication, for the delegated grant to the old provider.

Step 3.6: Build or Buy Each Part?

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Content never crosses the jurisdiction line; only the tenant map (domains, tenant IDs, regions, owner epochs) and aggregate billing counts are global. Inbound mail finds the standby on its own through the second MX record, and the standby hands it back home within the same jurisdiction.

Synthesizing vector architecture diagram...

Each cell adds a tenant map lookup, a phishing stage, a second stream consumer (the eDiscovery corpus), a holds layer that every destroying path reads, per-tenant keys, an audit pipeline and an import path.

The audit pipeline. Every admin action, sign-in, eDiscovery search, export and access to a mailbox by anyone other than its owner is an event (about 300M a day, 500 bytes each: 150 GB a day). API servers pack up to 500 events (about 250 KB) into each Amazon Data Firehose record, because Firehose bills every record in 5 KB increments, and 500 is also the most events its deaggregation splits out of one record. Firehose partitions by tenant and writes compressed files to an audit bucket in the tenant's home region under S3 Object Lock in compliance mode for the tenant's contract period: nobody, including us, can delete them before it ends. Dynamic partitioning allows 500 active partitions per stream by default, so the US's 15,000 tenants go to 30 streams and the EU's 10,000 to 20, chosen by tenant hash. The EU's audit traffic (40% of 150 GB a day, about 0.7 MB/s) fits even the small default Direct PUT throughput some regions start with (1 MiB/s per stream in Frankfurt). Delivery is at least once, so each event carries an event_id and readers deduplicate by it.

Tracing a phishing message

Synthesizing vector architecture diagram...

Authentication passed because the sender really owns acrne.example. The lookalike check, not DMARC, is what caught it.

Tracing a legal hold preventing deletion (numbered steps)

1. Legal places h_2291 on Dana's and Raj's mailboxes. The admin API writes the hold and an audit event.
2. A job finds every message in scope and starts S3 Batch Operations to set Object Lock legal hold on the body-pack and attachment versions they use.
3. Dana deletes the "Falcon" thread and empties her trash. The messages become Hidden; she no longer sees them.
4. The nightly purge finds Dana's hidden messages past retention, reads the holds table, finds h_2291 covers Dana and the dates, and skips them.
5. Compaction finds Dana's pack is 60% dead but locked: it copies the live messages of other mailboxes into a new pack and leaves the old version in place.
6. A bug in a new purge release tries to delete one of the locked pack versions by version ID. S3 refuses; the purge job raises a P1 alarm. Even a delete without a version ID would only have added a delete marker.
7. Months later, legal releases h_2291. A job removes the Object Lock holds on versions no other hold references, and the next purge run deletes what retention allows.

Tracing an EU tenant's message, with the home region down

Synthesizing vector architecture diagram...

The sender never saw an error. The standby kept trying to relay the message home; once it became the owner, about 12 minutes in (R3.6), it parsed the queue itself.

R3.6 Numbers and Cost

Tenants and volume per jurisdiction (assumptions from the scope raise: 100 messages per seat a day, 30% of attachment bytes repeated within a tenant)

KMS requests for tenant keys (per message: tenant body-pack writes 0.04 (1B × 20 KB ÷ 512 KB packs = 39M a day), new attachments 0.21, indexer reads 1.0, user reads 0.6, scanner reads 0.21, old-mail and attachment downloads 0.08, replication reads 0.25: 2.39 at home; plus 0.25 replica writes in the standby)

We budget Bucket Keys at a 90% reduction for tenant keys (thousands of keys, each reused less than one busy key would be): Frankfurt's tenant-key peak falls to about 3,300 a second and their cost to about $24K a month. The spool's single per-cell key adds about 2.9 requests per message (spool writes, the parsers', repacker's and outbound MTAs' reads, replication); one heavily reused key gets close to the 99% figure, about 400 a second at Frankfurt's peak and $2.6K a month. Together, about 3,700 a second: 19% of Frankfurt's quota. A quota increase alone would leave us paying about $241K and one throttling incident away from refusing mail. Tenants who turn Bucket Keys off for per-object visibility are budgeted against the quota individually.

eDiscovery corpus. Per message about 7 KB of text (4 KB of body, plus 0.3 attachments × about 10 KB of extracted text), about 2.3 KB compressed (an assumption: 3× compression): 2.3 TB a day, 2.52 PB after 3 years. Stored in S3 Intelligent-Tiering (the files are large, so its monitoring fee is negligible): about $0.0074 per GB-month in the US and $0.00845 in Frankfurt for a 10/20/70 mix of its frequent, infrequent and archive-instant tiers. One full-text search over a 5,000-seat tenant's 3 years: 5,000 × 100 × 1,095 = 548M messages × 2.3 KB = 1.26 TB scanned × $5/TB ≈ $6.30.

Failover timing (dependent steps add):

Inbound mail doesn't wait for any of this: senders move to the second MX on their own, and the standby spools it. RPO: metadata, the replication delay, usually a few seconds (global tables replicate asynchronously); the spool, 15 minutes for 99.99% of objects under S3 Replication Time Control; packs and attachments replicate without RTC, but anything from the last 7 days can be rebuilt from the replicated spool. Mail whose spool object hadn't replicated when a region was lost is delayed until that region returns; it would only be lost if the region's S3 data were destroyed.

Monthly cost at the 3-year point (list prices, decimal GB, 730 hours. US home at us-east-1 prices; EU home at Frankfurt prices: S3 Standard $0.0245 / $0.0235 / $0.0225 per GB by tier, Standard-IA $0.0135, Glacier IR $0.005, DynamoDB about 22% above us-east-1, OpenSearch about 20% above; us-west-2 and eu-west-1 standbys at their own prices, which for the lines used here equal us-east-1's except DynamoDB in Ireland, about 13% higher. "Round 2 lines" means Round 2's per-line cost times the jurisdiction's share of the 1B messages.)

About $0.39 per seat a month: $0.38 in the US and $0.42 in the EU, where storage, DynamoDB and search all cost more and no CDN helps. Two numbers to argue about: the standby (replica tables, the second copy of 83 PB and its replication, standby compute) costs about $1.14M a month, 29% of the bill, the price of surviving a region within each jurisdiction; and compared with Round 2's $2.39M for the same billion messages, residency's price uplift, cells and the new features (keys, eDiscovery, anti-phishing, audit) add another $0.42M. A migration like step 3.5's adds its own $5.2K of provisioned writes for the 100 hours it runs.

Retention vs cost. Each extra year a tenant keeps its mail adds about 27.5 PB of S3 (75.3 KB per message × 1B × 365, all of it in Glacier IR by then) in each of home and standby, and about 544 TB of metadata:

The users' search index stops at 3 years (older mail is found through eDiscovery or a slower archive search), so it isn't in these rows.

R3.7 Trade-Offs

Shared cells vs a cell per tenant

Retention vs cost. The table in R3.6: every extra year is about $472K a month once stored, and a hold can keep a mailbox's mail long past its retention. We default to 3 years for ordinary mail and let tenants choose more, priced per seat, because "keep everything forever" is a cost that only grows.

Anti-phishing strictness vs friction

Closing the loop. Round 1 answered "how do we accept mail without losing it?" with 250 only after durable storage and a queue in front of the parser. Round 2 kept that and changed how bytes are stored (packs, hashes), how views stay exact (conditional counters, threads, an index fed from the stream) and how we treat the hostile internet in both directions. Round 3 added companies, courts, attackers and borders, and still every piece hangs off the same spine: mail is accepted only once it's durable, every change to a mailbox is one conditional transaction with the next change number, and everything else (search, eDiscovery, push, audit) follows that ordered record. The holds, the keys and the standby all protect that record rather than replace it.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per cell and per tenant OPS 8 · REL 6

Reputation incident for one tenant SEC 10

1. Confirm: complaints from distinct recipients, deferrals and spam-trap hits concentrated on one tenant.
2. Pause the tenant's outbound (command 6); its queued deliveries wait, nothing is dropped.
3. Move the tenant to the quarantine pool before resuming, at a low rate.
4. With the tenant's admin: find and reset the compromised accounts; review their sent mail.
5. Check blocklists for the pool's IPs; request delisting once the source is stopped.
6. Resume gradually while complaints stay under the threshold.

Backlog drain REL 7

1. Check the queue's depth and age (commands 2 and 3).
2. Scale the parsers (command 4): the cell runs 36 parser tasks, and we add 100. Each 4-vCPU task handles about 200 messages a second, so 100 extra tasks drain 20,000 a second beyond the arrival rate: a backlog of 3.6M messages clears in about 3 minutes.
3. Watch DynamoDB throttling while draining; the write quota, not the parsers, is often the real ceiling.
4. Redrive the DLQ only after the cause is fixed (command 5).

Regional failover REL 13

1. Confirm it's the region: the standby's canaries, several services and AWS Health agree.
2. Check replica lag (command 10) and spool replication lag to estimate what may be delayed.
3. From the standby region, stop the lease acks and wait until the last ack_until plus 30 s has passed. Then move each affected tenant's owner with a new epoch (command 11), and only then flip the ARC routing control (command 12). ARC's cluster has endpoints in five regions; try them in turn until one answers.
4. Let the standby's parsers drain the spooled mail; confirm deliveries and sends succeed.
5. Fail back tenant by tenant at quiet hours after replication catches up; run the reconciliation.

Go deeper: CLI playbook

Plain commands an on-call engineer runs one at a time. Replace names, times and IDs with real ones.

textCopy
# 1. Alarms firing for a cell
aws cloudwatch describe-alarms --region eu-central-1 --state-value ALARM --alarm-name-prefix mail-c1-

# 2. Age of the oldest message in the parse queue, last hour
aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage --dimensions Name=QueueName,Value=parse-queue-c1 --start-time 2026-09-28T08:00:00Z --end-time 2026-09-28T09:00:00Z --period 60 --statistics Maximum

# 3. Depth of the parse queue
aws sqs get-queue-attributes --region eu-central-1 --queue-url https://sqs.eu-central-1.amazonaws.com/111122223333/parse-queue-c1 --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# 4. Scale the parsers
aws ecs update-service --region eu-central-1 --cluster mail-c1 --service parser --desired-count 136

# 5. Redrive the dead-letter queue after the fix
aws sqs start-message-move-task --region eu-central-1 --source-arn arn:aws:sqs:eu-central-1:111122223333:parse-dlq-c1 --destination-arn arn:aws:sqs:eu-central-1:111122223333:parse-queue-c1

# 6. Pause one tenant's outbound mail
aws dynamodb update-item --region eu-central-1 --table-name tenant-policy --key '{"tenant_id":{"S":"t_globex"}}' --update-expression "SET outbound_paused = :t, paused_reason = :r" --expression-attribute-values '{":t":{"BOOL":true},":r":{"S":"complaint spike"}}'

# 7. Pause purge and compaction
aws ssm put-parameter --region eu-central-1 --name /mail/purge/paused --value true --type String --overwrite

# 8. Is this pack version under legal hold?
aws s3api get-object-legal-hold --region eu-central-1 --bucket mail-packs-euc1 --key t/t_acme/packs/2026/09/28/p_7f3a --version-id 3HL4kqtJlcpXroDTDmJ.rmSpXd3dIbrHY

# 9. A tenant key's state, in both regions
aws kms describe-key --region eu-central-1 --key-id arn:aws:kms:eu-central-1:444455556666:key/mrk-1a2b3c4d5e6f47a8b9c0d1e2f3a4b5c6 --query "KeyMetadata.KeyState"
aws kms describe-key --region eu-west-1 --key-id arn:aws:kms:eu-west-1:444455556666:key/mrk-1a2b3c4d5e6f47a8b9c0d1e2f3a4b5c6 --query "KeyMetadata.KeyState"

# 10. Replica status of the messages table, seen from the standby
aws dynamodb describe-table --region eu-west-1 --table-name messages-c1 --query "Table.Replicas"

# 11. First, move a tenant's ownership to the standby with a new epoch, run in the standby
aws dynamodb update-item --region eu-west-1 --table-name tenant-map --key '{"tenant_id":{"S":"t_acme"}}' --update-expression "SET owner_region = :r, owner_epoch = owner_epoch + :one" --condition-expression "owner_region = :old" --expression-attribute-values '{":r":{"S":"eu-west-1"},":one":{"N":"1"},":old":{"S":"eu-central-1"}}'

# 12. Then flip the EU pair's routing control to the standby (ARC data plane)
aws route53-recovery-cluster update-routing-control-state --region eu-west-1 --endpoint-url https://<cluster-endpoint>/v1 --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/0123456789abcdef/routingcontrol/abcdef0123456789 --routing-control-state On

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Routes by domain to a tenant, checks tenancy on every call, and uses cells to bound the blast radius, with a separate path for cross-tenant reporting.
• Treats legal holds as a layer that every destroying path reads at the moment it acts, and adds an independent second layer (Object Lock), knowing its delete-marker and lifecycle behavior.
• Separates eDiscovery (tenant-wide, batch, audited) from user search (per mailbox, interactive).
• Defends against targeted phishing with checks that don't depend on authentication failing.
• Designs residency as home and standby within a jurisdiction, uses the second MX for inbound continuity (with the standby relaying home all the time and filtering as hard as home), fences the old home with a lease on its own clocks, and runs failover from the standby behind a human gate.
• Explains exactly what customer keys promise, including replica keys and derived copies, and does the KMS quota arithmetic.
• Imports through a dedicated, idempotent, budgeted path.

Follow-up questions

1. "A customer revokes its key at noon. What exactly stops working?" Answer: once both the primary and the replica key are disabled, S3 can't decrypt that tenant's packs and attachments: reads fail and new mail for that tenant gets 451 at our MTAs, so senders keep it and retry. Our 5-minute cache of the tenant's field data key expires, so subjects and senders in the metadata become unreadable. We delete its search indices and eDiscovery query caches. The 7-day spool, under our cell key, expires on schedule unless the tenant has a dedicated cell. Re-enabling the key restores everything except the indices, which we rebuild.

2. "Why not keep the EU standby in the US for cheaper storage?" Answer: the contract says backups stay in the EU, and a standby copy is a backup. Glacier IR in Ireland costs the same $0.004 per GB-month as us-west-2, so residency doesn't even cost us on that line; the EU premium is in Frankfurt's home prices.

3. "Can a legal hold be placed on mail that arrives after the hold?" Answer: yes, and it must be. That's why the hold is a scope (mailboxes and dates) in its own table, read by every destroying path when it acts, rather than a flag stamped on existing messages. A job also extends Object Lock to new packs and attachments that held messages use.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening any round: "A mailbox is a small, hot index over a large, cold pile of immutable messages, so I store the raw message durably before answering 250, parse it asynchronously, and commit each mailbox change as one conditional transaction with the next change number; views, search and sync all follow that record."
• When scale arrives: "I'll run our own MTAs with cheap checks first, store attachments once by hash with a collector that re-checks before deleting, keep counters in the same transactions, feed search from the change stream, parse hostile mail as a stream with limits, and treat every 4xx from a receiver as a request to slow down."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels