Design a Distributed Email Service

1. Problem Statement & Scope

System Mission

Design a massively scalable, secure, and highly available distributed email infrastructure capable of handling tens of billions of daily inbound and outbound messages, supporting standard protocols (SMTP, IMAP, POP3, JMAP), real-time spam and malware filtering, cryptographic sender authentication (SPF, DKIM, DMARC), decoupled blob storage for attachments, and sub-second full-text mailbox search.

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Functional Requirements

1. Inbound Email Ingestion (MX): Accept incoming RFC 5322 MIME messages over SMTP (Port 25/587 with STARTTLS).
2. Outbound Email Dispatch: Queue, sign with DKIM private keys, and reliably deliver outbound emails to external recipient MX hosts.
3. Mailbox Hierarchy & Threading: Support user folders (Inbox, Sent, Trash, Drafts, Custom Labels), read/unread status, and conversation threading via In-Reply-To and References headers.
4. Decoupled Storage & Attachments: Separate lightweight mailbox metadata from large raw MIME bodies and binary attachments (up to $25\text{ MB}$).
5. Full-Text Search: Sub-second search across email subjects, sender addresses, recipient lists, and message body contents.
6. Security & Spam Filtering: Verify SPF, DKIM, and DMARC alignment; evaluate incoming payloads against real-time IP reputation lists and Bayesian spam classifiers.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.999\%$ uptime SLA for mail access and delivery ($< 5.26\text{ minutes}$ downtime/year).
• Data Durability: Zero data loss ($100\%$ durability for committed emails; 11 9s durability via Amazon S3).
• Delivery Latency: Outbound mail delivered to recipient MX in $< 5\text{ seconds}$ ($P95$); Inbound mail visible in user inbox in $< 2\text{ seconds}$.
• Mailbox Read Latency: Rendering first 50 folder messages in $< 50\text{ ms}$ ($P99$).
• Search Query Latency: Full-text query across 100,000 user emails in $< 300\text{ ms}$.

Out-of-Scope

• Real-time collaborative calendar and scheduling suites (CalDAV/iCalendar protocols).
• Web-based video conferencing integrations.

2. Capacity & Scale Estimation

Traffic & Throughput Calculations

• Active Mailboxes: $100\text{ Million}$ accounts.
• Daily Inbound & Outbound Emails: $1\text{ Billion}$ emails/day. $\text{Average Ingestion QPS} = \frac{10^9 \text{ emails}}{86,400 \text{ s}} \approx 11,574 \text{ emails/sec}$ $\text{Peak Ingestion QPS (3}\times\text{ morning surge)} = \mathbf{35,000\text{ emails/sec}}$

Storage Sizing & Bandwidth Math

• Average Email Size: $100\text{ KB}$ (Metadata: $1\text{ KB}$, Text Body: $20\text{ KB}$, Attachments: $79\text{ KB}$ average). $\text{Daily Raw Email Ingestion} = 10^9 \times 100\text{ KB} = 100\text{ TB/day} \implies \mathbf{36.5\text{ PB/year}}$
• Storage Tier Allocation:
  • DynamoDB Metadata (Hot): $10^9 \times 1\text{ KB} = 1\text{ TB/day} = \mathbf{365\text{ TB/year}}$.
  • Amazon S3 Raw MIME & Attachments: $10^9 \times 99\text{ KB} = 99\text{ TB/day} = \mathbf{36.135\text{ PB/year}}$.
  • OpenSearch Search Inverted Index: Indexing subject + body text ($\approx 10\text{ KB/email}$) with 1 primary + 1 replica $\implies 20\text{ TB/day}$.
• Network Egress / Ingress Bandwidth: $\text{Peak Network Throughput} = 35,000 \text{ msgs/s} \times 100\text{ KB} = 3.5\text{ GB/s} = \mathbf{28\text{ Gbps}}$

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Component Responsibility Breakdown

4. API Interface Design & Wire Protocols

1. Send Outbound Email (REST API)

httpCopy
POST /v1/mail/messages/send HTTP/1.1
Host: mail.production.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "to": ["architect@distributed-systems.org"],
  "cc": ["lead-sre@company.com"],
  "subject": "System Design Architecture Review: LSM vs B-Trees",
  "body_html": "<p>Team, attached is the revised proposal for the write-optimized storage engine.</p>",
  "body_plain": "Team, attached is the revised proposal for the write-optimized storage engine.",
  "in_reply_to": "<msg_88129a@distributed-systems.org>",
  "attachments": [
    {
      "file_name": "lsm_benchmark_results.pdf",
      "content_type": "application/pdf",
      "s3_temp_upload_key": "temp-uploads/user_101/tmp_pdf_998124"
    }
  ]
}

Response: 202 Accepted

jsonCopy
{
  "message_id": "<msg_20260904_998124756@mail.aws.internal>",
  "status": "QUEUED_FOR_DELIVERY",
  "queued_at": 1718000050123
}

2. List Folder Messages (Cursor Pagination)

httpCopy
GET /v1/mail/folders/INBOX/messages?limit=25&cursor=eyJkYXRlIjoxNzE4MDAwMDAwLCJtc2dfaWQiOiJtc2dfMTAxIn0= HTTP/1.1
Host: mail.production.aws.internal
Authorization: Bearer <jwt_token>

Response: 200 OK

jsonCopy
{
  "folder": "INBOX",
  "unread_count": 4,
  "total_count": 1420,
  "messages": [
    {
      "message_id": "msg_20260904_998124756",
      "thread_id": "thread_abc123xyz",
      "from": {"name": "Alice Wang", "email": "alice@cloud.org"},
      "subject": "System Design Architecture Review",
      "snippet": "Team, attached is the revised proposal for the write-optimized...",
      "has_attachments": true,
      "is_read": false,
      "received_at": 1718000050000
    }
  ],
  "next_cursor": "eyJkYXRlIjoxNzE3OTk5MDAwLCJtc2dfaWQiOiJtc2dfMDk5In0="
}

5. Data Models & Storage Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

DynamoDB Single-Table Schema (MailboxCoreTable)