Design a Scalable Distributed Notification System

1. Problem Statement & Scope Clarification

System Mission

Design a globally distributed, highly available, multi-channel notification platform (similar to Twilio, OneSignal, and Amazon SNS/SES) capable of delivering billions of notifications daily across Mobile Push (Apple APNs, Google FCM), SMS, Transactional Email, and Webhooks, with strict priority isolation, user preference compliance (quiet hours, category opt-outs), intelligent deduplication, and rate-limiting.

Functional Requirements

1. Multi-Channel Dispatch Engine: Unified API supporting Mobile Push (iOS APNs HTTP/2, Android FCM v1), SMS (Amazon SNS / Twilio), Email (Amazon SES), and In-App WebSockets.
2. Strict Priority Queuing: Critical transactional messages (OTP, 2FA, fraud alerts) must bypass marketing batches and achieve sub-second delivery.
3. Template Rendering & Localization: Dynamically render multi-lingual templates with personalized user variables fetched from profile stores.
4. Intelligent Deduplication: Suppress duplicate notifications for the same event trigger within a configurable sliding TTL window (e.g., preventing duplicate billing alert pushes).
5. User Preference & Regulatory Compliance: Enforce per-user channel preferences, quiet hour dampening across local user timezones, and statutory opt-out compliance (TCPA, GDPR, CAN-SPAM).
6. Device Token Lifecycle Management: Automatically handle device token invalidations (e.g., APNs 410 Gone / FCM UNREGISTERED) without crashing worker pipelines.

Non-Functional Requirements (SLAs & SLOs)

• High Availability: $99.999\%$ uptime SLA for critical OTP/transactional notification paths.
• Latency:
  • High-Priority (OTP / 2FA / Security Alerts): Delivery P99 $< 3\text{ seconds}$.
  • Low-Priority (Marketing / Daily Digests): Delivery within 15 minutes of batch trigger.
• Scale: Ingest peak of $> 100,000\text{ notifications/sec}$; deliver over $2.0 \times 10^9\text{ notifications/day}$.
• Durability: Zero acknowledged message loss ($\text{Durability} \ge 99.999999999\%$).

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Traffic & Ingest Profile

• Total Daily Notification Volume: $2,000,000,000\text{ notifications/day}$ (2 Billion/day).
• Channel Distribution:
  • Mobile Push (APNs / FCM): $65\%$ ($1.30\text{ Billion/day}$).
  • Transactional & Marketing Email (SES): $25\%$ ($500\text{ Million/day}$).
  • SMS & WhatsApp (SNS / Twilio): $10\%$ ($200\text{ Million/day}$).
• Average Ingest QPS: $\text{Average Ingest QPS} = \frac{2 \times 10^9}{86,400\text{ sec}} \approx \mathbf{23,148\text{ msg/sec}}$
• Peak Ingest QPS ($3.5\times$ multiplier during global flash sales): $\text{Peak Ingest QPS} = 23,148 \times 3.5 \approx \mathbf{81,000\text{ msg/sec}}$

Bandwidth & Worker Sizing

• Average Ingestion Payload Size: $1.5\text{ KB}$ (Metadata + template variables).
• Peak Ingest Network Bandwidth: $\text{Bandwidth} = 81,000\text{ req/sec} \times 1.5\text{ KB} = 121.5\text{ MB/sec} \approx \mathbf{972\text{ Mbps}}$
• Push Worker Fleet Sizing: An HTTP/2 persistent connection to APNs handles $\approx 500\text{ dispatches/sec}$. $\text{Required APNs HTTP/2 Connections} = \frac{50,000\text{ peak push/sec}}{500\text{ req/conn}} = \mathbf{100\text{ Multiplexed Connections}}$

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

4. API Interface Design & Wire Protocol

Unified Notification Ingest Protocol (notification.proto)

protobufCopy
syntax = "proto3";

package hispeeddesign.notification.v1;

service NotificationService {
  rpc SendNotification (SendNotificationRequest) returns (SendNotificationResponse);
  rpc SendBatchNotification (SendBatchNotificationRequest) returns (SendBatchNotificationResponse);
}

enum PriorityLevel {
  PRIORITY_UNSPECIFIED = 0;
  PRIORITY_CRITICAL = 1;    // Bypasses quiet hours, delivers via FIFO queue (OTP/MFA)
  PRIORITY_HIGH = 2;        // Transactional receipts, billing notices
  PRIORITY_LOW = 3;         // Marketing promotions, weekly newsletters
}

enum ChannelType {
  CHANNEL_PUSH = 0;
  CHANNEL_SMS = 1;
  CHANNEL_EMAIL = 2;
  CHANNEL_IN_APP = 3;
}

message SendNotificationRequest {
  string user_id = 1;
  string idempotency_key = 2;            // Client-supplied deduplication key
  PriorityLevel priority = 3;
  repeated ChannelType preferred_channels = 4;
  string template_id = 5;
  map<string, string> template_variables = 6;
  int32 deduplication_window_seconds = 7; // Default: 300s
}

message SendNotificationResponse {
  string notification_id = 1;
  enum Status {
    QUEUED = 0;
    DROPPED_DUPLICATE = 1;
    DROPPED_OPT_OUT = 2;
    DROPPED_QUIET_HOURS = 3;
  }
  Status status = 2;
  int64 timestamp_ms = 3;
}

5. Data Models & DynamoDB Preference & Token Schema

DynamoDB Table: NotificationUserDataTable