Design a Notification System

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Notification System?

You Already Know One: the Post Office

A notification system is the post office of an app. Other services drop off letters ("order 5521 was confirmed", "here is your login code"). We look up the address, put the letter in the right envelope, stamp it, and hand it to a carrier. The carriers are companies we don't run:

What Makes It Hard

We don't control the last mile. Apple, Google, the inbox providers and the phone networks decide whether and when a message arrives. Each has its own limits, its own error codes and its own bad days. Three things follow from that:

• Sending is slow and can fail, so nobody else should wait for it.
• A retry can send the same message twice. A login code that arrives twice confuses people; a payment reminder sent twice angers them.
• Sending the wrong thing costs more than sending nothing. A marketing push at 3 a.m. loses users. A text to someone who opted out can break the law. A spammy email campaign can get all our email sent to spam folders.

The Question the Whole Loop Answers

How do we get the right message to the right person once, at the right time, through carriers we don't control?

The answer gets sharper every round:

• Round 1: put a queue between "a thing happened" and "send it", retry safely, and be honest about where "exactly once" stops.
• Round 2: not all messages are equal. Login codes need their own lane; marketing must respect preferences, quiet hours, caps, the law and the carriers' speed limits.
• Round 3: we are the carrier's customer on behalf of thousands of other companies. One tenant's mistake must not cost everyone else their inbox placement, and a region can go down.

---

Round 1 · Mid-level · "Order Confirmations for One App"

~35 min · SDE II (L5) · 1 region, 3 AZs · ~1M notifications/day · ~50/s peak · 99.9% · handed to the provider within 1 min

R1.1 Establish Design Scope

The interviewer says: "Our shop needs to tell customers about their orders. Design the notification system." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round: SMS, marketing, user preferences and quiet hours, priorities between message types, and other teams using the system. We write that list down; Round 2 brings all of it back.

R1.2 Functional Requirements, Derived Step by Step

Not yet: priorities (everything is transactional), preferences and opt-outs, quiet hours, SMS, and campaigns to millions of users.

R1.3 Non-Functional Requirements: the Questions

We state each quality in words first. Numbers come in R1.7.

• Don't block the order flow. Checkout's latency must not depend on Apple or an email provider. If they are slow, we are still fast.
• Don't lose a notification. Once we answer "accepted", the notification must eventually be handed to the provider or recorded as failed with a reason.
• Don't send duplicates. A retry anywhere (the order service, our workers, the queue) must not produce a second email, as far as that's possible.
• Keep a small audit trail. Support asks "did the customer get the shipping email?" We must be able to answer: sent when, to which address, and what the provider said.

R1.4 The API

Send a notification

httpCopy
POST /v1/notifications HTTP/1.1
Content-Type: application/json
Idempotency-Key: order-5521-confirmed

{
  "user_id": "u_1001",
  "template_id": "order_confirmed",
  "channels": ["email", "push"],
  "variables": { "first_name": "Ana", "order_id": "5521", "total": "42.90 EUR" }
}

httpCopy
HTTP/1.1 202 Accepted
Content-Type: application/json

{ "notification_id": "ntf_01J8Z6Q4M3T9V2K7H5B1C0D8E4", "status": "ACCEPTED" }

Why 202 Accepted and not 200 OK? 200 says "done". We aren't done: nothing has gone to Apple or the email provider yet. 202 says "we accepted the job and will do it later". It's the honest answer for asynchronous work, and it tells the caller not to wait for delivery. The caller can check progress later:

httpCopy
GET /v1/notifications/ntf_01J8Z6Q4M3T9V2K7H5B1C0D8E4 HTTP/1.1

jsonCopy
{
  "notification_id": "ntf_01J8Z6Q4M3T9V2K7H5B1C0D8E4",
  "deliveries": [
    { "channel": "email", "status": "SENT", "provider_message_id": "0100019a-...", "sent_at": "2026-09-26T18:02:11Z" },
    { "channel": "push", "status": "FAILED", "reason": "NO_ACTIVE_DEVICE" }
  ]
}

Register a push token. Tokens come from the phone's operating system, not from us. The app sends its token on every launch, because tokens change (after a reinstall, a restore, or when the OS rotates them).

httpCopy
POST /v1/users/u_1001/devices HTTP/1.1
Content-Type: application/json

{ "platform": "IOS", "token": "a1b2c3...", "app_version": "5.2.0" }

jsonCopy
{ "device_id": "dev_7f3a", "status": "ACTIVE" }

The Idempotency-Key is chosen by the caller. The order service uses order-<id>-<event>, so if it retries the same event, it sends the same key. We define "retry" as "same key", and we'll use that in step 1.3.

Status codes

Recap

• One call to send, one to register a device, one to check status.
• About 200,000 orders a day, three events each, email and push.
• The caller never waits for a provider; we answer 202.
• Retries are safe because the caller's key identifies the request.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From a Direct Call to a Queue With Workers

Every step follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is usually the next problem.

Step 1.0: The Baseline

The order service calls the email provider and the push service directly, right after it saves the order.

Synthesizing vector architecture diagram...

It works on day one: few moving parts, and the order service knows immediately whether the send succeeded. Its weakness is that checkout's speed and uptime now include the email provider's speed and uptime.

Step 1.1: Checkout Waits on the Email Provider

This also answers the drill question "why not have the order service call the other service directly and wait?": a direct call couples checkout's latency and availability to the provider's; a queue absorbs bursts and outages, and lets each side scale on its own.

Synthesizing vector architecture diagram...

The order service's work ends at the API's 202. Everything to the right of the queues can be slow or down for a while without checkout noticing.

Primitive: Message Queues vs Event Streams · Drill: Message queue order pipeline

Step 1.2: The Provider Returned an Error

This answers the other drill question, "what happens to an event if the consumer throws halfway through?": the worker never deleted the message, so it reappears after the visibility timeout and is retried; after the maximum number of receives, the redrive policy moves it to the DLQ.

Why 10 receives and a 15-minute cap? With delays capped at 10, 20, 40, 80, 160, 320, 640, 900, 900 s between ten attempts, the total is 10 + 20 + 40 + 80 + 160 + 320 + 640 + 900 + 900 = 3,070 s, about 51 minutes. A short provider blip is retried through; a message that fails for 51 minutes is unusual enough that a person should look. For a long provider outage, burning attempts is wrong anyway; R1.9 pauses consumption instead.

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance

Step 1.3: A Retry Sent the Confirmation Twice

The claim, in pseudocode. DynamoDB has no server-side clock in a condition, so the worker passes its own time and we allow for small clock differences between workers:

textCopy
UpdateItem  PK = "NTF#ntf_01J8...", SK = "DELIVERY#email"
  SET status = "SENDING", claimed_at = :now, attempt = attempt + 1
  CONDITION  status = "QUEUED"
         OR (status = "SENDING" AND claimed_at < :now_minus_60s)

The 60-second takeover window is longer than a send takes, and far longer than the few seconds workers' clocks can differ. If a worker died mid-send, the next one takes over after 60 s; that's exactly the "unknown outcome" case, and it resends.

Where the records live. DynamoDB, keyed by what we look up:

We keep idempotency records for 24 hours. DynamoDB TTL deletes expired items only eventually (typically within a few days), so the API treats a record with expires_at in the past as absent, and TTL is only the cleanup crew.

Step 1.4: Where Do Push Tokens Come From, and Some Are Dead

We hash the token for the sort key: tokens have no fixed length (Apple says not to assume one), and the hash gives a fixed-length key without putting the raw token in the key.

Step 1.5: Marketing Wants to Change the Wording

The version in the sort key is zero-padded. Sort keys compare as strings, and as strings V#10 sorts before V#9; fixed width keeps "latest version" a simple reverse query.

An example template (email body):

textCopy
Hi {{first_name}}, your order {{order_id}} is on its way.
Track it here: {{tracking_url}}

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Follow a request from the top. The order service signs its call with its IAM role; the private API only accepts calls from inside our network. The ingest function checks the template's required variables, writes the idempotency record, and puts one message per channel on that channel's queue, then answers 202. Each worker function receives batches of up to 10 messages, claims each delivery, renders the template, and sends. Bounces and complaints come back from SES later, as events, and mark the address so we stop mailing it.

Tracing one order confirmation

1. The order service sends POST /v1/notifications with Idempotency-Key: order-5521-confirmed.
2. Ingest writes IDEM#orders#order-5521-confirmed with a condition that it doesn't exist; it succeeds, so this is new. It writes the notification and two deliveries (QUEUED), sends one message to each queue, and returns 202 with the ID.
3. The email worker receives a batch, claims DELIVERY#email (QUEUED → SENDING), reads the contact and template version 3, renders, and calls SES SendEmail.
4. SES accepts and returns a message ID. The worker writes SENT with that ID and deletes the SQS message.
5. The push worker does the same through SNS, using the collapse ID order-5521-confirmed so a retry can only replace, never stack.
6. An hour later, SES reports a hard bounce for a different customer's address. The event handler marks that contact BOUNCED, and future emails to it fail fast as permanent errors. (With SES's account-level suppression list turned on, SES also refuses sends to addresses that recently hard-bounced or complained: a second guard.)

Why Lambda workers and not servers? At ~50 messages a second, a server fleet would sit idle most of the day. Lambda's SQS integration polls the queue for us, and its maximum concurrency setting caps how many copies run at once. That cap limits parallel invocations, not sends per second: if SES answers in 30 ms, five copies sending back to back would reach about 167 a second. So each invocation also paces itself (R1.7), and the concurrency cap is only the ceiling.

R1.7 Numbers

Targets

Traffic

Queue depth during a provider outage

• SES is unreachable for one hour at peak: 20/s × 3,600 s = 72,000 emails wait in the queue. SQS has no practical limit on backlog; this is nothing.
• When SES returns, workers drain at our speed limit of 50/s (below) while new mail keeps arriving at 20/s. Spare capacity is 50 − 20 = 30/s, so the backlog clears in 72,000 ÷ 30 = 2,400 s = 40 minutes.
• A whole-day outage at the average email rate (400,000 ÷ 86,400 ≈ 4.6/s) leaves 400,000 messages; spare capacity is 50 − 4.6 ≈ 45/s, so draining takes 400,000 ÷ 45 ≈ 8,900 s ≈ 2.5 hours. Retention is 14 days, so nothing expires.

SES quota. SES limits each account per region with a sending quota (emails per rolling 24 hours) and a maximum send rate (per second). A new account starts in the sandbox (200 emails a day, 1 per second). We ask to leave the sandbox with a quota of 1,000,000 a day and 50 a second: 2.5× our daily volume and our drain speed.

Workers

Pacing inside each invocation, times the concurrency cap, is what keeps a 72,000-message backlog from turning into SES throttling errors: the drain is held at or under the rate SES granted, so the drain times above stand. If SES still answers with a throttling error (for example, while a quota change is propagating), that send is treated as retryable, with backoff, like any other throttle. (A shared token bucket, as in Round 2, is the alternative when workers aren't Lambda.)

Latency when healthy: API Gateway plus a warm ingest function, one DynamoDB write and one SQS send: well under 100 ms to 202 (a cold start adds a few hundred ms). SQS to worker: typically under a second. SES or SNS call: about 100 ms. Far inside one minute.

Rough monthly cost (us-east-1 list prices, 30.4 days a month; check the AWS Pricing Calculator before quoting):

About 82% of the bill is SES. The rest of the system costs less than a team lunch.

R1.8 Trade-Offs

Push through SNS vs straight to APNs and FCM

At this volume, SNS saves weeks of work for $9 a month. Round 2 revisits the choice at 1.3 billion pushes a day, and money won't be the reason we change.

One queue for everything vs one queue per channel

One queue is simpler, but an SES outage then fills it with email retries while push messages wait behind them. With a queue per channel, each channel's workers, retries, speed limit and alarms are separate. Two queues cost nothing extra in SQS (we pay per request, not per queue), so we choose one per channel.

Lambda consumers vs polling servers

R1.9 Failure Modes

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Asks what triggers a notification, which channels, how fast, and whether duplicates matter, and says what each answer changes.
• Puts a queue between producers and providers, returns 202, and explains why 202 and not 200.
• Separates retryable from permanent errors, uses backoff with jitter, and knows what a DLQ is for.
• Uses an idempotency key at the API and a send record at the worker, and says honestly that a timeout after the provider accepted can still cause a duplicate.
• Keeps a device registry and removes tokens the providers reject.
• Derives traffic, backlog and drain time, and ties worker concurrency to the provider's rate.

Follow-up questions

1. "The order service saves the order, then calls your API, and that call fails. Is the notification lost?" Answer: yes, if the order service simply gives up: this is the dual-write problem (two systems written without a shared transaction, so one can succeed while the other fails). The fix belongs in the order service: write the order and an "outbox" row in the same database transaction, then publish from the outbox. Publishing can read the database's change log (change data capture, for example DynamoDB Streams or Debezium reading a Postgres write-ahead log), or poll the outbox table every half second. Change data capture adds no query load and picks up changes within about a second; polling is simpler to build but runs a query forever, adds load and index churn on a busy table, and adds up to one polling interval of delay. Either way, our idempotency key makes the repeated publish harmless. Drill: CDC outbox dual-write drift

2. "Why not use SQS FIFO to get exactly-once?" Answer: FIFO removes duplicate sends into the queue within a 5-minute window. Our duplicates come from elsewhere: a timeout after SES accepted, or a producer retry after 5 minutes. The idempotency record and send claim handle those. FIFO would also add ordering we don't need.

3. "How do you know the email was delivered?" Answer: SES publishes delivery, bounce and complaint events for each message through a configuration set. For push, APNs and FCM only tell us they accepted it; to know it reached the phone, the app reports receipt to our API. Status reads "SENT" until one of those arrives.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "2B Notifications a Day, OTPs and Marketing"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 2B notifications/day · ≈ 23K/s average, ≈ 81K/s peak · 99.99% · OTP P99 < 3 s, alerts P99 < 15 s, marketing within 15 min

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built order notifications for one shop: about a million a day, 50 a second at peak, email and push, one region, 99.9%. The order service calls our API with an idempotency key and gets 202 Accepted in milliseconds; a conditional write on that key makes producer retries safe. The API puts one message per channel on an SQS queue, email and push separately. Lambda workers claim each delivery with a conditional write before sending, render a versioned template, and call SES or SNS mobile push. Errors are sorted: permanent ones fail at once, retryable ones back off with full jitter, and anything unexplained lands in a DLQ after 10 receives. During a provider outage we pause the consumer instead of burning retries. A device registry drops tokens the providers reject. It costs about $1,480 a month, mostly SES. Open costs: 'exactly once' stops at a timeout after the provider accepted, everything is one priority, and there are no preferences or opt-outs."

Architecture v1, compact

textCopy
 order service ──IAM-signed──► API Gateway ──► ingest Lambda ──conditional put (Idempotency-Key)──► DynamoDB
                                                   ├──► SQS email ──► email worker (max 5) ──claim──► SES
                                                   └──► SQS push  ──► push worker (max 5)  ──claim──► SNS ──► APNs / FCM
 each queue ──after 10 receives──► DLQ        SES bounces ──► SNS ──► Lambda ──► mark address

Round 1 step summary

Open costs: one priority for everything, no preferences or opt-outs, no speed limits of our own, and a design sized for 50 messages a second.

R2.1 The Scope Raise

Interviewer: "Every team in the company now sends through you: 2 billion notifications a day. Login codes, the one-time passwords (OTPs), must arrive in seconds, even while marketing sends a push to 50 million users. Users choose their channels and quiet hours, opt-outs are the law, and the same billing alert must never arrive twice in a day. SMS and an in-app inbox join. And Apple and Google are already throttling us."

A scope raise is not the end of scoping. Before we fix anything, we ask back, and we say what each answer changes.

Scope change

The "Not yet" list from R1.2 is now mandatory: priorities, preferences and opt-outs, quiet hours and SMS.

R2.2 What Breaks in the Round 1 Design

The order we fix it in: first what can hurt users or break the law (priorities, preferences, dedup: 2.1–2.3), then what the providers force on us (2.4, 2.5), then what users feel over a day (2.6). Sizing comes after the design, in R2.6.

R2.3 New Requirements and API Additions

Templates carry the category, and the category decides the priority. A team registers a template once; a reviewer approves its category. Requests can't choose their own priority.

httpCopy
POST /v1/templates HTTP/1.1
Content-Type: application/json

{
  "template_id": "login_otp",
  "owner_team": "identity",
  "category": "security_code",
  "channels": ["sms", "push"],
  "sms": { "body": "{{code}} is your login code. It expires in 5 minutes." },
  "push": { "title": "Login code", "body": "{{code}} is your login code." },
  "required_vars": ["code"]
}

Send request, extended

httpCopy
POST /v1/notifications HTTP/1.1
Content-Type: application/json
Idempotency-Key: billing-job-7731-run-3

{
  "user_id": "u_1001",
  "template_id": "invoice_reminder",
  "variables": { "invoice_id": "123", "amount": "19.99 USD" },
  "dedup_key": "invoice-123:reminder",
  "dedup_window_seconds": 86400,
  "expires_at": "2026-09-27T09:00:00Z",
  "drop_if_deferred": false
}

• dedup_key: the producer's name for "this logical message". Defaults to the idempotency key.
• dedup_window_seconds: default 300, maximum 86,400 (24 hours).
• expires_at: after this, the message is worthless and must not be sent (an OTP sets it 5 minutes out).
• drop_if_deferred: if the user is in quiet hours, drop instead of holding (a flash-sale push that is useless by morning).

Response statuses

200 for the drops, because the request is finished: we decided, and there is nothing left to do. A retry of an accepted request (same idempotency key) still returns the original 202 body.

Campaigns. Marketing doesn't loop over 50M users and call us 50M times. It hands us a segment and a start time, and our fan-out job expands it at a controlled speed:

httpCopy
POST /v1/campaigns HTTP/1.1
Content-Type: application/json

{
  "campaign_id": "fall-sale-2026",
  "template_id": "fall_sale_push",
  "segment_id": "seg_active_90d_us",
  "start_at": "2026-10-03T16:00:00Z",
  "drop_if_deferred": true
}

The preference model

jsonCopy
{
  "user_id": "u_1001",
  "timezone": "America/Chicago",
  "quiet_hours": { "start": "22:00", "end": "08:00", "applies_to": ["push", "sms"] },
  "channels": { "push": true, "email": true, "sms": true, "in_app": true },
  "categories": {
    "marketing": { "push": true, "email": false, "sms": false },
    "digest": { "email": true }
  },
  "version": 17,
  "updated_at": "2026-09-20T14:03:11Z"
}

Quiet hours apply to push and SMS by default, the channels that buzz a phone. The time zone comes from the device registration (the phone knows it), so it follows the user when they travel.

Frequency-cap policy (one row per category and channel, set by the platform, not by each team):

jsonCopy
{ "category": "marketing", "caps": { "push": 3, "email": 2, "sms": 1 }, "per": "local_day" }

Opt-out, with an audit trail

httpCopy
POST /v1/users/u_1001/opt-outs HTTP/1.1
Content-Type: application/json

{
  "channel": "sms",
  "category": "marketing",
  "source": "sms_keyword_stop",
  "occurred_at": "2026-09-26T21:14:02Z"
}

Every opt-in and opt-out also becomes an append-only consent record: who, what, which channel, when, and how (settings screen, an email's unsubscribe link, a STOP reply). Legal asks "prove this user agreed to marketing texts on this date", and we answer from this log.

In-app inbox. Templates can set "inbox": true. The worker then also writes the message to the user's inbox (DynamoDB, PK = USER#<id>, SK = MSG#<ULID>). A ULID is a 26-character ID that starts with the time, in a fixed-width alphabet, so string order is time order and "newest first" is one query. If the app is open, the WebSocket gateway pushes it live; see WebSocket, SSE & Long Polling and the chat loop for how connections are tracked.

R2.4 Design Evolution: Priorities, Policy, Dedup and the Providers' Limits

Step 2.1: A Login Code Waited 20 Minutes Behind a Marketing Blast

Synthesizing vector architecture diagram...

A 30M-message backlog in the bulk queues can't touch the critical path: different queue, different workers, and a reserved share of each provider's budget.

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance

Step 2.2: We Texted a User Who Opted Out

The hold table

The releaser never relies on TTL: DynamoDB deletes expired items only eventually (typically within a few days). It deletes each item after queueing it. If it crashes in between, the next run queues the item again, and the send marker from step 2.3 stops the second copy. Two releaser runs overlapping behave the same way, so we need no lock.

What the law asks, in general terms. In the US, marketing texts generally need the recipient's prior consent, and a STOP reply must be honored; commercial email must carry a working unsubscribe that is honored promptly (CAN-SPAM); in the EU, GDPR requires a lawful basis for processing, often consent for marketing, and the ability to prove it. The exact rules differ by country and change, so legal owns them; our job is to make every one of them enforceable in one place, with a record.

Step 2.3: The Same Billing Alert Arrived Twice Today

Why Redis fails open for dedup. If the dedup cluster is unreachable, we send anyway and count the skipped checks. A rare duplicate is a smaller harm than a login code that never arrives. (Preferences, in contrast, fail closed for marketing; see R2.8.)

The eviction trap. Redis can evict keys when memory is full. The ElastiCache default policy, volatile-lru, evicts keys with an expiry, least recently used first; volatile-ttl evicts the keys with the shortest remaining TTL first, which here means 2-minute OTP fingerprints and send markers about to expire. Every eviction is a silent duplicate. So this cluster runs with noeviction: when full, writes fail loudly, we fail open and page, and we size so it never happens (alarm at 70% memory).

Primitive: Distributed Cache Patterns & Eviction

Step 2.4: FCM Returns 429 During a Blast, and Our Workers Retry Harder

Why a token bucket, and why shared? Two drill questions live here.

• Local counters on each worker or one shared budget? If each of 20 workers enforces "quota ÷ 20" locally, the total is right only while exactly 20 are running: autoscaling to 30 overshoots by 50%, and a skewed load leaves budget unused on idle workers. A shared bucket in Redis holds the one true count. Leasing 100 tokens at a time keeps it to a few hundred Redis calls a second instead of tens of thousands.
• Token bucket or fixed window? FCM counts over rolling, unaligned minutes. A fixed window of our own ("600K per clock minute") allows 600K in the last second of one minute and 600K in the first second of the next: 1.2M inside FCM's one minute. A token bucket refills continuously; with rate r per second and burst b, the most it can send in any 60 seconds is 60r + b, and we keep that under the quota.

Headers that matter. For APNs: apns-priority: 10 (deliver now) for OTPs and alerts, 5 (deliver when it suits the device's power state) for marketing; apns-expiration set to the OTP's expiry, so Apple never delivers a code after it's useless; apns-collapse-id (at most 64 bytes) so repeats replace each other; and apns-push-type: alert. FCM has the same ideas: Android priority high or normal, a ttl, and a collapse_key. Payloads stay under 4 KB on both.

Primitive: Distributed Rate Limiting · Drill: Rate limiting partner API · Loop: Design a Distributed Rate Limiter

Step 2.5: Millions of Dead Tokens Waste Sends

Step 2.6: Ten Teams Each Sent This User a Push Today

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Follow a request from the top. Producers connect with mutual TLS. Ingest runs the dedup and policy stage (Redis for dedup and caps, DynamoDB for preferences), then either holds the message or puts it on one of nine queues. Each priority has its own worker pool. Workers lease send tokens from the shared buckets, look up devices or contacts, render, and call the provider. Every outcome is packed into Firehose records and lands in S3. Bounces, complaints, delivery receipts and STOP replies flow back into the suppression list.

Why these AWS services. SMS goes through AWS End User Messaging SMS, AWS's SMS service (SNS can also send SMS; End User Messaging gives us the origination numbers, opt-out lists and delivery events directly). Email stays on SES. We do not build on Amazon Pinpoint: it has been closed to new customers since 2025-05-20 and its support ends on 2026-10-30; its SMS, push and voice features continue as AWS End User Messaging. Push goes straight to APNs and FCM from our workers (R2.7 explains why we left SNS).

Trace 1: an OTP during a 50M-user blast

1. At 16:02 the identity service sends login_otp for u_1001 with expires_at 5 minutes out. Ingest finds the category security_code: priority critical.
2. In parallel: SET dedup:<fp> ... NX EX 120 in Redis and a strong read of preferences in DynamoDB. SMS is on, the number isn't on the STOP list. Quiet hours and caps don't apply.
3. Ingest sends to critical-sms and returns 202 about 40 ms after the request arrived.
4. A critical worker receives it within milliseconds: its queue is almost empty while bulk-push holds 30M messages.
5. It takes one token from the SMS bucket's critical share, sets send:<id>:sms to SENDING:<attempt_id> with NX and a 60-second expiry, and calls End User Messaging. Accepted about 600 ms after the request; the marker is overwritten with SENT.
6. The bulk blast is unaffected and unaware.

Trace 2: a marketing push at 23:30 in Chicago

1. The fall-sale campaign reaches u_2002, whose preferences say America/Chicago, quiet hours 22:00–08:00. drop_if_deferred is false.
2. The policy stage computes the release time: 08:00 local plus a random 23 minutes = 13:23 UTC. It writes HOLD#2026-09-27T13:23#17 and returns DEFERRED_QUIET_HOURS.
3. At 13:24 UTC the releaser reads minute 13:23, all 64 shards. The user opted out of marketing push at 07:50; the policy stage now returns DROPPED_OPT_OUT. Nothing is sent, and the hold item is deleted.

Trace 3: a 410 from Apple

1. A bulk worker sends to token ab12... and gets 410 with reason Unregistered and a timestamp of 2026-09-20 10:00 UTC.
2. It writes status = INVALID on that device if last_registered_at is earlier than that timestamp. It is (last registered 2026-08-02), so the write succeeds; 410 isn't counted as an error, and the message is deleted from the queue.
3. The user has a second, active iPhone; the worker had sent to both, and that send succeeded.

Where the audit trail stops. The lake holds the notification ID, user ID, template and version, channel, provider response and timestamps. It does not hold the rendered text or raw addresses: email addresses and phone numbers are stored as an HMAC (a keyed hash), so support can still search "did we text +1 555 0100?" by hashing the number with the same key, and a leaked lake file reveals no contact details.

R2.6 Numbers and Cost

Traffic

Push connections. Apple publishes no per-connection throughput. It says to use many connections for high volume, not to assume how many streams a connection allows, and to spread traffic across connections. So we plan with an assumption: about 500 pushes per second per HTTP/2 connection, to be replaced by what we measure.

FCM quota. Android is the other 55%: 52.7K × 0.55 ≈ 29.0K/s ≈ 1.74M a minute, about 2.9× FCM's default 600K a minute. FCM grants increases of up to 25% at a time, and only when we already use at least 80% of the quota for 5 minutes in a row daily, with under 5% client errors. Three increases take us 600K → 750K → 937.5K → ≈ 1.17M a minute, ≈ 19.5K/s. For a planned event, Firebase Support can grant temporary quota if we ask at least 15 days ahead (30 days above 18M a minute), at most twice a year.

Our limiter runs at 17,500/s with a burst of 17,500: the most it can send in any 60 s is 17,500 × 60 + 17,500 = 1,067,500, under the 1.17M quota. Shares: critical 1,000/s, high 8,500/s (Android high traffic peaks at 23.5K × 65% × 55% ≈ 8.4K/s), bulk 8,000/s guaranteed plus whatever the others leave.

What that means for a 50M-user push. Android recipients: 50M × 55% = 27.5M.

So the "within 15 minutes" promise holds for campaigns up to 8,000 × 900 = 7.2M Android recipients, about 13M users in total. For bigger ones, marketing either starts earlier, or we book temporary FCM quota 15 days ahead. That's a conversation to have before the campaign, not during it. (The iOS half is not the limit: 22.5M ÷ 900 s = 25K/s, inside our 32K/s.)

Other provider limits. SES quotas are per account and per Region: we need about 500M emails a day and a send rate well above our 5,787/s average, which means a negotiated increase with AWS; bulk email is shaped to whatever rate is granted. US SMS throughput is per sender: a short code starts at 100 message parts a second, so 8,100/s at peak needs raised throughput on several short codes; each sender gets its own bucket. REL 1

Other daily volumes we price below (assumptions, stated so the cost lines can be checked):

Dedup memory. The track's formula, then a correction:

64 bytes per key is a floor: a 32-byte binary key plus a TTL plus one hash-table entry. Real Redis keys cost more: the key string's header and allocator rounding (a 36-byte string takes a 48-byte slot), the main table entry and its bucket, and a second entry in the table of keys with an expiry. The dedup key also stores a value, the ~30-character notification_id, which takes about 64 bytes more as a small string object. We plan with about 200 bytes per dedup key and about 128 bytes per send marker (both to be confirmed with MEMORY USAGE on a test cluster):

We size for the worst case, because the window is the producer's choice, not ours. Dedup cluster: 15 shards, each a primary and one replica in another AZ, on cache.r7g.2xlarge (52.82 GiB). ElastiCache reserves 25% of memory by default, leaving ≈ 39.6 GiB per shard, ≈ 594 GiB in all: the worst case fills 407 ÷ 594 ≈ 69%, just under our 70% alarm, which is why the alarm is there. Caps cluster: about 300M users receive bulk each day (assumption), ≈ 1.5 capped counters each, ~100 B per counter: 450M × 100 B ≈ 45 GB ≈ 42 GiB for one day of keys; 2 shards give 79 GiB usable (53%).

Audit lake

Firehose bills every record in 5 KB steps, so we pack events before sending: nine times fewer records, one ninth of the bill.

OTP latency budget (P99). Parallel steps cost the slower of the two, not the sum. Adding P99s is pessimistic; if the sum fits, the real P99 fits.

That's inside 3 s with a retry to spare. What we promise is handed to the provider; how long a carrier takes to reach the handset isn't ours to promise, and we say so.

Monthly cost (us-east-1 list prices, 30.4 days; SMS priced as US 10DLC for illustration):

SMS is 97% of the bill. Email is about 3%. Everything we built, queues to caches to the lake, is about $92K, or 0.17%. The most valuable cost decision in this system isn't a cache size: it's moving a message from SMS to push or email whenever the user and the law allow, because at the illustrative $0.0088 one SMS costs as much as about 88 emails. COST 5

R2.7 Trade-Offs

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Sees head-of-line blocking and isolates priorities end to end: queues, workers, connections and provider budget.
• Puts preferences, opt-outs, caps and quiet hours in one policy stage, and re-checks held and redriven messages.
• Separates "same request" (idempotency key) from "same message" (dedup key and window), and sizes the memory for the worst case the API allows.
• Knows each provider's limits and signals: FCM's per-project quota and how it grows, APNs' per-token 429 and its 410 timestamp.
• Counts things once, in a place that isn't at-least-once.
• Finds the real cost: SMS, not infrastructure.

Follow-up questions

1. "Why not let producers mark a message critical when it's really urgent?" Answer: because every team's message is "really urgent" to that team. Priority comes from the template's category, approved by a reviewer. A team that needs a critical template asks for one, and the critical pool stays small enough to keep its promise.

2. "A user changes time zone mid-flight. What happens to held messages?" Answer: the hold time was computed when the message was accepted. On release, the policy stage runs again with the new time zone; if the user is now in quiet hours, the message is held again. The device registration updates the time zone on the next launch.

3. "Could you send OTPs by push first and fall back to SMS, to save money?" Answer: yes, and at roughly $0.0088 per US text (illustrative; carrier fees vary) it's worth it: send the push with a short expiry; if the app hasn't confirmed receipt within ~5 seconds, send the SMS. It needs the app's receipt beacon, and it adds 5 seconds for users whose push doesn't arrive, which fits inside a 5-minute code.

Interview gotchas from this round

---

Round 3 · Architect · "Global, Multi-Tenant, and Deliverability"

~45 min · Principal (L7) · 3 legal zones, 6 regions · ~10B notifications/day · ≈ 116K/s average, ≈ 405K/s peak · per-tenant SLOs · no tenant can degrade another

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run the company's notification platform in one region: 2 billion a day, about 23,000 a second on average and 81,000 at peak, 65% push, 25% email, 10% SMS, 99.99%. Priority comes from each template's reviewed category: critical, high or bulk. Each priority has its own queues, worker pool and reserved share of every provider's rate, so a 50-million-user blast can't delay a login code; OTPs are handed to the provider in about 650 ms at P99. One policy stage at ingest applies suppression, preferences, frequency caps and quiet hours in the user's time zone, holds messages until morning, and re-checks them on release. A producer-chosen dedup key, hashed with user, channel and template, is set in Redis with a window of up to 24 hours; we sized that for the worst case, about 400 GB. Workers lease tokens from per-provider buckets, keep long-lived HTTP/2 connections to APNs and FCM, and clean tokens on 410 and UNREGISTERED. It costs about $55 million a month, 97% of it SMS; our own infrastructure is about $92,000. Open costs: one region, one company's reputation, one SMS route per country, and nobody outside the company can use it."

Architecture v2, compact

textCopy
 teams ─mTLS─► ALB ─► ingest: dedup (Redis SET NX EX) → suppression + prefs (strong read) → caps (INCR) → quiet hours (hold table)
                        └─► SQS: {critical, high, bulk} × {push, email, sms} + DLQs
 worker pools per priority ─lease tokens (per-provider buckets, reserved shares)─► APNs · FCM · SES · End User Messaging SMS
 outcomes ─packed─► Firehose ─► S3 (5 years)      bounces · complaints · STOP ─► suppression list

Steps so far

Open costs: one region; one set of sending identities; one SMS route per country; one customer.

R3.1 The Scope Raise

Interviewer: "We now sell this platform to other companies. About 10 billion notifications a day across all tenants, in North America, Europe and Asia. One tenant's spammy campaign must not get everyone's email sent to spam. We send SMS to about 100 countries, each with its own carriers and rules. EU tenants' data must stay in the EU. A provider or a whole region can go down. And tenants want delivery analytics."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Tenants and API keys. Every call carries a tenant API key. We store only a hash of it, scoped to one tenant and one environment (test or live), and support two active keys so tenants can rotate without downtime.

httpCopy
POST /v1/notifications HTTP/1.1
Host: api.eu.notify.example
Authorization: Bearer ntk_live_7Hq...
Idempotency-Key: 5f1c...
Content-Type: application/json

{ "user_id": "cust-88", "template_id": "login_otp", "variables": { "code": "482913" } }

The host name names the tenant's legal zone. An EU tenant's keys only work on api.eu....

Tenant settings

jsonCopy
{
  "tenant_id": "t_acme",
  "plan": "enterprise",
  "home_zone": "EU",
  "home_region": "eu-west-1",
  "partner_region": "eu-central-1",
  "quotas": { "critical_per_sec": 500, "high_per_sec": 5000, "bulk_per_sec": 20000, "bulk_per_day": 50000000 },
  "email": { "ses_tenants": ["acme-transactional", "acme-marketing"], "ip_pool": "dedicated-acme" },
  "push": { "apns_key_secret": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:t_acme/apns", "fcm_project": "acme-prod" }
}

Sending identities. A tenant adds its domain and gets DNS records to publish:

httpCopy
POST /v1/tenants/t_acme/sending-domains HTTP/1.1
Content-Type: application/json

{ "domain": "mail.acme.example" }

jsonCopy
{
  "domain": "mail.acme.example",
  "dns_records": [
    { "type": "CNAME", "name": "abc1._domainkey.mail.acme.example", "value": "abc1.dkim.amazonses.com" },
    { "type": "MX", "name": "bounce.mail.acme.example", "value": "10 feedback-smtp.eu-west-1.amazonses.com" },
    { "type": "TXT", "name": "bounce.mail.acme.example", "value": "v=spf1 include:amazonses.com ~all" },
    { "type": "TXT", "name": "_dmarc.mail.acme.example", "value": "v=DMARC1; p=none; rua=mailto:dmarc@acme.example" }
  ],
  "status": "PENDING_VERIFICATION"
}

(SES's Easy DKIM actually gives three DKIM CNAMEs; one is shown.) SMS senders are registered per tenant and country (POST /v1/tenants/t_acme/sms-senders), because in the US and many other countries the tenant's business, not ours, is the registered sender.

Webhooks

httpCopy
POST /v1/tenants/t_acme/webhooks HTTP/1.1
Content-Type: application/json

{ "url": "https://hooks.acme.example/notify", "events": ["delivered", "bounced", "complained", "opened", "clicked"] }

Each delivery to the tenant is signed:

httpCopy
POST /notify HTTP/1.1
Host: hooks.acme.example
Content-Type: application/json
X-Notify-Signature: t=1790445735,v1=5d41402abc4b2a76b9719d911017c592...

{ "events": [ { "event_id": "evt_01J9...", "type": "delivered", "notification_id": "ntf_...", "channel": "email", "occurred_at": "2026-09-26T18:02:13Z" } ] }

v1 is an HMAC-SHA256 of t + "." + body with the tenant's webhook secret. The tenant recomputes it and rejects the call if it doesn't match or if t is more than 5 minutes old (so a captured request can't be replayed later). Delivery is at least once; tenants deduplicate by event_id.

R3.4 Design Evolution: Tenants, Reputation, Routes, Regions and Events

Step 3.1: One Tenant's Blast Starved Everyone

The two tenant-hotspot questions. Your biggest tenant is about to grow 10×; what happens? In a shared pool, they'd become the permanent noisy neighbor, and the fair queue would keep everyone else safe but slow them down. We move them to a dedicated cell before the migration, with a routing override (tenant → cell) that's a configuration change, not a data migration: new messages go to the new cell, and the old queues drain. How do you run a report across all tenants? Not by querying 5,000 tenants' partitions one by one from the live system (a scatter-gather that loads production). Platform-wide reports read the analytics lake with Athena: eventually consistent, minutes behind, and nowhere near the send path.

Primitive: Distributed Rate Limiting · Database Sharding & Partition Keys · Drill: Sharding tenant hotspot

Step 3.2: One Tenant's Spam Hurt Everyone's Inbox Placement

How the brake works. SES publishes every bounce and complaint through the tenant's configuration set; our event handler keeps a rolling rate per SES tenant; crossing our threshold flips the tenant's marketing to paused in our policy stage (new campaigns get 403 TENANT_MARKETING_PAUSED), and we can also pause the SES tenant directly (R3.9). SES's own reputation findings arrive as EventBridge events, so an SES-side pause reaches our dashboards within seconds.

Step 3.3: SMS to Brazil Is Failing on One Route

The two thread-stall questions. Why does a slow provider exhaust our workers even though it returns no errors? Each in-flight call holds a worker slot for its whole duration. By Little's law, calls in flight = arrival rate × latency: at 1,000 sends a second, 200 ms latency means 200 in flight, but 5 s means 5,000, and a pool of 1,000 slots is gone. The route's own bulkhead (a fixed slot pool per route) keeps that stall inside the route, and the breaker moves traffic away. Why not just set a 50 ms timeout on every provider call? A healthy SMS API often takes a few hundred ms, so a 50 ms timeout would fail healthy calls; each failed call becomes a retry, adding load exactly when things are slow; and for SMS, a timeout after the provider accepted means a duplicate text. Timeouts set just above the normal P99 catch hung calls; the breaker, looking at the error and delivery rates over a window, decides when a route is unhealthy, and half-open probes decide when it's back.

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance · Drill: Circuit breaker cascading thread stall

Step 3.4: EU Data Must Stay in the EU, and a Region Can Fail

Conflicts during a failover. DynamoDB global tables check a condition only against the local copy, and by default resolve concurrent writes to the same item with last writer wins. Normally each tenant writes only in its home region, so there's no conflict. During a failover, both regions can write: a user opts out in the partner region while the home region, still half-alive, writes an older opt-in to the same item. Last writer wins can then keep the opt-in. Two rules prevent that:

• Consent is append-only. Every opt-in or opt-out is a new item (CONSENT#<occurred_at>#<event_id>), never an overwrite, so replication can't lose one. The current state is computed from the log.
• Opt-out wins ties. If an opt-in and an opt-out for the same channel and category are within 5 minutes of each other (more than any clock skew we expect), the opt-out wins. Suppression entries are add-only during a failover.

The replication-lag questions. A user opts out in Frankfurt, and a second later a campaign in Ireland texts them anyway: what went wrong? Asynchronous replication: Ireland read its local copy before the opt-out arrived, typically under a second behind. For marketing in normal operation this can't happen, because each tenant's traffic runs only in its home region, where the opt-out was written; during a failover, the window exists, which is why marketing doesn't fail over. Why not replicate every write synchronously? Every write, including 10 billion notifications' worth of status, would wait for another region, and a partner outage would block the healthy region. We scope strong consistency narrowly: if legal asks for guaranteed cross-region consent, DynamoDB's multi-Region strong consistency could hold just the consent table; it runs in exactly three Regions and adds cross-region latency to each consent write, a fine price for a rare write.

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active · Drill: Replication multi-region consistency

Step 3.5: Tenants Want Delivery Analytics and Webhooks

Primitive: Change Data Capture & the Outbox Pattern · Message Queues vs Event Streams

Step 3.6: Should We Build the Providers Ourselves?

Pinpoint, precisely. Amazon Pinpoint closed to new customers on 2025-05-20, and its support ends on 2026-10-30 (about a month from now). Its SMS, voice, push and WhatsApp capabilities continue as AWS End User Messaging, and email stays on SES. Campaign and journey features have no direct successor in that family, so anything built on them must move. We don't propose Pinpoint for anything new.

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Read it zone by zone. Each zone is a pair of regions running the full pipeline, with its tenant data replicated only inside the pair. A tenant's traffic normally runs only in its home region; the partner holds a copy of its settings, preferences, consent and tokens, and spare capacity for its critical and high traffic. Delivery fans out to SES (per-tenant SES tenants and IP pools), an SMS router with several routes per country, push with the tenant's own credentials, and WhatsApp. Events flow to the tenant's webhooks and to a tenant-partitioned lake in the same zone. The APAC zone has the same layout.

Trace 1: a bad tenant's campaign is stopped

1. A self-serve tenant, t_newco, uploads 400,000 addresses and starts a campaign at 10:00. Their quota caps them at 200 a second, so the campaign takes over half an hour; the fair queue keeps other self-serve tenants' mail moving.
2. In the first 90 seconds, 18,000 messages go out. Bounce events for them arrive within minutes: by 10:06, 1,100 hard bounces (6.1%) and 21 complaints (about 0.12%).
3. Our rolling rate crosses 4% bounces. The policy stage pauses t_newco's marketing: queued messages are dropped at send time with TENANT_MARKETING_PAUSED, new campaign sends get 403, and their account manager is notified.
4. The SES tenant newco-marketing was also flagged by SES's reputation policy; because it sits in the self-serve AWS account, no enterprise tenant's sending was at risk.
5. t_newco's password resets (SES tenant newco-transactional) keep flowing.

Trace 2: SMS route failover for Brazil

1. At 09:00 route A's delivered-within-60-s share for Brazil drops from 97% to 40%. API calls still succeed.
2. At 09:05 the 5-minute window confirms it; route A's breaker opens for Brazil. OTPs move to route B, where every tenant sending OTPs to Brazil was registered at onboarding.
3. OTPs sent on route A between 09:00 and 09:05 whose receipts never came are re-sent on route B with the same code. Some users get the code twice.
4. At 09:20 the breaker lets 1% of Brazilian bulk traffic probe route A (half-open); at 10:10 its delivery share is back to normal, and the breaker closes.

Trace 3: an EU tenant during a regional outage

1. eu-west-1 has a severe outage at 14:00. Route 53 health checks on api.eu... fail, and within a couple of minutes the name resolves to eu-central-1.
2. t_acme's login-code requests arrive in Frankfurt. Preferences, consent, suppression and tokens are there (replicated a second or so behind). The partner's reserved capacity serves their critical and high traffic.
3. Their bulk requests get 503 BULK_PAUSED_FAILOVER; the campaign tool retries later.
4. Messages already queued in eu-west-1 wait there. When the region recovers, they drain; the policy stage re-checks each (expired OTPs are dropped, opt-outs made in Frankfurt during the outage have replicated back and are honored).
5. No EU data went outside the EU at any point, apart from the message itself going to Apple, Google, SES's EU endpoints and carriers, which every tenant's data-processing agreement lists.

R3.6 Numbers and Cost

Traffic

Each region is Round 2's design at Round 2's scale or a little above: the largest home region carries about 1.25× Round 2's 81K/s peak, plus failover headroom. Its dedup cluster is sized with the same method: its daily volume × 128 bytes for the worst case.

Webhook volume

Dedicated IPs. Assume 400 enterprise tenants with 3 dedicated IPs each: 1,200 × \$24.95 ≈ \$30K a month.

Monthly cost. Two very different kinds of money: fees we pass through to tenants (per message, charged to the tenant who sent it), and our own platform.

Where the money is. Provider fees are about 99.8% of what flows through the system; our platform is about $0.56M a month, 0.2%. That shapes the business: we charge tenants the provider fees plus a margin, and our costs scale with orchestration, not delivery. It also shapes the engineering: routing SMS by cost among healthy routes (step 3.3) moves more money than any infrastructure tuning could. COST 3

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we get the right message to the right person once, at the right time, through carriers we don't control? The answer is now:

• The right message: templates with reviewed categories, rendered from versions that never change.
• To the right person: one owner per device token, contact details kept clean by the providers' own answers, and consent recorded as facts that replication can't lose.
• Once: idempotency keys for requests, dedup keys with windows for logical messages, send markers for redeliveries, collapse IDs on devices, and honesty about the one case no one can close: a timeout after the provider accepted.
• At the right time: priorities with their own lanes, quiet hours in the user's time zone, caps across teams, and fairness across tenants.
• Through carriers we don't control: we learn each carrier's rules and stay under them, score routes by what actually arrives, keep each tenant's reputation its own, and fail over only where the law allows.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per region and per tenant OPS 8 · REL 6

Emergency: stop all marketing now (a reputation incident, a legal instruction, a bad template sent to everyone). Each step can be repeated safely. OPS 10

1. Set the bulk-pause flag (CLI 3). The fan-out job stops expanding campaigns, and bulk workers stop receiving within a few seconds. Messages stay in the queues.
2. If one campaign's queued messages must never be sent (for example, a wrong price in a template), cancel the campaign (CLI 4). Bulk workers keep a list of cancelled campaigns, refreshed every few seconds, and drop that campaign's messages at send time with CAMPAIGN_CANCELLED: the same mechanism as TENANT_MARKETING_PAUSED in Trace 1. Never purge a shared bulk queue: there are no per-campaign queues, so it holds every tenant's marketing. A purge is only for a queue in a giant tenant's dedicated cell, when everything in it must go (CLI 4b).
3. If one tenant is the cause, pause its SES marketing tenant directly (CLI 7) as well.
4. When it's fixed: clear the flag; bulk resumes at its shaped rate.

DLQ redrive always targets the intake queue, which runs the policy stage again: expired OTPs are dropped, new opt-outs honored, and each message's redrive count is increased; at 2 it goes to quarantine instead.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace names, account IDs and times with real ones.

textCopy
# 1. How old is the oldest OTP waiting in eu-west-1? (age is a CloudWatch metric, not a queue attribute)
aws cloudwatch get-metric-statistics --region eu-west-1 --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage --dimensions Name=QueueName,Value=critical-sms --statistics Maximum --period 60 --start-time 2026-09-26T14:00:00Z --end-time 2026-09-26T14:30:00Z

# 2. How many messages are waiting and in flight in a bulk queue?
aws sqs get-queue-attributes --region eu-west-1 --queue-url https://sqs.eu-west-1.amazonaws.com/111122223333/bulk-push --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# 3. Pause all marketing in this region (our own flag, read by fan-out and bulk workers)
aws ssm put-parameter --region eu-west-1 --name /notify/bulk/paused --value true --type String --overwrite

# 4. Cancel one campaign: bulk workers drop its queued messages at send time (other campaigns and tenants untouched)
aws ssm put-parameter --region eu-west-1 --name /notify/campaigns/t_acme/fall-sale-2026/cancelled --value true --type String --overwrite

# 4b. IRREVERSIBLE, dedicated cells only: delete every message in one giant tenant's cell bulk queue (never a shared queue, never critical or high)
aws sqs purge-queue --region eu-west-1 --queue-url https://sqs.eu-west-1.amazonaws.com/111122223333/cell-acme-bulk-push

# 5. Move fixed messages from a DLQ back through the policy stage, at a gentle rate
aws sqs start-message-move-task --region eu-west-1 --source-arn arn:aws:sqs:eu-west-1:111122223333:critical-sms-dlq --destination-arn arn:aws:sqs:eu-west-1:111122223333:redrive-intake --max-number-of-messages-per-second 50

# 6. Check the redrive's progress
aws sqs list-message-move-tasks --region eu-west-1 --source-arn arn:aws:sqs:eu-west-1:111122223333:critical-sms-dlq

# 7. Pause one customer's marketing SES tenant
aws sesv2 update-reputation-entity-customer-managed-status --region eu-west-1 --reputation-entity-type RESOURCE --reputation-entity-reference arn:aws:ses:eu-west-1:111122223333:tenant/newco-marketing/tenantId --sending-status DISABLED

# 8. Look at an SES tenant's status and its open reputation findings
aws sesv2 get-tenant --region eu-west-1 --tenant-name newco-marketing
aws sesv2 list-recommendations --region eu-west-1 --filter '{"RESOURCE_ARN":"arn:aws:ses:eu-west-1:111122223333:tenant/newco-marketing/tenantId"}'

# 9. Is the SES account itself healthy? (enforcement status, send quota)
aws sesv2 get-account --region eu-west-1

# 10. Account bounce rate over the last day
aws cloudwatch get-metric-statistics --region eu-west-1 --namespace AWS/SES --metric-name Reputation.BounceRate --statistics Maximum --period 3600 --start-time 2026-09-25T14:00:00Z --end-time 2026-09-26T14:00:00Z

# 11. Numbers that replied STOP to a tenant's senders
aws pinpoint-sms-voice-v2 describe-opted-out-numbers --region eu-west-1 --opt-out-list-name t-acme

# 12. Stop the bulk push workers entirely (messages stay queued)
aws ecs update-service --region eu-west-1 --cluster notify-euw1 --service bulk-push-workers --desired-count 0

A purge can take up to 60 seconds to finish, and a queue can't be purged again within 60 seconds. End User Messaging SMS still uses the pinpoint-sms-voice-v2 name in the CLI; that's the SMS service, not Pinpoint's campaign features.

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Treats every shared resource as a way one tenant can hurt another: capacity, provider quota, sending reputation, data, analytics. Isolates each, and names the cost.
• Knows how inbox reputation actually works (domains, IPs, SPF/DKIM/DMARC, complaint rates, bulk-sender rules) and builds automatic brakes below the providers' own thresholds.
• Measures SMS routes by delivery, not by API success, and ties failover to sender registration.
• Designs residency first and failover inside it, and explains what global tables do with conditions and conflicts, including during a failover.
• Separates pass-through costs from platform costs, and decides build vs buy per layer, including a vendor's end of life.

Follow-up questions

1. "A tenant wants to move from the US zone to the EU zone." Answer: it's a migration, not a setting. We create the tenant in the EU zone, copy its settings, preferences, consent log, suppression and tokens there, set up EU SES tenants and verify its domains in the EU regions, switch its API keys to the EU host, then drain and delete the US data. Its lake history moves too, or is deleted, as its contract says.

2. "A tenant says a user never got their code. How do you answer?" Answer: from the tenant's event stream and our audit: accepted at, which priority and route, handed to the provider at, and the provider's receipt (or none). If the SMS route shows no receipt, the breaker history for that country and time tells us whether it was a route problem.

3. "Why not let big tenants run their own SES accounts and just orchestrate?" Answer: some will want exactly that, and it's a valid enterprise option ("bring your own sending account"): their reputation becomes fully theirs. The costs are onboarding complexity, per-account quotas they must manage, and that our brakes become advice. We'd offer it to the largest tenants, not as the default.

Interview gotchas from this round

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. For the queue underneath all of this, see Design a Distributed Message Queue.

The Two Sentences That Matter Most

• Opening a round: "Before I design: what triggers a message, which channels, how fast must it be handed over, and how bad is a duplicate compared with a missed message?"
• When the scope is raised: "Here's what breaks, and I'll fix it in this order: anything that sends the wrong thing to the wrong person, then anything that delays login codes, then the carriers' limits, then cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels