Design a Distributed Message Queue

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is a Message Queue?

You Already Use One: a To-Do List Between Two People

One person writes tasks on a shared list. Another person picks one up whenever they are free, does it, and crosses it off. The writer never waits for the reader, and if twenty tasks arrive at once, they simply sit on the list until someone gets to them.

Services do the same thing. The order service writes "ship order 42" and moves on. A fulfillment worker picks it up a moment later, or an hour later if it is busy.

A message queue is that list, run as a service. What it buys us:

• The producer doesn't wait. Sending a message takes milliseconds, even if the work takes minutes.
• Bursts don't crush the consumer. The queue absorbs a spike, and consumers work through it at their own pace.
• Either side can fail without taking the other down. If the workers crash, messages wait for them.

What Changes When the List Is Distributed

On one machine, the list is easy. Once it spans many machines, three things get hard:

• The list must survive machines dying. A message the queue said "got it" to must not vanish when a disk or a server does.
• Two workers must not both take the same task at the same time, at least not on purpose.
• A task a worker dropped must come back. If a worker takes a message and crashes, somebody else has to get it.

The Question the Whole Loop Answers

How do we make sure every message is delivered, none are lost, and order holds where we promised it, while machines and consumers fail?

The answer gets sharper every round:

• Round 1: a replicated, partitioned queue. Messages are leased to one worker at a time and come back if the worker dies.
• Round 2: order per key, no duplicates from retries, and many teams reading the same messages, which turns the queue into a log.
• Round 3: regions, laws and money, and whether we should run this at all.

---

Round 1 · Mid-level · "A Task Queue for One Team's Order Pipeline"

~35 min · SDE II (L5) · 1 region, 3 AZs · ~2K msg/s peak · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Design a distributed message queue." Before we draw anything, we ask questions, and we say out loud what each answer changes.

Out of scope for this round:

• Ordering. Nobody needs it yet.
• Several teams reading the same messages. One team, one queue.
• Replaying old messages. A processed message is gone.
• Other regions. One AWS region, three Availability Zones (AZs).

The agreement that makes the whole design possible is in two rows of that table: at-least-once delivery, plus idempotent consumers. Everything below leans on it.

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into a requirement:

Not yet: ordering, removing duplicates, delaying a message, many teams reading one message, replay. The interviewer may bring these back.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the qualities and why each matters:

• Availability. The producer must never be blocked. If the queue can't take a message, the order service can't hand off work, and customers wait.
• Durability. Once we say "got it" to the producer, the message must survive any single machine failing, including its disk.
• Latency. Close to real time: a waiting worker should see a message within a fraction of a second of it being sent.
• Scalability. If traffic grows, we add brokers (the servers that store and hand out messages) and get more throughput.
• Delivery. At-least-once, as agreed.

Why not exactly-once? Picture the last step of a task: the worker ships the order, then calls delete. If the worker crashes between the two, the queue can't tell whether the order shipped, so it must hand the message out again, or risk losing it. If instead the worker deletes first and then ships, a crash between the two loses the task. No choice of order inside the queue closes this gap, because the queue and the worker's database are two different systems. "Exactly once" only happens when the worker makes its own effects idempotent, which ours already are. The queue's job is "at least once, and duplicates are rare".

Delivery semantics are set by when each side acknowledges, not by the queue alone:

R1.4 The API

Three calls, internal only, over HTTPS.

Send

httpCopy
POST /v1/queues/order-fulfillment/messages HTTP/1.1
Host: mq.internal
Content-Type: application/json

{
  "body": "{\"orderId\": \"42\", \"action\": \"ship\"}",
  "attributes": { "source": "order-service" }
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "messageId": "m-3-18442" }

A 200 means the message is stored on a majority of its replicas (step 1.3). Before that, the answer never comes.

Receive

httpCopy
POST /v1/queues/order-fulfillment/messages:receive HTTP/1.1
Host: mq.internal
Content-Type: application/json

{ "maxMessages": 10, "waitSeconds": 20, "visibilitySeconds": 30 }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "messages": [
    {
      "messageId": "m-3-18442",
      "receiptHandle": "rh.3.18442.1.Zk9wQ2xhVGhlU2ln",
      "body": "{\"orderId\": \"42\", \"action\": \"ship\"}",
      "receiveCount": 1,
      "sentAt": "2026-09-26T12:00:00.000Z"
    }
  ]
}

• waitSeconds (0 to 20) is how long the broker may hold the request open if there's nothing to return yet (step 1.5).
• visibilitySeconds is how long this message stays hidden from other workers while this worker processes it (step 1.2).
• The receipt handle identifies this particular receive of the message, not just the message. If the message is received again later, it gets a new handle. Round 1's broker only uses the handle to find the message; Round 2 makes it enforce the lease.

Delete (acknowledge)

httpCopy
DELETE /v1/queues/order-fulfillment/messages/rh.3.18442.1.Zk9wQ2xhVGhlU2ln HTTP/1.1
Host: mq.internal

httpCopy
HTTP/1.1 204 No Content

Recap

• Three calls: send, receive with a wait and a visibility time, delete.
• At-least-once delivery; workers are idempotent.
• About 50 million messages a day, around 2 KB each, never more than 256 KB.
• Unprocessed messages kept up to 4 days.
• A "got it" means the message survives losing any one machine.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From an In-Memory List to a Replicated Queue

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

One server holding a list in memory. send appends to the list. receive removes the first item and returns it.

Synthesizing vector architecture diagram...

What's good about it: it's fast and simple, and for a prototype it works.

What it costs us: everything. A restart loses every message. A worker that crashes loses the message it took. One machine is all the capacity there is.

Step 1.1: The Broker Restarted and the Messages Are Gone

The log for one queue, on disk:

textCopy
queue order-fulfillment, log 3
  00000000000000000000.log     ← segment: messages 0 .. 18,399
  00000000000000000000.index   ← every ~4 KB: offset → byte position
  00000000000000018400.log     ← active segment: new records are appended here
  00000000000000018400.index

one record in a segment:
┌────────┬───────┬───────────┬──────────┬────────────┬──────────────┬─────────┐
│ length │ CRC32 │ type      │ offset   │ timestamp  │ attributes   │ body    │
│ 4 B    │ 4 B   │ msg / ack │ 8 B      │ 8 B        │ variable     │ ≤256 KB │
└────────┴───────┴───────────┴──────────┴────────────┴──────────────┴─────────┘

The CRC32 is a checksum of the record, so a half-written record at the end of the file after a crash is detected and discarded. An ack record has no body; it names the offset it acknowledges.

Primitive: Write-Ahead Log & LSM-Trees

Step 1.2: A Worker Took a Message and Crashed

Synthesizing vector architecture diagram...

A message is only ever in one of these states. The only way out of "in flight" without a delete is back to "visible", which is the whole point.

Primitive: Distributed Locks & Leases

Step 1.3: The Broker's Disk Died

Synthesizing vector architecture diagram...

The leader answers after the fastest follower. A slow or dead third replica doesn't slow the send.

Go deeper: Raft versus Kafka's ISR. Kafka replicates differently, and it's worth being able to say how. Kafka's leader keeps a list of in-sync replicas (ISR): followers that have kept up recently. With acks=all, the leader answers only after every replica in the ISR has the message, and min.insync.replicas=2 refuses writes if fewer than two are in sync. A follower that falls behind (by default, for 30 s) is dropped from the ISR instead of slowing everyone down. Kafka uses Raft too, but only for its cluster metadata (KRaft), not for the messages. Both designs survive losing one of three replicas without losing acknowledged data; they differ in how they treat a slow replica and how a leader is chosen. We use Raft for every log, as some Kafka-compatible brokers such as Redpanda do, because the majority rule gives one clear answer to "when is a message safe?" and one clear rule for elections.

Primitive: Distributed Consensus: Raft & Paxos · Drill: Raft consensus split brain

Step 1.4: One Broker Can't Take All the Traffic

Primitive: Database Sharding & Partition Keys

Step 1.5: Workers Poll Constantly and Mostly Get Nothing

Step 1.6: One Bad Message Crashes Every Worker, Forever

Drill: Message queue order pipeline

Round 1 Step Summary

Two costs are still open: there's no ordering, and only one team can read a queue, since a message is gone once one worker deletes it. Round 2 starts there.

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Follow a message from the top: the order service sends through the load balancer to whichever broker it lands on, and that broker appends it to a partition it leads and replicates it to the other two AZs. Workers skip the load balancer: they learn the brokers' addresses from the metadata store once, then hold a long poll open to each broker. The metadata store is off the path of every send and receive.

The pieces:

• Brokers: three EC2 m7g.large instances (2 Graviton vCPUs, 8 GiB), one per AZ. Each stores the segment logs of all 6 partitions on a 1 TB gp3 EBS volume, leads 2 partitions and follows 4.
• Internal Network Load Balancer (NLB): a layer-4 load balancer that passes TCP connections through. It spreads producers' connections across brokers and health-checks them.
• Metadata store: three small instances (m7g.medium) running one Raft group. It records the queues, their settings (visibility timeout, maxReceiveCount, retention), which brokers hold each partition, and the current leaders.
• Client library: used by producers and workers. It retries failed sends with backoff, keeps one connection per broker for receives, and refreshes broker addresses from the metadata store when a broker stops answering.
• CloudWatch: metrics and alarms, including backlog depth, age of the oldest message and DLQ depth.

Trace 1: a send.

Synthesizing vector architecture diagram...

The producer waits for one cross-AZ round trip and two fsyncs (the leader's and the first follower's), which run at about the same time.

Trace 2: a receive, a lease, and a delete.

Synthesizing vector architecture diagram...

Trace 3: the worker crashes. The worker receives m-1-90211 at 12:00:00 with a 30-second lease and is killed at 12:00:05. Nothing happens until 12:00:30, when broker 2's lease deadline passes and the message becomes visible again. The next receive, from any worker, gets it with receiveCount = 2 and a new receipt handle. Order 42 ships once the second worker finishes; if the first worker had in fact shipped it before dying, the second worker sees that and does nothing, because workers are idempotent.

R1.7 Numbers

Traffic. A day has 86,400 seconds. We plan the peak at 3.5 times the average, the same ratio Round 2 uses.

(We use decimal units throughout: 1 KB = 1,000 bytes, 1 GB = 10⁹ bytes.)

Storage, worst case. Normally workers keep up, and the backlog is minutes of messages. We size for the case we promised to survive: workers down for the full 4 days.

$$100 \text{ GB/day} \times 4 \text{ days} \times 3 \text{ replicas} \times 1.2 = 1{,}440 \text{ GB} \approx 1.44 \text{ TB}$$

The 1.2 covers record headers, ack records, indexes and free space. Each broker holds a third: 480 GB, on a 1 TB volume, so even the worst case leaves the disk under half full.

Partitions. How much one partition can take is a design choice, not a law. We set a planning ceiling of 1,000 msg/s per partition, low on purpose, so each partition's fsync rate and lease bookkeeping stay small.

$$\text{partitions} = \frac{2{,}025 \text{ msg/s}}{1{,}000 \text{ msg/s per partition}} \approx 2.0 \;\Rightarrow\; 3 \text{, doubled for headroom} \;\Rightarrow\; 6$$

Six partitions over three brokers: each leads two.

Disk operations. Each log fsyncs at most once per 2 ms group-commit window, so at most 500 times a second. A broker holds 6 logs, so at most $6 \times 500 = 3{,}000$ fsyncs a second, plus the data writes themselves. gp3 gives 3,000 IOPS (disk operations per second) as a baseline, so we provision 6,000. At our real peak, each log sees about $2{,}025 / 6 \approx 340$ messages a second, so the true fsync rate is lower.

Brokers. Three is the minimum for 3 replicas in 3 AZs, and it's far more than the load needs: 4 MB/s in is a few percent of one m7g.large. After losing a broker, the other two lead all 6 partitions without strain.

Availability budget. 99.9% of a 30.4-day month is $43{,}776 \text{ min} \times 0.001 \approx 43.8$ minutes of downtime. A broker failure costs its partitions 1 to 2 seconds of failed sends, which producers retry, so the budget is really for mistakes: a bad deploy, a misconfigured queue.

Latency budget: P99 from send to a waiting worker under 100 ms.

The leader's fsync and the follower round trip run at the same time, so only the slower of the two counts. These are estimates, which we confirm with a load test before launch. They leave about 11 times headroom under the 100 ms target. The target is for a waiting worker: if every worker is busy, a message waits in the backlog, and that is queueing, not latency we can fix in the broker.

Monthly cost (us-east-1 on-demand prices, 730 hours a month, 3,040 GB of messages a month):

The same workload on Amazon SQS. SQS charges per request: $0.40 per million for standard queues, and each 64 KB of payload counts as one request. Every message needs a send, a receive and a delete.

Long polling's empty receives add a few dollars at most: 60 idle workers each waiting 20 seconds make about 7.9 million requests a month even if the queue were empty all month.

Say it plainly in the interview: at this scale, the broker fleet costs about the same as SQS, and SQS comes with no brokers to patch, no disks to watch and no on-call rotation. The fleet bill isn't the real cost; the people who run it are.

R1.8 Trade-Offs

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Asks whether order matters and whether duplicates are acceptable, and explains that "at-least-once plus idempotent consumers" is the agreement the design rests on.
• Explains why a queue doesn't delete on receive, and how a visibility timeout brings back a dead worker's message.
• Stores messages in an append-only log, fsyncs before answering, and replicates to a majority across AZs before saying "got it".
• Names the replication protocol and says how a new leader is chosen without losing committed messages.
• Uses long polling and knows why pull protects slow consumers.
• Handles poison messages with a receive count and a DLQ, and puts an alarm on the DLQ.
• Does the numbers, and says honestly that SQS is the right answer at this size.

Follow-up questions

1. "Why not just use a Postgres table as the queue?" Answer: at 2,000 messages a second, it can work: workers take rows with SELECT ... FOR UPDATE SKIP LOCKED, which lets each worker lock a different row without waiting on the others. The costs show up as volume grows: every message is an insert, an update and a delete, so the table churns constantly and the database spends a lot of effort cleaning up dead rows; the queue shares a database with everything else; and it doesn't grow past one primary. It's a fair choice for a small system that already has Postgres. A log is built for exactly this workload.

2. "Processing takes 8 seconds at P99 and up to 60 seconds for rare big orders. What visibility timeout do you set?" Answer: not the average, and not 60 seconds for everyone. We set 30 seconds, comfortably above the P99, and have the worker extend its lease every 10 seconds while it's still working. A big order keeps its lease as long as its worker is alive, and a crashed worker's message comes back within 30 seconds instead of 60.

3. "A broker is cut off from the other two for a minute, but it still thinks it's the leader. What stops it from accepting sends and losing them?" Answer: Raft's majority rule. The cut-off leader can append messages to its own log, but it can't get a second replica to store them, so nothing commits and it never answers 200. Its producers time out and retry through other brokers. Meanwhile the other two replicas elect a new leader in a higher term (Raft's election number). When the old leader reconnects, it sees the higher term, steps down, and throws away its uncommitted entries. No acknowledged message was ever on it alone.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Every Team Publishes and Subscribes"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · ~81K msg/s peak · 99.99% · order per key, no duplicates from retries, replay for 7 days

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built a task queue for one team: the order service hands 'ship this order' tasks to fulfillment workers. About 50 million messages a day, 2,000 a second at peak, 2 KB on average, one region, three AZs, 99.9%. Delivery is at-least-once and workers are idempotent. Each queue is split into 6 partitions. Each partition is an append-only segment log, fsynced with group commit, and replicated with Raft to 3 brokers in 3 AZs; a send is acknowledged once 2 of 3 replicas have it. A receive leases a message with a visibility timeout, 30 seconds by default: it's hidden until the worker deletes it, and it comes back if the worker dies. Deletes are ack records in the log. Workers long-poll for up to 20 seconds. After 5 failed receives, a message goes to a dead-letter queue with an alarm. Three m7g.large brokers, about $810 a month, and we said honestly that SQS would be the right buy at this size. Three costs are still open: there's no ordering, only one team can read a queue, and producer retries create duplicates."

Architecture v1, compact

textCopy
 order service ──► internal NLB ──► 3 brokers (m7g.large), one per AZ
                                      6 partitions, each a Raft group of 3; commit on 2 of 3
                                      segment log + ack records on 1 TB gp3, fsync (group commit)
                                      leases in the leader's memory: visibility 30 s, extend
 workers ──► long-poll every broker (≤ 20 s) ──► receive, delete
 receive count > 5 ──► DLQ + alarm          metadata store: 3-node Raft group, off the path

Round 1 step summary

Open costs: no order; one consumer team per queue (a message is gone once deleted); producer retries duplicate messages.

R2.1 The Scope Raise

Interviewer: "Your queue worked, and now every team wants it. We're at 2 billion messages a day. Payments needs messages for the same account processed in order. Producer retries must stop creating duplicates. Three teams want the same order events, and analytics wants to replay last week's events after it fixes a bug. Some teams want to send a message now and have it delivered in ten minutes. You must survive losing a whole AZ, and we want 99.99%."

A scope raise is not the end of scoping. Before we fix anything, we ask back, and we say what each answer changes, just as in R1.1.

Scope change

The "Not yet" list from R1.2 is now mandatory: ordering, removing duplicates, delay, many teams reading one message, and replay. Many readers and replay change the design most.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

We now have two kinds of topics, and the difference is the heart of this round:

• A queue topic works like Round 1: each message goes to one worker, and a delete removes it for good.
• A stream topic keeps every message for its retention period, whether anyone has read it or not. Each reading team is a consumer group with its own position, and nobody deletes anything.

Send, with order, deduplication and delay (queue or stream topic):

httpCopy
POST /v1/topics/payments/messages HTTP/1.1
Host: mq.internal
Content-Type: application/json

{
  "body": "{\"accountId\": \"acct_42\", \"paymentId\": \"pay_9001\", \"amountCents\": 1299}",
  "groupKey": "acct_42",
  "deduplicationId": "pay_9001",
  "delaySeconds": 0
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "messageId": "m-17-5520391", "partition": 17, "offset": 5520391, "duplicate": false }

• groupKey decides the partition, so every message with the same key lands in the same partition, in order (step 2.1). It plays the role of SQS FIFO's MessageGroupId.
• deduplicationId: a second send with the same ID within 5 minutes stores nothing and returns the first message's ID with "duplicate": true (step 2.2).
• delaySeconds (0 to 900) is only allowed on queue topics without a groupKey. A delayed message is stored at once but stays invisible until its time. On an ordered group it would block every message behind it, so ordered queues don't take per-message delays (SQS FIFO has the same rule: delay only per queue).

Batch send and batch receive. POST /v1/topics/{t}/messages:batch takes up to 100 messages, and a receive can return up to 100. A batch shares one request, one TLS record and one group-commit wait.

Stream topics: fetch by offset, commit the offset.

httpCopy
POST /v1/topics/order-events/groups/email-service/partitions/4:fetch HTTP/1.1
Host: mq.internal
Content-Type: application/json

{ "fromOffset": 88120775, "maxBytes": 1048576, "waitMs": 500 }

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "messages": [ { "offset": 88120775, "body": "..." }, { "offset": 88120776, "body": "..." } ],
  "highWatermark": 88121940 }

httpCopy
POST /v1/topics/order-events/groups/email-service/offsets HTTP/1.1
Host: mq.internal
Content-Type: application/json

{ "partition": 4, "offset": 88120777 }

The committed offset is the position of the next message the group will read. The high watermark is the offset of the last committed message plus one: consumers never see anything that isn't on a majority yet.

Seek, by offset or by time, for replay:

httpCopy
POST /v1/topics/order-events/groups/analytics/seek HTTP/1.1
Host: mq.internal
Content-Type: application/json

{ "toTimestamp": "2026-09-21T00:00:00Z" }

The broker finds, in each partition, the first message at or after that time (step 2.4 explains how) and sets the group's offsets there. { "partition": 4, "toOffset": 88000000 } seeks one partition to an exact offset instead. A group must have no active consumers while it seeks, so two instances can't disagree about where they are.

Quotas. Each team has a produce rate and a fetch rate, in bytes per second. Over quota, the answer is 429 Too Many Requests with Retry-After (step 2.7).

Two consumption contracts, side by side

R2.4 Design Evolution: Order, Duplicates, Fan-Out and Replay

Step 2.1: Payments for One Account Must Be Processed in Order

Primitive: Database Sharding & Partition Keys

Step 2.2: A Producer Retry Created a Duplicate

sqlCopy
-- In the consumer's own database, committed together with the business change
CREATE TABLE processed_messages (
  dedup_id     VARCHAR(128) PRIMARY KEY,   -- the message's deduplicationId
  processed_at TIMESTAMP    NOT NULL
);

Primitive: Change Data Capture & Outbox Pattern · Drill: CDC outbox dual-write drift

Step 2.3: Three Teams Want the Same Events, and Analytics Wants Last Week Again

Primitive: Message Queues vs Event Streams

Step 2.4: A Replaying Reader Slowed Everyone Else Down

Synthesizing vector architecture diagram...

Three kinds of reader, three paths. Only the live readers use the page cache, so nothing the other two do can push their data out of it.

textCopy
Sparse indexes for one segment (base offset 5,520,000)
  .index      relative offset → byte position       one entry every ~4 KB of log (about 2 messages of 2 KB)
              0 → 0
              2 → 4,096
              4 → 8,192
              ...
  .timeindex  timestamp (ms) → relative offset      one entry every ~4 KB of log
              1790424000000 → 0
              1790424000001 → 2
              1790424000002 → 4
  find offset 5,520,005: relative offset 5; newest index entry ≤ 5 is 4 → start reading at byte 8,192,
                         skip record 4, return record 5

Primitive: Message Queues vs Event Streams

Step 2.5: A Stale Worker Deleted a Message Another Worker Now Holds

Primitive: Distributed Locks & Leases · Drill: Distributed lock fencing token

Step 2.6: Consumer Groups Freeze Every Time One Consumer Restarts

Step 2.7: One Tenant's Hot Key Is Starving Others

Primitive: Distributed Rate Limiting · Drill: Sharding tenant hotspot

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Producers ask once where each partition's leader is, then talk to leaders directly; the NLB only helps them find the cluster. Inside the fleet, every partition is a Raft group with one replica per AZ. Stream consumers read from the replica in their own AZ, which keeps their traffic from crossing AZs (R2.6). Sealed segments flow down to S3, and reads older than 24 hours flow back up from it without touching the brokers' page cache.

The pieces that changed:

• 12 brokers, 4 per AZ, each an m7g.2xlarge (8 vCPUs, 32 GiB) with a 2.5 TB gp3 volume. About 2,000 partitions across all topics, each with 3 replicas: $6{,}000 / 12 = 500$ replicas per broker.
• Controllers replace Round 1's metadata store: the same kind of 3-node Raft group, now also holding quotas and consumer-group assignments.
• Clients connect to partition leaders directly. At this scale, sends for a key must reach that key's leader, so a load balancer in the middle would only add a hop.
• Consumers fetch from followers in their own AZ. A follower only serves messages up to the high watermark it has heard from the leader, so it never shows uncommitted data; it adds a millisecond or two. (Kafka supports the same thing with client.rack.)

The write path at this scale: one journal per broker. R2.2 found that an fsync per partition log no longer fits: 500 logs per broker, each fsyncing every few milliseconds, would need tens of thousands of disk operations a second. So each broker now writes every incoming record, for all its partitions, to one shared journal file first. The journal is fsynced once per 2 ms window (or 256 KB), which is at most 500 fsyncs a second per broker, whatever the partition count. Only after the journal fsync does a replica count as "has it". Records are then copied into each partition's own segments in memory, and those segment files are written to disk in the background without waiting. If a broker crashes, it replays the journal to rebuild the segments' tails; once segments are safely on disk, the journal is trimmed. Databases call this group commit to a write-ahead log, and Apache BookKeeper (the storage under Apache Pulsar) splits its journal from its data files the same way.

Heartbeats for hundreds of Raft groups are also batched: between any two brokers, one message carries the heartbeats of every partition they share.

Trace 1: an ordered, deduplicated send.

Synthesizing vector architecture diagram...

The retry never reaches the log. If the payments service had restarted instead and resent with a new producer ID, the 5-minute pay_9001 entry would catch it the same way.

Trace 2: a replay from a timestamp.

Synthesizing vector architecture diagram...

The replay only ever touches S3 and the separate cold read path until it reaches recent data, so the payments team's latency doesn't move.

R2.6 Numbers and Cost

Traffic and storage, re-derived from the current track's figures:

Two corrections to the track's figures. First, the 20% isn't really "index": a sparse index has one small entry (8 bytes in the offset index, 12 in the time index) per ~4 KB of log, well under 1%. The 20% is record headers, lease and ack records, and free space, and it's still a reasonable planning figure. Second, the track's single "P99 4.8 ms" for a send isn't derived from anything; the budget below is.

Per-broker load. Every message is written to 3 replicas, so the fleet takes in $162 + 2 \times 162 = 486$ MB/s at peak (from producers, plus replication) and sends out $324 + 324 = 648$ MB/s (to consumers, plus replication out of leaders). Spread over the brokers:

After an AZ loss, each partition still has 2 of its 3 replicas, which is a majority, so commits continue. The headroom above 30% is for catch-up readers, for the returning AZ copying what it missed (which we throttle), and for TLS. The gp3 volumes are provisioned at 250 MB/s (baseline is 125 MB/s), and the instance's own EBS baseline is 312.5 MB/s. These per-broker figures are planning numbers; we confirm them with a load test before launch.

Partitions. About 200 teams' topics, averaging 10 partitions each, gives about 2,000 partitions, or 500 replicas per broker. As a sanity check, AWS recommends at most 2,000 partition replicas per broker for Amazon MSK's kafka.m7g.2xlarge; we're a different system, but the order of magnitude is right. Throughput isn't what sets the count: at a planning ceiling of 5 MB/s per partition, 162 MB/s needs only 33. The count is set by how many consumers each team wants to run in parallel, since a group can't use more consumers than partitions.

Page cache. Of each broker's 32 GiB, about 24 GiB is left for the page cache. At 40.5 MB/s of writes, that holds $24{,}000 / 40.5 \approx 590$ seconds, about 10 minutes, of everything the broker stores. Live readers are seconds behind, so they're served from memory.

What the S3 tier offloads

Tiered storage cuts the brokers' disk by 86% and saves about $13,700 a month. The 2.5 TB volumes are 48% full, which leaves room for a whole day of S3 upload failures before a disk fills.

Cross-AZ traffic ($0.01/GB in each direction, so $0.02 per GB that crosses; 4 TB/day × 30.4 = 121.6 TB a month):

Fetching from followers saves about $3,200 a month. Replication traffic can't be avoided: it is the durability.

Latency budget: P99 from send to a waiting reader under 15 ms.

That leaves about 4 ms, and the least predictable piece is the fsync on network block storage. If load tests show its P99 breaking the budget, the lever is the one Kafka uses by default: count a replica as "has it" once the message is in its memory, and flush to disk in the background. Messages are then lost only if replicas in two AZs lose power before the flush. That's a real trade, and we'd make it per topic, in writing, not silently for everyone.

Availability budget. 99.99% is 4.4 minutes a month. A broker crash makes its ~167 leader partitions refuse sends for 1 to 2 seconds while followers elect new leaders; producers retry within their timeout, so callers rarely see it. The budget goes to what retries can't hide: a bad deploy, a misconfigured quota, a controller problem.

Monthly cost (us-east-1, on-demand)

The single biggest line is replication traffic between AZs: more than the brokers themselves.

Build vs buy at this size.

The honest answer for most companies: MSK for the streams, SQS for the task queues. MSK costs about what our fleet does, without the on-call rotation, because its free replication traffic offsets its higher broker price. We would build only if we need something neither offers, and in Round 3 we'll see whether scale changes that.

R2.7 Trade-Offs

Acknowledgment levels. When does the leader answer the producer? Kafka names the options with its acks setting, and every log-based system makes the same choice:

The alternatives, side by side (figures from AWS's documentation at the time of writing):

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Separates the two consumption models, queue and log, explains why each exists, and supports both.
• Gives order per key, not globally, and names head-of-line blocking and hot groups as its costs.
• Stops duplicates at the door with an idempotent producer and a dedup window, and still says processing is at-least-once.
• Protects live readers from replays with tiered storage and a separate cold read path.
• Fences stale workers with lease epochs, and keeps lease state across leader changes.
• Knows why consumer groups freeze and fixes it with cooperative rebalancing and static membership.
• Sizes the fleet for an AZ loss and finds that cross-AZ traffic is the biggest line on the bill.

Follow-up questions

1. "How do you build an exactly-once pipeline when the queue is at-least-once?" Answer: decide where "once" must hold. Inside the log, a read-process-write job can be exactly-once with Kafka-style transactions: the idempotent producer, plus a transaction that writes the output messages and the consumer's new offsets atomically, plus downstream readers that only read committed transactions (read_committed). If the job crashes, the transaction is aborted and redone, and readers never see the half-done attempt. That guarantee stops at the log's edge. The moment the effect is outside it (charging a card, sending an email, updating a database), we need idempotency there: the idempotency-key table in the same transaction as the effect, or the outbox. End to end, "exactly once" is always at-least-once delivery plus idempotent effects.

2. "Why is Kafka so fast on ordinary disks?" Answer: four reasons. Sequential I/O: it only appends to the end of files and reads forward, which is the fastest thing disks do. The page cache: it lets the operating system keep recent data in memory instead of keeping its own cache, so live readers rarely touch the disk. Batching: producers send, brokers store and consumers fetch messages in batches, often compressed, which spreads each request's fixed cost over many messages. Zero-copy: for plaintext connections, the broker uses the sendfile system call, so data goes from the page cache to the network card without being copied into the broker's own memory. The caveat: with TLS, the broker must encrypt the data itself, so it has to copy it into its own memory, and plain zero-copy is lost. Linux kernel TLS can move encryption into the kernel so sendfile works again, but that depends on the runtime and the platform, so don't assume it.

3. "The payments topic needs more partitions. Can we just add them?" Answer: not without breaking order for a while. hash(groupKey) mod P changes for most keys when P changes, so acct_42's next payment could land in a new partition and be processed before older ones still waiting in the old partition. The safe way is a cutover: create the new layout, pause producers for the moved keys (or the whole topic, briefly), let consumers drain the old partitions for those keys, then resume on the new layout. Better still, create ordered topics with enough partitions for years of growth, since idle partitions cost little.

Interview gotchas from this round

---

Round 3 · Architect · "Global, Regulated, and at Petabyte Retention"

~45 min · Principal (L7) · 4 regions · ~810K msg/s peak · 99.999% per region · RPO of seconds for critical topics

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run the company's messaging platform in one region across three AZs: 2 billion messages a day, about 81,000 a second at peak, 2 KB on average, 99.99%. There are two kinds of topics. Queue topics work like SQS: a message is leased to one worker with a visibility timeout, deleted on success, moved to a DLQ after 5 tries. Stream topics work like Kafka: messages stay 7 days, and each consumer group commits its own offsets, so many teams read the same data and can replay it by time. Order is per group key: the key hashes to a partition, and a queue hands out one message of a group at a time. Producers are idempotent, with IDs and sequence numbers, and there's a 5-minute dedup window. Receipt handles carry a lease epoch, so a stale worker can't delete a message someone else holds. Every partition is a Raft group with a replica in each AZ, committing on 2 of 3, through one fsynced journal per broker. Brokers keep 24 hours locally and 7 days in S3, and replays read from S3 on their own path. Twelve m7g.2xlarge brokers, about $12,800 a month, with cross-AZ replication as the biggest line; managed MSK costs about the same. Four costs are still open: it's one region; offsets and order don't exist outside it; long retention would be expensive; and we can't delete one customer's data from an append-only log."

Architecture v2, compact

textCopy
 ~200 teams (client library: producer ID + sequence, dedup IDs, batches, quotas → 429)
   │ bootstrap via NLB, then straight to partition leaders
   ▼
 12 × m7g.2xlarge, 4 per AZ · ~2,000 partitions · each a Raft group of 3 (one per AZ), commit on 2
   │ one journal per broker, fsync per 2 ms window; per-partition segments written behind it
   │ queue topics: leases with epochs (replicated lease records), DLQ after 5
   │ stream topics: 7-day retention, consumer groups, __offsets (compacted)
   ├── sealed segments hourly ──► S3 (7 days); reads older than 24 h come back from S3
   └── controllers: 3-node Raft group (topics, leaders, quotas, group assignments)
 consumers fetch from the replica in their own AZ

Steps so far

Open costs: one region is one blast radius; offsets, order and dedup stop at the region's edge; long retention is expensive; one customer's data can't be removed from an immutable log.

R3.1 The Scope Raise

Interviewer: "The platform now runs in four regions, and we must survive losing any one of them. For critical topics, we can lose at most a few seconds of messages. EU tenants' messages must stay in the EU. Finance must keep some topics for 90 days, and legal sometimes puts data on hold. A customer can demand that we delete their data. We're at 20 billion messages a day. And the platform's bill is now a board-level question."

Again, we ask back before we design:

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

A topic policy carries everything this round adds: home region, residency, mirroring, retention tiers and encryption.

jsonCopy
{
  "topic": "payments-eu",
  "tenant": "t_bank_eu_17",
  "homeRegion": "eu-west-1",
  "residency": "EU",
  "replication": { "mirrorTo": ["eu-central-1"], "targetLagSeconds": 5 },
  "retention": { "brokers": "24h", "s3Standard": "7d", "s3GlacierInstantRetrieval": "90d" },
  "encryption": {
    "tenantKey": "arn:aws:kms:eu-west-1:111122223333:key/mrk-1234abcd12ab34cd56ef1234567890ab",
    "perCustomerKeys": true
  },
  "ordered": true
}

• homeRegion is the one region that accepts writes for this topic's keys (step 3.2).
• residency limits where the topic, its mirrors, its S3 objects and its keys may live. A policy that mirrors an "EU" topic outside the EU is rejected with 400 and the code RESIDENCY_VIOLATION.
• retention names a tier for each age (step 3.4).
• tenantKey is a KMS multi-Region key, so its replica in the paired region can decrypt mirrored data (step 3.3).

Legal hold

httpCopy
POST /v1/legal-holds HTTP/1.1
Host: mq-control.internal
Content-Type: application/json

{ "scope": { "tenant": "t_bank_eu_17", "topics": ["payments-eu"] },
  "reason": "Litigation hold", "ticket": "LEGAL-2231" }

While a hold is active, retention and deletion skip the held data: segments aren't deleted, S3 objects get an S3 Object Lock legal hold (an object under a legal hold can't be deleted until the hold is removed), and any key that protects held data can't be destroyed.

Deletion requests

httpCopy
POST /v1/deletion-requests HTTP/1.1
Host: mq-control.internal
Content-Type: application/json

{ "tenant": "t_bank_eu_17", "customerId": "cust_88311", "ticket": "PRIV-5530" }

httpCopy
HTTP/1.1 202 Accepted
Content-Type: application/json

{ "requestId": "del_7c1e", "status": "PENDING", "statusUrl": "/v1/deletion-requests/del_7c1e" }

202 means "accepted, in progress": deletion takes steps that finish at different times, and the status URL reports each one (step 3.3, trace 2 in R3.5). A request that touches data under a legal hold is recorded and waits, with status ON_HOLD.

Per-tenant encryption keys. Every tenant gets its own KMS key. Producers encrypt message bodies with data keys generated by KMS and encrypted under it, and topics that carry personal data also encrypt each customer's fields with a per-customer key (step 3.3).

R3.4 Design Evolution: Regions, Laws and Money

Step 3.1: A Region Is Gone

Synthesizing vector architecture diagram...

Nothing in us-east-1 waits for us-west-2. The copy trails the source by the mirror's lag, and the checkpoint topic lets a consumer group that moves to us-west-2 find its place in the copy.

Primitive: Cloud Disaster Recovery & Multi-Region Active-Active

Step 3.2: After Failover, Messages Arrived Twice and Out of Order

Drill: Replication multi-region consistency

Step 3.3: EU Data Must Stay in the EU, and a Customer Wants Their Data Deleted

The key table, and what the watch-list says about it:

• Create keys only in the key's home region, with a conditional put, attribute_not_exists(customer_key_id), so two producers can't create two different keys for the same customer.
• The table is a DynamoDB global table across the EU pair, so the paired region can decrypt after a failover. Global tables check a conditional write only against the local region's copy and, by default, settle conflicts by "last writer wins". If both regions could create a key for the same customer at once, one key would silently replace the other, and data encrypted under the loser would be unreadable forever. That's why creation happens only in the home region. (DynamoDB's multi-Region strong consistency mode would check conditions across Regions, but it runs in exactly three Regions (three replicas, or two replicas plus a witness), so we would need a third EU Region for a witness, and it doesn't support TTL. We choose not to add one, and create keys only in the home region instead.)
• Never use TTL to expire keys. DynamoDB TTL deletes items only eventually, typically within a few days, so it can't mark the moment a key stops working. Deletion is an explicit DeleteItem, recorded in the deletion request's audit trail.
• Backups hold keys too. A deleted row still exists in any backup of the table taken before the deletion. Shredding is complete only when the last such backup has expired, so the key table's backup retention is kept short and written down, and the deletion request's status only becomes COMPLETE after it has passed.

Step 3.4: 90-Day Retention Costs a Fortune

The storage bill is in R3.6: with tiers and compression, about 600 TB stored for about $24,000 a month, instead of 13 PB for over a million.

Step 3.5: One Bad Config Took Down Every Topic

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance Patterns

Step 3.6: Should We Run This at All?

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Read it as two independent pairs. Inside each region, cells share nothing with each other. Critical topics flow one way, from their home region to its pair; EU data never leaves the EU pair. The control plane only pushes policies and the cell map, in waves, and ARC's routing controls decide which region of a pair is home. Mirroring is drawn one way for clarity: topics whose home is us-west-2 or eu-central-1 mirror in the other direction the same way.

Trace 1: us-east-1 fails, and payments moves to us-west-2.

Synthesizing vector architecture diagram...

Nothing switches automatically. An engineer confirms the outage and flips one routing control per topic group. Consumers find their place from the checkpoint and reread a few seconds; the last ~3 seconds of us-east-1 wait there until it returns.

Failing back. When us-east-1 is healthy, the mirror runs the other way (us-west-2 to us-east-1) until its lag is near zero. Then an operator switches the home back, consumers move using the checkpoints in us-east-1, and the unmirrored tail from the outage, now old, has already gone to reconciliation. Failback rereads a few seconds too.

Trace 2: a customer's deletion request.

Synthesizing vector architecture diagram...

No segment is rewritten anywhere. Once the customer's key row is gone, their encrypted fields in brokers, S3, Glacier and the mirror are unreadable; the request completes when no backup still holds the key.

R3.6 Numbers and Cost

Traffic per region. 20 billion messages a day is $2 \times 10^{10} / 86{,}400 \approx 231{,}500$ messages a second on average, and $\times 3.5 \approx 810{,}000$ at peak. A planning split:

Each cell's 81K already includes Round 2's headroom for losing an AZ. The extra 30% covers growth and consumer surges. A region failure doesn't overload its pair: when us-east-1 fails, its critical producers (25% of 243.1K, so 60.8K a second) start writing to us-west-2, and the mirror stream they replace was exactly that size. The pair's planned peak stays at 222.8K.

Availability budget. 99.999% per region is $43{,}776 \text{ min} \times 0.00001 \approx 26$ seconds a month. A leader election takes 1 to 2 seconds, so the budget only survives because an election touches one broker's partitions, not the region, and the client library's retries hide it from callers. What would spend the budget is anything region-wide, which is why cells and waves matter: a bad change caught in one canary cell costs that cell's tenants, not the region.

Cross-region mirroring. 25% of the average rate is mirrored: $0.25 \times 231{,}500 \approx 57{,}900$ messages a second, at 1 KB each after compression, which is about 57.9 MB/s, or 5 TB a day.

$0.02 per GB is the published rate between these US regions and between these EU regions at the time of writing; treat it as a floor and check the current price list for each pair.

Storage by tier (compressed, 1 KB per message; "own" is 20 TB/day, "mirrored" 5 TB/day, finance is 20% of own, 4 TB/day):

The S3 Standard figure is an upper bound: past the first 50 TB a month, the price per GB steps down. For comparison: all 90 days on brokers, uncompressed, is about 13 PB and over $1 million a month (step 3.4).

Fleet, all four regions. 18 cells × 12 brokers = 216 m7g.2xlarge, and 54 controllers (3 per cell) on m7g.large. The broker disks are 42% full; the rest leaves more than a day of room if S3 uploads fail.

Cross-AZ traffic inside cells. All four regions together write $20 + 5 = 25$ TB a day into their cells, so 760 TB a month.

Monthly total (us-east-1 prices everywhere; the EU regions cost somewhat more, so this is a floor):

Cross-AZ traffic ($40.5K) is almost as big as the brokers ($51.5K), and three quarters of it is replication.

The same shape on Amazon MSK.

MSK's broker hour costs 2.5 times the EC2 instance's; its free replication traffic wins back about $30K of that. Net, building saves about $53K a month at this scale, the input to step 3.6's decision. MSK also offers Express brokers, priced and sized differently, which could need fewer brokers for the same throughput; we'd price that option too before deciding.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we make sure every message is delivered, none are lost, and order holds where we promised it, while machines and consumers fail? The answer is now layered:

• Delivered: leases with a visibility timeout and fenced receipt handles, at least once, with idempotent consumers, in every round.
• Never lost: committed on a majority of replicas in three AZs before we say "got it"; mirrored to a second region for critical topics, where "never" becomes "at most about 5 seconds", a number the business chose.
• Ordered where promised: per key, in one partition, in one home region, and switched between regions only by a deliberate act.

And the architect's addition: for most workloads, the best way to keep those promises is to buy them from a service that already does, and spend our effort on the policies around them.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per cell and per region OPS 8 · REL 6

DLQ redrive procedure OPS 10

1. Look before moving. Read the first few DLQ messages and their error attributes. Find the one cause; don't redrive a mix of causes.
2. Fix the cause. Deploy the consumer fix, or correct the bad data at its source.
3. Test with one. Redrive a single message and watch it succeed.
4. Redrive at a limited rate, well under what the consumers handle, so old messages don't swamp live traffic. For an ordered queue, redrive a group's messages in order and before that group's newer messages are released, or send them to reconciliation instead.
5. Watch the DLQ. If messages come back, stop and return to step 1.
6. Close with a note of the cause and the fix, and an alarm or test that would have caught it earlier.

Region evacuation procedure REL 13

1. Confirm it's the region. Several cells unhealthy in one region at once, and AWS Health or our own cross-region probes agree. One cell down is a cell incident, not a region one.
2. Check what we'd lose. Note each critical topic's last mirror lag: that's the data that stays behind.
3. Flip the home for each critical topic group with its ARC routing control, one group at a time, payments first.
4. Watch consumers resume from checkpoints in the paired region; expect a few seconds of rereads.
5. Protect the pair. Confirm it's under its planned peak; pause any wave deploys in it.
6. When the region returns: send its unmirrored tail to reconciliation topics, run the mirror in reverse until its lag is near zero, then fail back the same way, and write the COE (Correction of Error).

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the ARNs, names and endpoints with real ones.

textCopy
# 1. Alarms currently firing for the platform in one region
aws cloudwatch describe-alarms --region us-east-1 --state-value ALARM --alarm-name-prefix mq-

# 2. Mirroring lag for one critical topic over the last hour (a custom metric the mirror workers publish)
aws cloudwatch get-metric-statistics --region us-west-2 --namespace MQ/Mirror --metric-name LagSeconds --dimensions Name=Topic,Value=payments --start-time 2026-09-26T11:00:00Z --end-time 2026-09-26T12:00:00Z --period 60 --statistics Maximum

# 3. Which region is home for payments right now (ask an ARC cluster endpoint in a healthy region)
aws route53-recovery-cluster get-routing-control-state --region us-west-2 --endpoint-url https://abcd1234.route53-recovery-cluster.us-west-2.amazonaws.com/v1 --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/0123456789abcdef/routingcontrol/payments-use1

# 4. Evacuate: turn us-east-1 off and us-west-2 on as home for payments
aws route53-recovery-cluster update-routing-control-state --region us-west-2 --endpoint-url https://abcd1234.route53-recovery-cluster.us-west-2.amazonaws.com/v1 --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/0123456789abcdef/routingcontrol/payments-use1 --routing-control-state Off
aws route53-recovery-cluster update-routing-control-state --region us-west-2 --endpoint-url https://abcd1234.route53-recovery-cluster.us-west-2.amazonaws.com/v1 --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/0123456789abcdef/routingcontrol/payments-usw2 --routing-control-state On

# 5. Put one archived segment under legal hold, and check it (the bucket must have Object Lock enabled)
aws s3api put-object-legal-hold --bucket mq-archive-eu-west-1 --key payments-eu/p17/00000000000005520000.log --legal-hold Status=ON
aws s3api get-object-legal-hold --bucket mq-archive-eu-west-1 --key payments-eu/p17/00000000000005520000.log

# 6. A tenant leaves: check its key, then schedule deletion with the longest waiting period (replica keys too)
aws kms describe-key --region eu-west-1 --key-id mrk-1234abcd12ab34cd56ef1234567890ab
aws kms schedule-key-deletion --region eu-central-1 --key-id mrk-1234abcd12ab34cd56ef1234567890ab --pending-window-in-days 30
aws kms schedule-key-deletion --region eu-west-1 --key-id mrk-1234abcd12ab34cd56ef1234567890ab --pending-window-in-days 30

# 7. Undo a mistaken deletion during the waiting period
aws kms cancel-key-deletion --region eu-west-1 --key-id mrk-1234abcd12ab34cd56ef1234567890ab

# 8. For task queues already moved to SQS (step 3.6): redrive a DLQ back to its source at 50 messages a second
aws sqs start-message-move-task --source-arn arn:aws:sqs:us-east-1:111122223333:order-fulfillment-dlq --destination-arn arn:aws:sqs:us-east-1:111122223333:order-fulfillment --max-number-of-messages-per-second 50
aws sqs list-message-move-tasks --source-arn arn:aws:sqs:us-east-1:111122223333:order-fulfillment-dlq

Command 4 asks a cluster endpoint in us-west-2 on purpose: during a us-east-1 outage, we use ARC's endpoints in the regions that are still up. A key scheduled for deletion is unusable for the whole waiting period, so step 6 is only for a tenant whose offboarding has been approved, after checking legal holds.

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Chooses async mirroring with an RPO set per topic, and can say how many messages that RPO means at peak.
• Knows that offsets, order and dedup stop at a region's edge, and designs each back in: offset translation, home regions, idempotent consumers.
• Treats failover as a deliberate, fenced decision rather than an automatic reaction to a health check.
• Designs for law and policy without overclaiming: residency enforced by placement, deletion by crypto-shredding, legal holds that beat both.
• Finds the real costs (cross-AZ replication, retention) and uses tiers and minimum-duration rules to cut them.
• Limits blast radius with cells and waves.
• Answers "should we run this?" with numbers, including the team, and is willing to recommend buying.

Follow-up questions

1. "Why not active-active: let both regions of a pair accept writes for every topic?" Answer: for unordered topics with idempotent consumers, we can: each region writes its own messages, mirrors them to the other, and consumers read both, tagged by origin so the mirror never copies a message back. For ordered topics it breaks the promise: two regions accepting writes for acct_42 can't agree on their order without a cross-region round trip on every write. So ordered topics have a home region, and unordered ones may be multi-writer by policy.

2. "A customer's deletion request arrives while their data is under legal hold. What does the system do?" Answer: the system doesn't decide which wins; it makes both visible and enforces whatever legal decides. The request is recorded with status ON_HOLD, the customer's key can't be deleted while a hold covers it, and the privacy team sees the conflict. When legal releases the hold, the deletion proceeds. Every step is in an audit log.

3. "Some newer Kafka-compatible systems write straight to S3 and keep no replicas on broker disks. Would that fit here?" Answer: for some topics, yes. S3 stores data redundantly across AZs itself, so there's no replication traffic between brokers, which is our biggest cost line ($30K a month). The price is latency: every write waits for an S3 upload, tens of milliseconds or more, so a 15 ms P99 is out of reach. It's a good fit for analytics and log topics that can wait a second, and a bad one for payments. A platform could offer it as a third kind of topic, with its own price and its own latency promise.

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. The other loops in this track: Distributed Key-Value Store and Distributed Unique ID Generator.

The Two Sentences That Matter Most

• Opening a round: "Before I design, let me ask two things: does order matter, and can a message be processed twice if the consumer is idempotent?"
• When the scope is raised: "Here's what breaks in the current design, and here's the order I'll fix it in, starting with anything that could lose an acknowledged message."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels