Queues & Delivery Semantics

Part 0. Start here

The problem: the order that vanished with every alarm green

Basketly sells groceries online. On Friday at about 20:00:07 the checkout of order o-7731 commits, and an OrderPlaced message for it lands on the fulfil queue. A fulfilment worker takes the message and asks the warehouse system (the WMS) for a pick list. The gift note on o-7731 has a character the WMS's label step can't handle, so the call hangs. After 40 seconds the worker gives up, logs an error and moves on. It doesn't delete the message.

The queue was set up with Amazon SQS's defaults, and two alarms that look reasonable:

• a 30-second visibility timeout (how long a taken message stays hidden) and 4-day retention, with no dead-letter queue;
• an alarm on "age of the oldest message > 5 minutes";
• and in a second version of the story (Part 5), someone adds a dead-letter queue on Saturday with an alarm on its NumberOfMessagesSent > 0: the message moves there after 2,880 deliveries, the alarm stays green, and it still expires on Tuesday.

Nobody is paged, so nobody tells the WMS team, and the bug is never fixed.

Neither alarm ever fires. On a standard queue, AWS's CloudWatch metrics page says: "if a message is received three or more times and not deleted, SQS moves it to the back of the queue", and "poison-pill messages … are excluded from this metric until successfully processed". (AWS's dead-letter queue page describes the same move only for queues whose redrive policy has maxReceiveCount above 3; this page follows the metrics page.) And the dead-letter alarm watches a metric that "does not capture messages sent to a DLQ because of failed processing attempts". On Wednesday a customer asks where the groceries are. Step 1.6 of the message queue loop tells the same story in one line: "for three hours it keeps coming back and taking workers down with it".

The big picture

Synthesizing vector architecture diagram...

What to notice: one checkout becomes one message on each of three queues, and b2b gets only the orders its filter matches. The red WMS box is the only broken thing, yet the whole fulfil path keeps paying for it. The dashed fulfil-dlq is where a failing message should go, and as shipped it doesn't exist.

What you'll be able to do after this page

1. Explain why a queue lends a message rather than handing it over, and say which delivery semantics follow from where you delete (Part 1).
2. Write a worker's receive loop: long polling, batch timers, the inbox check, the delete, and the in-flight limit (Part 2).
3. Size a visibility timeout, extend it with a heartbeat, and explain a delete that deletes nothing (Part 3).
4. Classify failures, back off with the visibility timeout, defer healthy work without burning receives, and bound retries with maxReceiveCount (Part 4).
5. Run a dead-letter queue that someone notices: the right alarm metric, retention, an owner, and a safe redrive (Part 5).
6. Consume a queue with Lambda without dead-lettering healthy batch-mates or losing a message, and cap its concurrency at the source (Part 6).
7. Use FIFO message groups, and choose between dead-lettering, blocking and parking when one message in a group fails (Part 7).
8. Read a partitioned log with a consumer group: commit after the effect, bound in-line retries, and avoid a rebalance cascade (Part 8).
9. Tell the age of the oldest message from the wait of a new one, compute a drain, and scale consumers on backlog (Part 9).
10. Fan out with SNS without silent drops, drain a subscription's dead-letter queue, and carry payloads over a hop's size limit (Part 10).
11. Trace one order through every hop, with the ack point and the guard at each (Part 11).
12. Map all of it to AWS, and name the look-alikes (Part 12).

You may have arrived from a step that relies on this: step 1.6 of the message queue loop (one bad message crashes every worker, forever), step 1.2 of the notification loop (the provider returned an error), step 2.6 of the job scheduler loop (deferral that must not burn receives), step 1.3 of the web crawler loop (one message per host group at a time), step 2.3 of the outbox ledger loop (one bad event blocks everything behind it), or the drill The Warehouse System That Fell a Day Behind. This page is the "why" behind all of them, and behind the queue and consumer lines in 22 of the 37 interview loops.

---

Part 1. A loan, not a hand-off

A worker took a message and died. Who has the message now? The answer decides whether the order is picked once, twice or never. This Part names the two shapes a message system can take, the one decision that sets its delivery semantics, and the example the rest of the page follows.

Queue or log

Step 2.3 of the message queue loop makes the same choice for three teams and a replaying analytics job. Each column wins some rows: the log wins replay and cheap fan-out, and the queue wins "only this message waits". A log can't skip one bad record without an error boundary you build (Part 8); a queue can't replay.

The ack point decides the semantics

Every consumer does three things: take the message, do the work, and tell the broker it is done (a delete on a queue, an offset commit on a log). That last step is the ack point. A crash can happen anywhere, so its position decides what a crash costs.

Synthesizing vector architecture diagram...

What to notice: the same crash, 0.2 s after the receive, loses the order if the delete came first, and only delays it if the delete comes after the work. Nothing else changed.

"Exactly once" exists only inside one store, such as the checkout's order row and outbox row committed in one transaction. Across a queue it is at-least-once plus an effect that ignores repeats. Idempotency & Effectively-Once Processing owns the keys and the inbox that make repeats harmless (Parts 1 and 6 there); this page owns where the ack sits and what the broker does between the receive and the delete.

Side rows 1x and 1y. A healthy order, o-7733, is enqueued at 20:00:20.65. The worker that takes it (W1 in both seeds) is killed 0.2 s later, before its WMS call returns, so the WMS creates nothing:

Why a queue at all? The drill The Warehouse System That Fell a Day Behind asks it directly. Without the queue, the checkout would call the WMS itself: a slow or broken WMS would slow or fail checkouts, and a checkout would have to retry the WMS on its own. With the queue, the checkout commits in milliseconds and the WMS works at its own pace. The price is everything on this page: the order is now picked later, at-least-once, and a failure is somebody's job to notice.

The example we follow

A real shop has many queues; that is too many to watch. So one story runs through the whole page: one order message that keeps failing. It is page 10's Basketly, on the same Friday: page 10's checkout at 20:00:07 committed with its outbox row, and this page follows the message it produced. Every trace on this page comes from running a private reference simulation of this setup (the queues, the workers, the WMS, the fulfil database, CloudWatch's metrics, and every guard as a setting), with two random seeds, not from working it out by hand. The example's numbers are small so that every delivery of one message fits on a line.

Each Part adds one guard and replays the same message: Q1 a sized lease and a heartbeat (Part 3), Q2 counting, backoff and a dead-letter queue (Part 4), Q3 a dead-letter queue that is watched and kept (Part 5), Q4 backlog alarms that mean something (Part 9) and Q5 fan-out that can't drop (Part 10). Each setting keeps all the guards before it. Parts 6 to 8 are different: they rewind to 20:00:07 on a copy and run the same message through a Lambda consumer, a FIFO queue and a Kafka topic, because those are alternatives, not extra guards. Side rows also run on copies, so nothing they do changes the main line.

The story in eleven beats: the loan, not the hand-off (Part 1); one message's round trip (Part 2); the attempt that outlives its lease (Part 3); the message that keeps failing (Part 4); the dead-letter queue nobody hears (Part 5); the batch it drags down (Part 6); the customer it blocks (Part 7); the partitions it blocks (Part 8); the backlog behind it (Part 9); the copies it fans out to (Part 10); one order, end to end (Part 11). Each Part shows only its own events; the full table is in Part 13.

Words we use

Events 1 to 3, as shipped (Q0)

Snapshot M1, t1 + 1 s (Q0)

---

Part 2. One message's round trip

At 19:59:50 Basketly is healthy: 15 orders a second arrive, and the 40 worker slots are mostly idle. Before anything fails, here is what one worker does with one message, and the three small choices in that loop that later decide whether a failure stays small.

The loop every worker runs

textCopy
WORKER LOOP (one per worker process, 10 slots)
  loop:
    n = free slots, at most 10          (n = 1 if you work one message at a time)
    msgs = ReceiveMessage(fulfil, MaxNumberOfMessages = n, WaitTimeSeconds = 20)
    for each msg, in parallel, one slot each:
        if the inbox already has msg.order:
            DeleteMessage(msg.receipt_handle); next msg
        start the heartbeat for msg                        (Q1 on, Part 3)
        pick = WMS: create pick list, Idempotency-Key = msg.order + ":pick"
        one fulfil DB transaction: insert the pick and the inbox row
        stop the heartbeat
        DeleteMessage(msg.receipt_handle)                  <- the ack point, after the effect

Event 4, the baseline. In the run, from 19:58 to 20:10, every receive that returned anything returned exactly one message (10,792 receives for 10,792 orders in seed 1). At 15 a second with 40 slots always waiting, a message is handed to a waiting receive the moment it arrives, so batches never form. Slots busy follow Little's law (in flight = arrival rate × time each one spends inside, page 10 Part 1): 15 a second × 0.508 s ≈ 7.6 of 40 (the run: 7.64 and 7.38 on average). Capacity is 40 slots ÷ 0.508 s ≈ 79 orders a second, so the fleet runs at about 19%.

Long polling

A receive with WaitTimeSeconds 20 waits up to 20 s for a message and returns as soon as one arrives. A receive without it (short polling) returns at once, usually empty, and samples only some of SQS's servers. At night (side row 2s, one order a minute):

Every call is a billed request. Step 1.5 of the message queue loop makes the same point at its scale: 1,800 requests a second of empty polls fall to about 9.

A batch starts every timer at once

A receive can return up to 10 messages, and every message in it gets its visibility deadline at the receive instant. Side row 2b forces a batch to show what that means. A deploy restarts all four workers at 20:00:07.0. W2 is back first, at 20:00:08.00, and its first receive returns 10 messages that arrived while nobody was reading, with o-7731 fifth. This W2 variant works its batch one message at a time.

Synthesizing vector architecture diagram...

What to notice: all ten 30 s timers end together at 20:00:38. W2 is still stuck on o-7731 then, so the five messages behind it become visible and other workers take and pick them. When W2 finally reaches them at about 20:00:50, its inbox check finds them done and its deletes use old receipt handles, so they do nothing.

Receive what you'll work

The rule that follows: process a batch in parallel, or receive only as many as you'll start now. If a worker handles one message at a time, or one per FIFO group (Part 7), it should ask for 1 (MaxNumberOfMessages 1). Taking 10 and handing 9 back with visibility 0 doesn't help: each hand-back was a receive, and every receive counts toward the redrive limit of Part 4. Step 1.3 of the web crawler loop had exactly this bug before its fix; it now receives one message per slot.

Deleting. Delete each message when its own effect is durable, or collect the finished ones and send DeleteMessageBatch (up to 10) to save calls. Never delete a batch as a unit before every message in it is done.

In-flight limits

Received-but-not-deleted messages are in flight. A standard queue allows about 120,000 in flight (FIFO: 120,000 too). Over the limit, short polling gets an OverLimit error and long polling simply returns nothing. Every consumer shares that one number, and messages held by long jobs or waiting out a backoff count toward it: the crawler loop's 120,000 in-flight messages cap how many hosts it can crawl at once.

What a queue is not

A queue is not a database. SQS keeps a message at most 14 days, and you can't look one up by order number. The truth about an order lives in your database; the message is a request to act on it (step 2.3 of the web crawler loop keeps its frontier's truth in DynamoDB for this reason).

---

Part 3. The attempt that outlives its lease

At 20:00:37.70 two workers are hanging on the same order. Nothing crashed. o-7731's first attempt is still waiting for the WMS, its 30 s lease has run out, and SQS has handed it to someone else.

Synthesizing vector architecture diagram...

What to notice: between t1 + 30 s and t1 + 40 s two workers hold o-7731 at once, each with a WMS connection. Nobody deleted anything, and nobody will: every attempt fails the same way.

Snapshot M2, t1 + 45 s (Q0)

The visibility timeout is a lease

A visibility timeout is a lease on the message: for that long, no one else gets it. Like any lease, it can expire under a worker that is still alive and still working. Leases, Fencing Tokens & Distributed Locks calls an SQS receipt handle "a lease without a fence" (Part 11 there): nothing stops the old holder's effects from landing after a new holder took over.

Sizing it

Set the visibility timeout above the slowest normal processing time, not the average. Our healthy orders take about 0.5 s (P99.9 about 2.5 s), so 30 s looks generous, until o-7740 renders a 75 s packing slip. Step R2.9 of the message queue loop lists "visibility set to the average" as a production gotcha for this reason. Q1 sets it to 60 s and adds a heartbeat, and it shortens the WMS client's timeout to 5 s, about 2 × the WMS's P99.9 (page 10's rule).

The heartbeat and its cap

textCopy
HEARTBEAT (a separate thread per message being worked; Q1 on)
  every 20 s while the work is running:
      ChangeMessageVisibility(msg.receipt_handle, 60 s)    -> hidden until now + 60 s
  stop when the work finishes, fails or is abandoned
  if the call is refused (the 12-hour cap), stop working: the message will come back

The new timeout counts from the call, not from the receive, and it isn't added to what was left. It is capped: "the visibility timeout has a maximum limit of 12 hours from when the message is first received. Extending the timeout doesn't reset this 12-hour limit" (SQS). A job that can run longer than 12 hours can't keep its message hidden: claim the job in your own table, delete the message, and keep your own lease (step 2.5 of the job scheduler loop). Step 2.2 of the YouTube loop uses 120 s extended every 30 s.

The delete that deletes nothing

Here is o-7740 (enqueued 20:00:20.65, a 75 s render) in three settings. The inbox check runs first on every delivery.

AWS on a delete with an old receipt handle: "the request will succeed, but the message might not be deleted". Our simulation takes the unsafe reading: it stays. So never rely on the delete to prevent a second effect. Two guards did that here: the inbox check first, which is what finally deleted o-7740, and the WMS's key, which would have returned the same pick list if a second render had reached it.

Slow failures hold resources

o-7731 in Q0 fails slowly: 40 s per attempt, repeated every 30 s, so it holds 40 ÷ 30 ≈ 1.33 worker slots and 1.33 WMS connections for four days. Those slots and connections are shared with every healthy order. With Q1, each attempt fails in 5 s and then waits out the rest of its 60 s lease: 5 ÷ 60 ≈ 0.08 slots and connections, and 86,400 ÷ 60 = 1,440 deliveries a day (the run: 1,441 in 24 hours, 0.083 slots). Cheaper, but still forever.

Letting go early

A worker that knows it can't do the work (it is shutting down, or the message is for a host it may not touch yet) should release the message at once with ChangeMessageVisibility 0, or a short delay, instead of letting the lease run out. Step 2.5 of the web crawler loop returns unstarted messages with visibility 1 s when a Spot instance gets its interruption notice.

---

Part 4. The message that keeps failing

With Q1, o-7731 fails in 5 s and costs 0.08 slots. It still fails 1,440 times a day, and nothing will ever stop it. Retrying is right for failures that can succeed later. For this one it only burns WMS calls until retention deletes the order.

Which errors to retry

Retries, Timeouts, Backpressure & Load Shedding owns the classification (Part 3 there). For a queue consumer it becomes:

Count, and back off with the visibility

Every receive raises the message's ApproximateReceiveCount, and the consumer sees it. Q2 uses it twice: to set a backoff and to decide when to give up.

textCopy
ON A FAILED ATTEMPT (Q2)
  n = msg.ApproximateReceiveCount
  if the error is permanent:
      SendMessage(fulfil-dlq, msg.body)              first send it...
      DeleteMessage(msg.receipt_handle)              ...then delete (a crash between = a harmless copy)
  else if the error is a throttle with Retry-After:
      ChangeMessageVisibility(msg.receipt_handle, Retry-After)
  else:
      ChangeMessageVisibility(msg.receipt_handle, random(0, min(900 s, 10 s x 2^(n-1))))
  if n = maxReceiveCount: emit LastAttemptFailed     (Q3 on, Part 5)

The backoff is page 10's full jitter (Part 4 there), applied by hiding the message: after the 1st failure it waits up to 10 s, after the 2nd up to 20 s, then 40, 80 and 160 s, capped at 900 s. Step 1.2 of the notification loop uses the same method with 10 receives, and about 51 minutes to the DLQ.

Bound it: maxReceiveCountand the dead-letter queue

A redrive policy on fulfil names a dead-letter queue (fulfil-dlq) and maxReceiveCount 5: "the number of times a consumer can receive a message from a source queue before it is moved" (SQS). So o-7731 is delivered five times, and it is moved by the next receive after the count reaches 5. Our workers always have a receive open, so that is the moment its 5th visibility timeout ends. If nothing is receiving, nothing is moved (Part 5).

Synthesizing vector architecture diagram...

What to notice: the only way into the DLQ is through a receive. The move is triggered by a consumer asking for work, not by a timer, so a queue nobody polls never moves anything.

A DLQ must be the same type as its source (a FIFO queue needs a FIFO DLQ), in the same account and Region.

Event 12 (Q2, seed 1). Five deliveries, each failing in 5 s, with jittered waits after them:

Snapshot M4, at the move (seed 1, 20:02:16.60)

Seed 2 moves it at 20:03:29.33 (t1 + 201.6 s). How long does it take in general? Each of the five attempts takes 5 s, and the jittered waits average 5 + 10 + 20 + 40 + 80 = 155 s, so about 180 s on average, anywhere from 25 s to 335 s. Over 5,000 seeds the run gives a mean of 179.7 s, a range of 38.7 to 318.6 s, and 91.4 to 268.7 s between the 5th and 95th percentiles. The 5th failure comes at a mean of 100.0 s, about 80 s before the move.

Deferring healthy work

Now a different failure: nothing is wrong with any order, but the WMS throttles everyone. Replay T (a copy of Q2): from 20:10:00 to 20:20:00 every WMS call returns 429 with Retry-After: 60. Four ways to wait it out (seeds 1 and 2):

Why 5,400? Each deferral is a receive, so an order that arrives before 20:20:00 − 4 × 60 s = 20:16:00 has all five of its deliveries inside the throttle and is moved: 360 s × 15 a second = 5,400 (the run: 5,374 and 5,421, all enqueued 20:10:00 to 20:16:00). Jitter makes it worse because its first waits are shorter than 60 s.

Use the visibility timeout for real retries of a message that failed. Defer healthy work by re-sending it with a delay (carrying your own attempt count), or by pausing the consumer, as step R1.9 of the notification loop does during a long provider outage. Step 2.6 of the job scheduler loop says it plainly: "every receive counts toward the queue's maxReceiveCount".

Delay queues and timers

DelaySeconds hides a new message for up to 15 minutes, set per queue or, on standard queues only, per message (FIFO queues don't support per-message timers). Step 1.4 of the ride-sharing loop uses a 16 s delayed message as a durable timer, at-least-once, guarded by a conditional update. For longer waits, SQS's documentation points to EventBridge Scheduler, which is a scheduler, not a queue (Part 12).

---

Part 5. The dead-letter queue nobody hears

Back to the hook. In Q0, o-7731 is delivered 11,520 times from Friday to Tuesday. The age alarm stays green, because from its third receive the message is left out of the metric. On Tuesday at 20:00:07.70 retention deletes it. Nobody was paged; on Wednesday a customer asks.

Snapshot M3, Saturday 20:00 (Q0)

Synthesizing vector architecture diagram...

What to notice: in "Queue" the message has been received 2,880 times in its first day; in "Workers and WMS" it holds a slot and a connection all the time; in "Alarms" everything is green, because the age metric doesn't count it.

Side row 5a (a copy of Q0, still no fix). On Saturday at 20:00 someone adds fulfil-dlq (default 4-day retention, maxReceiveCount 5) and an alarm on its NumberOfMessagesSent > 0. o-7731's count is already 2,880, so it moves at its next receive, Saturday 20:00:07.70. The alarm never fires: the move isn't a send. The DLQ's own age metric starts at the move and reads 24 h on Sunday, 48 h on Monday and 72 h at 19:59 on Tuesday, while retention still counts from Friday: the message expires on Tuesday at 20:00:07.70, 3.00 days after it arrived. The age metric makes it look a day younger than its expiry clock says.

Five ways a DLQ fails silently

The age metric's blind spot

The hook's quote again: "if a message is received three or more times and not deleted, SQS moves it to the back of the queue … This reordering occurs even when a redrive policy is in place", and such messages are "excluded from this metric until successfully processed" (the CloudWatch metrics page). AWS's DLQ page describes the reordering only for standard queues whose maxReceiveCount is above 3, so treat the exact condition as a detail that may change; the lesson doesn't. The age alarm catches backlogs, not one poison message. The receive count, the DLQ and your own last-attempt metric catch that. FIFO queues don't reorder, so there the age alarm does see a stuck message (Part 7).

An alarm that fires (Q3)

Q3 keeps Q2's policy and adds: fulfil-dlq retention 14 days; an alarm on its ApproximateNumberOfMessagesVisible ≥ 1; LastAttemptFailed; and an owner, the fulfil on-call, who pages the WMS team. CloudWatch is modelled as 1-minute datapoints (the maximum over the minute, published at the minute's end), with each alarm firing on its first breaching datapoint.

In seed 2 the 5th failure is at 20:01:55.91 (alarm 20:02:00), the move at 20:03:29.33 (DLQ alarm 20:04:00 or 20:19:00), and the pick at 21:40:00.68.

Snapshot M5, 21:40:01 (Q3), next to M3

Synthesizing vector architecture diagram...

What to notice: the same three lanes as M3. In "Queue" the message is gone and the DLQ is empty; in "Workers and WMS" one pick list exists, found by its key; in "Alarms" both alarms fired within a minute of each other, which is what got the bug fixed.

Bringing it back

A redrive (StartMessageMoveTask) moves messages from a DLQ back to their source, or to another queue. Four facts shape how you use it:

• A redriven message is a new message: new ID, new enqueue time, receive count 0. If it fails again, it goes through all five deliveries again.
• It interleaves with new traffic; order is not restored.
• A custom velocity of up to 500 messages a second; a task runs at most 36 hours. (A message moved into a FIFO DLQ has its deduplication ID replaced by its original message ID.)
• It works only for DLQs whose sources are SQS queues, not for an SNS subscription's DLQ (Part 10).

textCopy
REDRIVE RUNBOOK (fulfil-dlq)
  1. look:   read a sample; group by error; confirm it is one cause
  2. fix:    deploy the fix, or get the owning team to (the WMS label step)
  3. test:   redrive ONE message; watch it succeed end to end
  4. move:   StartMessageMoveTask at a capped velocity (10/s here; far below the fleet's 79/s)
  5. watch:  the DLQ draining, the source's age and errors; stop the task if failures return
  6. note:   what failed, how many, when fixed, when redriven

Step R3.9 of the message queue loop uses the same six steps at 50 a second.

Side row 5c (a copy of Q3). Someone redrives at 21:00, before the 21:30 fix. o-7731 becomes a new message with a count of 0, hangs five more times, and is back in the DLQ at 21:04:05 (seed 2: 21:03:25). Nothing was gained, and the WMS took five more hung calls. Look before you move.

Repair instead of replay. Sometimes the message is only a pointer, or the order has changed since it was sent. Then re-read the current state from the source and act on that, instead of redriving the old message: Change Streams & the Transactional Outbox owns this (Part 6 there).

---

Part 6. The batch it drags down

This Part rewinds to 20:00:07 on a copy of Q2 (the same queue and fulfil-dlq, maxReceiveCount 5) and replaces the worker fleet with a Lambda function. Ten orders end up in the dead-letter queue, and only one of them was ever broken.

How a Lambda reads a queue

A Lambda event source mapping polls the queue for you, gathers messages into a batch and invokes your function with the whole batch. You choose:

• the batch size: up to 10,000 for a standard queue, 10 for FIFO; a size above 10 needs a batching window of at least 1 s;
• the batching window: how long to gather messages before invoking, up to 5 minutes, on standard queues only.

When messages are available, Lambda starts with 5 concurrent batches and adds up to 300 more concurrent invokes a minute, up to 1,250 per mapping. (Provisioned mode instead keeps a set of dedicated pollers, each handling up to 1 MB a second and 10 concurrent invokes, and scales faster.) The mapping deletes a batch's messages when the function succeeds. When the invoke fails as a whole, it deletes none of them.

Timeouts that fit. AWS recommends a visibility timeout of at least "six times your function timeout, plus the value of MaximumBatchingWindowInSeconds", so that Lambda can retry a throttled batch before the messages come back. And the function's own timeout must cover a whole batch, including one hung call: 9 healthy orders × about 0.5 s + one 5 s WMS timeout ≈ 9.6 s. With a 10 s function timeout, that batch timed out in 25.5% of 1,000 seeds in the run, so replay L uses 15 s.

Synthesizing vector architecture diagram...

What to notice: the same batch of 10 with the same broken order, and three handlers. "Throws" sends all ten to the DLQ; "Catches, returns an empty list" loses the broken order silently; only "Returns o-7731's ID" sets aside exactly the one message that failed.

A failed batch comes back whole

Where the broken message sits in the batch matters when the handler throws: every mate before it is handled 5 times. Averaged over all ten positions, that is 18 repeat runs per batch; if SQS returns the re-formed batch in a shuffled order, 13.8 (the run: 200 seeds per position). AWS re:Post has the symptom in its title: "Lambda retrying valid SQS messages and placing them in my DLQ".

Report failures by ID

textCopy
BATCH HANDLER WITH ReportBatchItemFailures ON
  failures = []
  for each record in the batch, in order:
      try:
          process(record)                    inbox first, WMS with the key, DB row
      catch any error:
          failures.append(record.messageId)
          on a FIFO queue: also append every record after it, then stop
  return batchItemFailures = one {"itemIdentifier": id} per id in failures

On a FIFO queue, AWS says the function "should stop processing messages after the first failure and return all failed and unprocessed messages", or the group's order breaks. With partial batch responses on, Lambda also doesn't slow its polling down when invokes fail.

Concurrency at the source, not at the function

To stop a burst from flooding the WMS, cap the event source mapping's maximum concurrency (2 to 1,000), not the function's reserved concurrency. When reserved concurrency throttles an invoke, the batch goes back to the queue with every message's receive count raised, and Lambda keeps retrying "until the message's timestamp exceeds your queue's visibility timeout". Healthy messages can reach the DLQ that way: in the AWS Compute Blog's demonstration, throttling sent 9 messages to the DLQ, and maximum concurrency sent none. Keep maximum concurrency at or below the reserved concurrency, and maxReceiveCount at least 5, as AWS advises. (Lambda's account concurrency, 1,000 by default, is shared by every function in the Region: an uncapped mapping can take all of it.)

Kinesis and DynamoDB Streams sources work differently: records are read by position, and a failing batch blocks its shard. Change Streams & the Transactional Outbox owns bisecting a batch, the maximum record age and the on-failure destination (Part 9 there).

---

Part 7. The customer it blocks

This Part rewinds to 20:00:07 on a copy of Q2's policies, applied to a FIFO queue fulfil.fifo with MessageGroupId = the customer. Customer c-19 sends three messages: o-7731 (20:00:07.70), "change the delivery slot for o-7731" (20:00:40.65), and a new order o-7802 (20:01:05.65). About 5,000 other customers keep ordering at 15 a second. Without a dead-letter queue, o-7802 is picked on Tuesday.

Message groups

A FIFO queue keeps order within a message group, and only there. While a group has a message in flight, "subsequent messages in that group are not made available", so a group is worked one receive's worth at a time; different groups run in parallel. A receive "attempts to return as many messages as possible with the same message group ID", which is why our workers ask for one message per receive (Part 2): taking a group's next nine messages and handing them back would burn their receives.

Synthesizing vector architecture diagram...

What to notice: order is kept only inside each group. c-19's two later messages (yellow) wait behind o-7731 (red); the other customers' groups don't wait for anything.

Head-of-line blocking, and why the age alarm works here

A DLQ breaks the order; parking keeps it

AWS says it directly: "Don't use a dead-letter queue with a FIFO queue if you don't want to break the exact order". Step R2.8 of the message queue loop weighs "move to DLQ and continue" against "block the group and page". F3 is a third answer: page 05's error boundary (Change Streams & the Transactional Outbox, Part 6 there), applied to a queue.

Snapshot M6, 20:02:30 (seed 1)

Choosing the key

Side row 7k (a copy of F1, first hour). Group by the smallest thing whose order matters, no wider:

A hot group, like a hot partition key, is a hot key (Sharding, Hot Keys & Rebalancing).

Deduplication is not exactly-once. A FIFO queue drops a repeated send with the same deduplication ID for 5 minutes. It still redelivers a message whose visibility timeout ran out, as Part 3 showed. Idempotency owns this (Part 6 there), and step 1.3 of the notification loop corrects "use an SQS FIFO queue for exactly-once".

Throughput

The transactions a second are shared by every producer and consumer of the queue.

Fair queues: tenants without order

Steps 3.1 of the notification loop and 3.2 of the job scheduler loop face the opposite problem: one tenant's flood delays everyone, and nobody needs order. SQS fair queues solve it on a standard queue: set MessageGroupId to the tenant, and when one tenant has "a disproportionately large number of in-flight messages", SQS "prioritizes message delivery for other tenants". A fair queue keeps no order, and it "does not limit the consumption rate per tenant".

---

Part 8. The partitions it blocks

This Part rewinds to 20:00:07 on a copy of Q1's client (a 5 s WMS timeout, no backoff) reading a Kafka topic instead of a queue. One record stops a third of all customers at once, and ten minutes later nothing reads the topic at all.

Consumer groups and offsets

In a group, each partition is read by one consumer, so a group can't use more consumers than partitions. A consumer's position in each partition is an offset, and committing it is the log's ack point: "the committed offset should always be the offset of the next message that your application will read" (the KafkaConsumer documentation). Commit after the effect is durable, and it is at-least-once; before, at-most-once. Auto-commit commits, at each poll(), the position of the records the previous poll() returned. That is at-least-once only if every record is finished before the next poll(), which our loop does.

Synthesizing vector architecture diagram...

What to notice: o-7731 is on partition 2, but partition 3 stops too, because C2's one poll() loop serves both. Two of six partitions, a third of all customers, wait behind one record.

One record stops a consumer's partitions, then the group

Synthesizing vector architecture diagram...

What to notice: each consumer that inherits partition 2 meets o-7731 and stops within seconds; the red bars follow each other until the group is empty. Stopped partitions grow the lag by 5, then 7.5, then 15 records a second.

The lag metric told the truth throughout: Amazon MSK emits consumer-lag metrics (MaxOffsetLag, SumOffsetLag, EstimatedMaxTimeLag and others) only while a group is STABLE or EMPTY, and here that was every second except the three 3 s rebalances (the third, at 20:15:25, leaves the group empty). A group that never settles, such as consumers crash-looping, has no lag datapoints at all, so alarm on missing data too. The run also shows the cost of each rebalance: 432 healthy records were read twice, and the inbox kept every pick single.

Side row 8a: auto-commit with a thread pool. A consumer that hands each polled record to a pool of 8 threads and polls again at once commits, at its next poll(), the position of records it has only handed off. Kafka's documentation warns that the committed offset can then "get ahead of the consumed position". On a copy, C2 does exactly this; o-7731 and its slot change occupy two pool threads, retrying. C2 is killed at 20:02:00, the group notices after the 45 s session timeout (20:02:45) and gives partition 2 to C1 from the committed offset, which is past o-7731 and the slot change. Both are skipped for good, silently. And the pool had already started o-7802 at 20:01:05.70, ahead of the order in front of it.

Rebalances

Two settings stop a restart from freezing the group (step 2.6 of the message queue loop):

• The cooperative assignor (CooperativeStickyAssignor, listed alone): a rebalance moves only the partitions that change owner, and the others keep flowing. The default list uses Range first, which revokes everything.
• Static membership (group.instance.id): a consumer that restarts within session.timeout.ms gets its partitions back without a rebalance at all. Size session.timeout.ms above a normal restart (45 s by default), so a routine deploy doesn't rebalance; the price is that a consumer that really died holds its partitions that long.

Neither helps a consumer that never calls poll(). That needs the next fix.

The fix: bound the retries, then page 05's boundary

textCopy
CONSUMER LOOP WITH AN ERROR BOUNDARY (K2)
  records = poll()
  for each key in records, in offset order (keys in parallel):
      if key is blocked: park the record in the fulfil DB; next
      for attempt in 1..4:                       waits 0.1 s, 1 s, 5 s between
          try: process(record); break
          catch: if attempt = 4:
                     record it in dead_letters (fulfil DB); block key
  commit the offsets after the whole poll is processed
  (cooperative-sticky assignor, group.instance.id set)

This is Change Streams & the Transactional Outbox's error boundary (Part 6 there): retry a few times, dead-letter in the consumer's own database, block the key, park its later records, and keep committing.

Retry topics (send the failing record to a retry topic and move on) also unblock the partition, but they give up the key's order, like a DLQ on a FIFO queue.

The same poison, four transports

Each column has its strength: the log wins replay and cheap fan-out, the standard queue wins "only this message waits", and FIFO gives order per group without a boundary you build.

Kinesis and DynamoDB Streams through Lambda retry a failing batch until its records expire by default ("a bad record can block processing on the affected shard for up to one week", AWS); bisecting, the maximum record age and the on-failure destination are Change Streams & the Transactional Outbox's (Part 9 there).

---

Part 9. The backlog behind it

Replay B runs on a copy of Q4 without o-7731: from 20:00:00 to 20:10:00 a flash sale sends 100 orders a second against a fleet that finishes about 79, then traffic drops back to 15. At 20:07 about 9,000 orders are waiting (9,085 in seed 1, 8,621 in seed 2; 21 a second × 420 s ≈ 8,800 by arithmetic). How late are they, and which number should page someone?

Two different ages

A queue's backlog answers two questions, and they have different answers.

Formula 1: how long a message arriving now will wait. It sits behind the whole backlog, which drains at the fleet's throughput:

$$\text{wait of a new arrival} \;\approx\; \frac{\text{backlog}}{\text{throughput}}$$

Formula 2: how old the oldest message is. The backlog was built by arrivals, so its oldest message arrived about one backlog's worth of arrivals ago:

$$\text{age of the oldest} \;\approx\; \frac{\text{backlog}}{\text{arrival rate while it built}}$$

They agree only when arrivals ≈ throughput. At 20:10 the backlog is about (100 − 79) × 600 = 12,600. A new order waits about 12,600 ÷ 79 ≈ 160 s; the oldest order is about 12,600 ÷ 100 ≈ 126 s old. And after the burst the oldest keeps ageing: the fleet now works through messages that arrived at 100 a second, advancing 79 ÷ 100 of a second of arrivals per second, so the oldest ages by 0.21 s every second until the fleet reaches messages that arrived after 20:10, about 160 s later. It peaks near 126 + 0.21 × 160 ≈ 160 s at about 20:12:40.

Synthesizing vector architecture diagram...

What to notice: the line that climbs to 166 at second 760 is replay B as described. It keeps rising for more than two minutes after the burst ends at second 600, then falls to zero as the fleet reaches the orders that arrived after the burst. The line that turns down at about 80 is the same burst with scaling on backlog per worker (side row 9s, below). The alarm line is 120 s and the SLO is 300 s; neither is drawn.

Drain time, in words: a backlog drains at capacity minus arrivals, not at capacity. Change Streams & the Transactional Outbox states it for a relay (Part 3 there) and Retries, Timeouts, Backpressure & Load Shedding for request queues (Part 6 there). The drill The Warehouse System That Fell a Day Behind applies it: its backlog grows by 2,200 − 400 = 1,800 orders a minute during a peak of T minutes and shrinks by only 400 − 150 = 250 a minute afterwards, so it drains in 1,800 T ÷ 250 = 7.2 T.

Age is the SLO; depth isn't

Depth says how much work is waiting; age says how late it is, in the units of the SLO. The same 12,600 messages clear in about 160 s at 79 a second whatever built them, but their oldest is 126 s old after a 100-a-second burst, and 12,600 ÷ 15 = 840 s, 14 minutes, if the fleet had stopped and they built up at 15 a second. Alarm on age. But know what each signal can't see:

For Kinesis Data Streams the matching signal is iterator age, and Event Time, Watermarks & Checkpoints explains how it differs from watermark lag (Part 12 there).

Q4 is the alarm set for the main line: age of the oldest > 120 s, DLQ depth ≥ 1, LastAttemptFailed ≥ 1, and more than 30 of the 40 slots in flight for 5 minutes. In replay B the in-flight alarm fires at 20:05:00, five minutes before the age alarm: a full fleet is the early warning, a late message the SLO signal.

Adding consumers

Side row 9s. AWS's method for scaling a queue's consumers (EC2 Auto Scaling) is backlog per worker: the acceptable backlog per worker = the latency you accept ÷ the time one worker takes per message. Here, 60 s of work per worker: a worker finishes 10 ÷ 0.508 s ≈ 19.7 orders a second, so 60 × 19.7 ≈ 1,181 messages per worker. Each minute the controller asks for backlog ÷ 1,181 workers, and a new worker is ready 2 minutes after it is asked for (our figure).

Scaling lags a burst by the launch time plus a datapoint or two, so it trims the peak rather than preventing it. A Lambda consumer ramps by up to 300 concurrent invokes a minute (Part 6), and a log's consumer group can't use more consumers than it has partitions (Part 8).

---

Part 10. The copies it fans out to

At 19:59 a deploy replaces the email queue's access policy, and the new policy drops the statement that lets the orders topic send to it. From 20:00 to 20:20, about 18,000 order confirmations are never sent, o-7731's among them. The publish succeeded, fulfil got every order, and nothing retried.

One publish, one copy per queue

SNS fan-out delivers one copy of each published message to every subscription: here three SQS queues. Each copy then lives its own life: its own visibility timeout, its own receive count, its own DLQ. A copy is wrapped in an SNS JSON envelope unless the subscription turns on raw message delivery. A filter policy on a subscription (on message attributes, or on the message body) decides which copies it gets: b2b takes only messages with channel = "b2b".

Synthesizing vector architecture diagram...

What to notice: the broken policy is on the email queue, not on the topic, so fulfil and b2b keep working. The dashed subscription DLQ exists only in Q5, and the drain script sends its messages straight to the email queue, never back to the topic.

What SNS and EventBridge retry, and what they drop

SNS sorts delivery failures into two kinds:

• Server-side errors (the endpoint's service is unavailable): to SQS and Lambda, SNS retries 3 times at once, 2 times 1 s apart, 10 times with backoff from 1 to 20 s, and 100,000 times 20 s apart: "100,015 times, over 23 days".
• Client-side errors, "when an owner deletes the endpoint … or when an owner changes the policy attached to the subscribed endpoint in a way that prevents Amazon SNS from delivering": SNS doesn't retry, and it discards the message "unless a dead-letter queue is attached to the subscription".

A queue encrypted with a customer managed KMS key is similar: SNS can send to it only if the key's policy grants the SNS service principal GenerateDataKey and Decrypt. AWS doesn't say whether a refused key is retried, which is one more reason to give every subscription a DLQ.

Why a drain script, not a redrive? SQS's redrive task (StartMessageMoveTask) accepts only DLQs "whose sources are other Amazon SQS queues"; a subscription DLQ's source is an SNS subscription. And never re-publish the messages to the topic: fulfil and b2b would get second copies.

textCopy
DRAIN A SUBSCRIPTION DLQ (email-sub-dlq -> email)
  after the cause is fixed, and after one message has gone through end to end:
  loop until email-sub-dlq is empty:
      msgs = ReceiveMessage(email-sub-dlq, up to 10, long poll)
      for each msg:
          SendMessage(email, msg.body)          the original body; raw delivery keeps it unwrapped
          DeleteMessage(email-sub-dlq, msg)     only after the send succeeded
      pace to 100 a second, below what the email senders can take

Those EventBridge figures describe the default bus and custom buses of the Classic type. EventBridge now also offers a newer Custom Event Bus with retention, FIFO delivery within an event group and 5-minute publish-time deduplication; its subscribers retry 5 times or for 300 s by default (settable up to 185 attempts and 24 hours) and drop an event that runs out, leaving only a metric, unless you give them a DLQ.

Size limits

A message that fits the queue can fail at the topic in front of it: SQS takes 1 MiB, but SNS stops at 256 KiB.

Side row 10c. At 20:05:00 a B2B order, o-7790, is 1.4 MB: over SNS's 256 KiB and SQS's 1 MiB. The outbox's size check (page 05) rejects it at the insert, so it never becomes a message that can't be sent. The fix is a claim check: store the payload in S3 and send a pointer of about 0.4 KB.

textCopy
CLAIM CHECK
  producer: PUT the payload to S3 (bucket, key, version); compute a checksum
            publish {order, bucket, key, version, checksum}             about 0.4 KB
  consumer: GET the object by bucket, key and version; check the checksum; process
            never delete the object: every copy of the pointer must still find it
  S3 lifecycle rule: delete objects after 30 days

Why 30 days? The object must outlive every copy of the pointer. A copy that fails lands in a standard DLQ, where it can live 14 days counted from its original enqueue time; after one redrive it is a new message, and if it fails again it can live up to 14 more days in the DLQ: 14 + 14 = 28 days, so a 30-day rule. With fan-out, several consumers read the same object, so no consumer may delete it. The SQS Extended Client Library for Java deletes the S3 object when you delete the message, by default: turn that off. (The Python client's default is off.) Step 1.3 of the email loop keeps raw mail in S3 and passes a reference for the same reason.

---

Part 11. One order, end to end

With Q5, every hop has a guard. Here is o-7731 through all of them (seed 1), from the checkout to the pick list.

Synthesizing vector architecture diagram...

What to notice: every arrow can repeat, and the boxes below each arrow are what make the repeat harmless. The one green box, the delete, comes last, after the effect is durable.

On the main line, none of Q4's backlog alarms fires: at 15 a second the fleet never runs short, and o-7731 is out of the age metric from its 3rd receive. The alarms that caught it were the two built for a poison message.

---

Part 12. On AWS

Every mechanism on this page is a managed feature somewhere in AWS. Three rules carry over: SQS, SNS and EventBridge deliver at least once, so you still choose the ack point and make the effect safe to repeat; each service has its own place where failed work lands, and each must be set and alarmed; and their limits differ, so check each hop.

Managed services that use it

SQS limits in one place

Running it yourself

Queues and their DLQs are Regional: a Region failover needs its own plan for the messages left behind (Multi-Region Failover).

Look-alikes that are not this mechanism

---

Part 13. What you've learned

Back to o-7731

As shipped, o-7731 failed 11,520 times over four days, held 1.33 worker slots and WMS connections the whole time, kept every alarm green and was deleted by retention on Tuesday. With every guard in place (Q5), it was set aside within minutes, someone was paged at 20:02, the WMS team fixed the bug at 21:30, and a redrive picked it at 21:40:00.52 with the one pick list the WMS had created at 20:00:08. Here is what each piece did:

• The ack point after the effect (Part 1) turned a crash from a lost order into a 30 s delay.
• Parallel batches, or receiving one at a time (Part 2), stopped healthy messages from outliving their leases behind a slow one.
• A sized lease with a heartbeat, and the inbox check first (Part 3) rendered o-7740 once, and kept every repeat from making a second pick.
• Counting, backoff and maxReceiveCount (Part 4) cut 11,520 WMS attempts to 5; deferring by re-sending or pausing kept 5,400 healthy orders out of the DLQ during a throttle.
• A watched, 14-day DLQ with a last-attempt metric (Part 5) paged someone at 20:02, and a capped redrive after the fix finished the job.
• Partial batch responses (Part 6), parking a FIFO group (Part 7) and a bounded error boundary on a log (Part 8) kept one bad message from taking its batch, its customer or a third of all customers with it.
• Age alarms and scaling on backlog per worker (Part 9) watched lateness in SLO units.
• A DLQ and a drain script on every subscription, and a claim check (Part 10) kept fan-out from dropping anything silently.

The snapshots, side by side

What it costs

1. Duplicates to make harmless: every consumer needs an inbox or a key, because at-least-once means some messages arrive twice.
2. A visibility timeout to tune: too short gives duplicates and stale deletes; too long makes a crashed worker's message wait. A heartbeat costs a thread and an API call per message.
3. An owner for every DLQ: an alarm, a runbook, and someone who can fix the cause and redrive.
4. Healthy work waits during deferral: pausing or re-sending keeps orders out of the DLQ but makes them late (10 to 11 minutes in replay T).
5. Ordered queues and partitions block: a failing message stops its group or its consumer's partitions, and parking needs tables and a repair job.
6. Metrics that lie in documented ways: the age metric's poison exclusion, NumberOfMessagesSent ignoring moves, late datapoints from idle queues, missing lag for groups that never settle, and an age that resets when a message is re-sent.

The whole story, event by event

Side rows and side replays run on copies and are marked "(side)". Seeds 1 and 2 unless stated; times are seed 1's.

The cheat card

Failure checklist

• Is every delete (or commit) after the effect is durable, and is every effect guarded by an inbox or a key?
• Does each worker receive only what it will start now, and work a batch in parallel?
• Is the visibility timeout above the slowest normal job, with a heartbeat that stops when the work does?
• Do you know that a delete with an old receipt handle may do nothing, and check the inbox before any work?
• Are errors classified, with permanent ones sent to the DLQ at once?
• Do real retries back off with jitter, and does deferral of healthy work avoid spending receives?
• Does every queue have a redrive policy with maxReceiveCount of at least 5?
• Is every DLQ alarmed on its visible depth (not NumberOfMessagesSent), with your own last-attempt metric as well?
• Is every DLQ's retention 14 days, longer than its source's?
• Does every DLQ have an owner, a runbook and a tested, rate-capped redrive, used only after the fix?
• Do Lambda handlers catch per record and return failed IDs, never an empty list after an error, with concurrency capped at the event source?
• On FIFO queues and log partitions, is there a decision for a failing message: block and page, or park the key, rather than a DLQ that silently breaks order?
• Are in-line retries on a log bounded well inside max.poll.interval.ms, with commits after processing?
• Does every SNS subscription have a DLQ, an alarm and a drain script, and does every hop's size limit fit the largest message?

Think-first drills

Interview questions

Where to go next

• Idempotency & Effectively-Once Processing: the keys and the inbox every redelivery relies on (Parts 1 and 6 there), and notifications that can't take a key (Part 7 there).
• Change Streams & the Transactional Outbox: the outbox that published o-7731, the drain formula (Part 3 there), the error boundary on a log (Part 6 there) and Lambda on DynamoDB Streams (Part 9 there).
• Leases, Fencing Tokens & Distributed Locks: the visibility timeout as a lease without a fence (Part 11 there).
• Retries, Timeouts, Backpressure & Load Shedding: error classes, full jitter and drain arithmetic (Parts 3, 4 and 6 there).
• Event Time, Watermarks & Checkpoints: iterator age vs watermark lag on a stream (Part 12 there).
• Sharding, Hot Keys & Rebalancing: a hot message group or partition key is a hot key.
• Multi-Region Failover: queues and their DLQs are Regional.
• Coming later: Rate Limiting Algorithms (the throttles that make you defer work in Part 4).
• Drill: The Warehouse System That Fell a Day Behind, both questions answered in Parts 1, 3, 4, 5 and 9.
• Background: Message Queues vs Event Streams, the broader primitive.
• Loops that rely on this page: the message queue (steps 1.2, 1.5, 1.6, 2.1 to 2.6, 3.6, R2.8, R2.9 and R3.9), notifications (steps 1.1 to 1.3, 2.1, 3.1, R1.9 and R3.9), the job scheduler (steps 1.3, 1.4, 2.5, 2.6 and 3.2), the web crawler (steps 1.1, 1.3, 2.3 and 2.5), email (steps 1.3 and 2.5), YouTube (step 2.2), the outbox ledger (steps 2.3, 3.5 and 3.6), ride-sharing (step 1.4), the news feed (step 1.4), chat (step 2.2), metrics and alerting (step 2.2), the URL shortener (steps 2.4 and 2.5), mobile chat, the digital wallet, the gaming leaderboard, hotel reservations, Google Drive, the offline-first news feed, the mobile stock-trading app, and the Uber and Netflix case studies.