The queue is healthy; the customer is still waiting

A hypothetical order worker deletes a message as soon as it receives it. Its dashboard shows an empty queue and no failures. The process then crashes before shipping the order.

The queue did what it was told: it forgot the only outstanding instruction.

Move the acknowledgement and watch the failure move

A received work-queue message is often hidden or leased rather than permanently removed. Its visibility timeout permits recovery when the worker disappears. Extending the lease does not stop a paused earlier attempt; protect the effect rather than trusting exclusive delivery.

pseudocodeCopy
receive only work you can start
validate message and intent identity
apply local inbox + effect atomically, or reconcile keyed external effect
record durable outcome
acknowledge this delivery

This is the worker protocol's shape; broker-specific receipt handles, deadlines, and extension limits must be checked against the service contract. A batch starts its messages' timers together, so receiving a large batch and processing it serially can expire the later messages before they start.

The message that never succeeds

Classify failures. A transient dependency failure can retry with bounded jitter/backoff. Invalid schema or a permanent business rejection requires a deliberate terminal path. An ambiguous side effect requires reconciliation with the same key, not classification as definitely failed.

A poison message retried forever can monopolize workers or block a partition. A dead-letter queue makes the failure inspectable, but it is another queue requiring retention, ownership, and alarms. Record the original event ID, failure reason, attempts, and timestamps. Moving it to a DLQ must itself avoid a send/delete loss window, usually by sending durably before acknowledging the original and tolerating duplicates.

Redrive is an operation: inspect, fix the cause, try one, replay at a bounded rate, and observe effects. Do not dump the entire DLQ into a recovering dependency. A re-enqueued message may get a new transport ID; business idempotency must use the original intent.

A stream keeps history instead of handing out tasks

A retained event log allows independent consumers to replay by position. One consumer group divides its partitions among workers; several groups independently consume the history. A work queue usually assigns each task delivery to one competing worker and removes completed work.

Ordering is scoped. A FIFO group or log partition serializes its own sequence; it does not order all business events globally. The smallest entity requiring order is often a useful key. Too few partitions cap parallelism; a hot entity can dominate one. Adding partitions can remap keys and must be treated as an ordering transition.

If a poison event is removed from an ordered stream, later updates may become invalid. Park that entity's later events or repair its state before advancing, rather than assuming a DLQ preserves order. In a consumer handling several partitions, one stuck callback can stall more than the bad partition; bound retry time and isolate work where the client permits it.

Commit offsets only after protected effects. Handle rebalances by stopping revoked work, preserving progress, and tolerating repeated delivery. Partial batch failure reporting should identify only failed records when the integration supports it; failing an entire batch can repeat healthy effects.

Age tells a different story from depth

For constant rates and a backlog of $B$, a new arrival waits roughly $B/\mu$ before service at throughput $\mu$. Draining while new work arrives at $\lambda$ takes $B/(\mu-\lambda)$ only if $\mu>\lambda$. The age of the oldest item depends on its history and may be distorted by poison-message handling or redrive.

Measure end-to-end business age as well as broker depth/age, in-flight saturation, consumer lag, retries, and DLQ presence. A green queue can hide a lost instruction, an exhausted retention horizon, or an unhealthy fan-out subscription. Each subscription needs its own delivery-failure handling.

For large payloads, a claim-check message can point to an immutable object plus checksum/version. Retain the object longer than every delivery/replay horizon and authorize access to it; a durable pointer to a deleted object is not a durable task.

Mental model: transport delivers attempts. Completion belongs to the durable business effect and its acknowledgement boundary.

Transfer: the long job

---

Source trail: 05-message-queues-vs-event-streams, 12-queues-and-delivery-semantics. See the source map for consolidation and reference-retained material.