The Shipping Labels That Were Bought Twice
The Shipping Labels That Were Bought Twice
This situation follows the lesson Idempotency & Effectively-Once Processing. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Kettleby is a web shop that sells kitchen goods. When a warehouse packer marks an order as packed, the order service writes a row to the shipments table and sends one message, shipment_ready, with the shipment ID (for example shp_1) to a queue. A pool of label workers reads the queue. For each message, a worker buys one shipping label from the carrier's HTTP API, writes the label ID into the shipment row, and deletes the message. The carrier bills Kettleby for every label it makes.
| Setting | Value |
|---|---|
| Shipments | 20,000 a day; a month has 30 days |
| Label price | $4.20 per label (assumption) |
| Queue | A standard queue, not a FIFO queue: it delivers every message at least once and has no dedup window. Its visibility timeout is 30 s, the queue's default. A received message stays hidden from other workers for 30 s; if no worker deletes it by then, the queue delivers it again |
| Worker code, as deployed | Receive the message. Read the shipment row; if it already holds a label ID, delete the message and stop. Otherwise call the carrier, write the label ID to the row, then delete the message. A worker reads the row and sends the carrier call 0.1 s after the receive, and it records an answer 0.05 s after the answer arrives |
| Carrier API | POST /labels makes a new label on every call and bills it. It takes no idempotency key. The call carries a customer_reference string, which the carrier stores with the label, and GET /labels?customer_reference=shp_1 returns the labels made with that reference (assumption: the carrier offers this search, and the search sees a label as soon as the carrier makes it) |
| Carrier call time | 2 s normally. In the carrier's busy periods a call takes 40 s. One shipment in 50 (2% of shipments) is processed in a busy period, and every call for that shipment takes 40 s (assumption, fixed values) |
| Worker HTTP timeout | 50 s per carrier call. After it, the worker treats the call as failed and leaves the message in the queue |
| Modelling rules | (1) A worker deletes the message when it finishes, and the delete succeeds whichever receipt handle it uses (assumption; the SQS documentation says a delete with an old receipt handle "might not" delete). (2) The queue delivers an undeleted message again exactly when its visibility timeout ends, and a free worker receives it at that moment. (3) The carrier makes and bills a label at the moment it answers |
Synthesizing vector architecture diagram...
shipment_ready shp_1 to the standard queue. The queue hands shp_1 to one of the label workers. The worker calls the carrier API with POST /labels and the customer reference shp_1, and the carrier makes a label and bills it, because it has no idempotency key and no memory of earlier calls. The worker then writes the label ID to the shipments table. The queue delivers at least once, the carrier bills every call, and the shipments table is written only after the call: no box between the queue and the carrier remembers that a label is already being bought.The failure. At the end of the month, the finance team compares the carrier's invoice with the shipment count. The shop shipped 20,000 × 30 = 600,000 parcels. The carrier billed 612,000 labels, which is 612,000 × $4.20 = $2,570,400 instead of 600,000 × $4.20 = $2,520,000. So 612,000 ÷ 600,000 = 1.02: the shop paid for 2% more labels than parcels, $50,400 more than expected. Every parcel left the warehouse with one label on it. Nobody changed the worker code this month. The error logs are empty: no worker crashed, and no carrier call failed or timed out.
Your task
Explain where the 12,000 extra labels a month come from, with the arithmetic. Then change the design so that one shipment buys one label, even when a carrier call is slow, a worker crashes or an answer is lost. A complete answer contains the cause with its numbers, the design, the two timing rules the design relies on (each with its numbers), and the one failure it still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Shipping Labels That Were Bought Twice
This situation follows the lesson Idempotency & Effectively-Once Processing. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Kettleby is a web shop that sells kitchen goods. When a warehouse packer marks an order as packed, the order service writes a row to the shipments table and sends one message, shipment_ready, with the shipment ID (for example shp_1) to a queue. A pool of label workers reads the queue. For each message, a worker buys one shipping label from the carrier's HTTP API, writes the label ID into the shipment row, and deletes the message. The carrier bills Kettleby for every label it makes.
| Setting | Value |
|---|---|
| Shipments | 20,000 a day; a month has 30 days |
| Label price | $4.20 per label (assumption) |
| Queue | A standard queue, not a FIFO queue: it delivers every message at least once and has no dedup window. Its visibility timeout is 30 s, the queue's default. A received message stays hidden from other workers for 30 s; if no worker deletes it by then, the queue delivers it again |
| Worker code, as deployed | Receive the message. Read the shipment row; if it already holds a label ID, delete the message and stop. Otherwise call the carrier, write the label ID to the row, then delete the message. A worker reads the row and sends the carrier call 0.1 s after the receive, and it records an answer 0.05 s after the answer arrives |
| Carrier API | POST /labels makes a new label on every call and bills it. It takes no idempotency key. The call carries a customer_reference string, which the carrier stores with the label, and GET /labels?customer_reference=shp_1 returns the labels made with that reference (assumption: the carrier offers this search, and the search sees a label as soon as the carrier makes it) |
| Carrier call time | 2 s normally. In the carrier's busy periods a call takes 40 s. One shipment in 50 (2% of shipments) is processed in a busy period, and every call for that shipment takes 40 s (assumption, fixed values) |
| Worker HTTP timeout | 50 s per carrier call. After it, the worker treats the call as failed and leaves the message in the queue |
| Modelling rules | (1) A worker deletes the message when it finishes, and the delete succeeds whichever receipt handle it uses (assumption; the SQS documentation says a delete with an old receipt handle "might not" delete). (2) The queue delivers an undeleted message again exactly when its visibility timeout ends, and a free worker receives it at that moment. (3) The carrier makes and bills a label at the moment it answers |
Synthesizing vector architecture diagram...
shipment_ready shp_1 to the standard queue. The queue hands shp_1 to one of the label workers. The worker calls the carrier API with POST /labels and the customer reference shp_1, and the carrier makes a label and bills it, because it has no idempotency key and no memory of earlier calls. The worker then writes the label ID to the shipments table. The queue delivers at least once, the carrier bills every call, and the shipments table is written only after the call: no box between the queue and the carrier remembers that a label is already being bought.The failure. At the end of the month, the finance team compares the carrier's invoice with the shipment count. The shop shipped 20,000 × 30 = 600,000 parcels. The carrier billed 612,000 labels, which is 612,000 × $4.20 = $2,570,400 instead of 600,000 × $4.20 = $2,520,000. So 612,000 ÷ 600,000 = 1.02: the shop paid for 2% more labels than parcels, $50,400 more than expected. Every parcel left the warehouse with one label on it. Nobody changed the worker code this month. The error logs are empty: no worker crashed, and no carrier call failed or timed out.
Your task
Explain where the 12,000 extra labels a month come from, with the arithmetic. Then change the design so that one shipment buys one label, even when a carrier call is slow, a worker crashes or an answer is lost. A complete answer contains the cause with its numbers, the design, the two timing rules the design relies on (each with its numbers), and the one failure it still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).