The Claims That Lambda Discarded While No Alarm Fired
The Claims That Lambda Discarded While No Alarm Fired
This situation follows the lesson Queues & Delivery Semantics. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Claimly is an insurance claims platform. When a customer files a claim, the claims service commits the claim row and an outbox row in one transaction, and an outbox relay publishes a ClaimCreated message to the SNS topic claims. The topic has three subscriptions:
- the SQS queue
payout, read by workers that pay approved claims; - the SQS queue
analytics, read by the reporting service; - the AWS Lambda function
fraud-check, subscribed directly to the topic, so that SNS invokes it for every message.
fraud-check sends the claim to a scoring model and writes the score into a scores table, one row per claim ID. Writing a score is an absolute write: a second write for the same claim stores the same row. A fraud analyst reads the scores table every hour and holds every claim whose score is above 0.8. The payout workers do not wait for the score (a business choice made before this incident). A claim that has no score row is never held.
| Setting | Value |
|---|---|
| Claims | About 40,000 a day: 5 a second from 09:00 to 10:00 and 0.27 a second the rest of the day (assumption), arriving at random |
fraud-check | A Lambda function. The scoring call takes 2 s (assumption); the function's timeout is 10 s; no reserved concurrency; no dead-letter queue and no on-failure destination |
| The broker system | 3% of claims come from a partner system, Brokerline, whose amount field is text with a thousands separator, "1,250.00". The function's parser throws on it before the scoring call: an error in the function's code, the same on every attempt |
| Lambda's rule for an asynchronous invoke (AWS documentation) | After a function error, Lambda runs the event two more times, waiting one minute before the second attempt and two minutes before the third. After the third failure it discards the event, unless a dead-letter queue or an on-failure destination is configured |
| Alarms as shipped | One alarm, on the depth of payout-dlq. A dashboard shows the function's Errors metric, with no alarm on it |
Synthesizing vector architecture diagram...
payout and analytics, are SQS queues: a copy there is leased, counted and, if a redrive policy exists, moved to a dead-letter queue. The third copy, drawn in red, is not a queue that Claimly owns. SNS hands the message to Lambda, which puts it in its own queue for asynchronous invokes, runs the function up to three times by its own rule, and then discards the event. The payout workers pay without reading the scores table, so a claim whose third attempt failed is paid and never held.What happened. Three weeks after Brokerline connected, the finance team found 25,200 claims that had been paid with no row in the scores table. During those three weeks the dashboard showed about 3,600 errors a day on fraud-check, every day; nobody connected the number to missing scores. The payout-dlq alarm never fired, because payout had no problem.
Your task
Say at which hop and at which ack point the 25,200 claims were lost and why the Errors metric did not stop it. Then design the fraud-check path so that no claim loses its score silently, a claim that fails is set aside within minutes and someone is paged, and the claims that failed are scored after the fix without a second copy reaching payout or analytics. A complete answer has the loss mechanism with its timing (the three attempts and the discard), the errors a day the metric showed, your design with its settings (the visibility timeout and how you derived it, the receive limit, the alarms), the time from a bad claim's arrival to the page, and one failure the design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Claims That Lambda Discarded While No Alarm Fired
This situation follows the lesson Queues & Delivery Semantics. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Claimly is an insurance claims platform. When a customer files a claim, the claims service commits the claim row and an outbox row in one transaction, and an outbox relay publishes a ClaimCreated message to the SNS topic claims. The topic has three subscriptions:
- the SQS queue
payout, read by workers that pay approved claims; - the SQS queue
analytics, read by the reporting service; - the AWS Lambda function
fraud-check, subscribed directly to the topic, so that SNS invokes it for every message.
fraud-check sends the claim to a scoring model and writes the score into a scores table, one row per claim ID. Writing a score is an absolute write: a second write for the same claim stores the same row. A fraud analyst reads the scores table every hour and holds every claim whose score is above 0.8. The payout workers do not wait for the score (a business choice made before this incident). A claim that has no score row is never held.
| Setting | Value |
|---|---|
| Claims | About 40,000 a day: 5 a second from 09:00 to 10:00 and 0.27 a second the rest of the day (assumption), arriving at random |
fraud-check | A Lambda function. The scoring call takes 2 s (assumption); the function's timeout is 10 s; no reserved concurrency; no dead-letter queue and no on-failure destination |
| The broker system | 3% of claims come from a partner system, Brokerline, whose amount field is text with a thousands separator, "1,250.00". The function's parser throws on it before the scoring call: an error in the function's code, the same on every attempt |
| Lambda's rule for an asynchronous invoke (AWS documentation) | After a function error, Lambda runs the event two more times, waiting one minute before the second attempt and two minutes before the third. After the third failure it discards the event, unless a dead-letter queue or an on-failure destination is configured |
| Alarms as shipped | One alarm, on the depth of payout-dlq. A dashboard shows the function's Errors metric, with no alarm on it |
Synthesizing vector architecture diagram...
payout and analytics, are SQS queues: a copy there is leased, counted and, if a redrive policy exists, moved to a dead-letter queue. The third copy, drawn in red, is not a queue that Claimly owns. SNS hands the message to Lambda, which puts it in its own queue for asynchronous invokes, runs the function up to three times by its own rule, and then discards the event. The payout workers pay without reading the scores table, so a claim whose third attempt failed is paid and never held.What happened. Three weeks after Brokerline connected, the finance team found 25,200 claims that had been paid with no row in the scores table. During those three weeks the dashboard showed about 3,600 errors a day on fraud-check, every day; nobody connected the number to missing scores. The payout-dlq alarm never fired, because payout had no problem.
Your task
Say at which hop and at which ack point the 25,200 claims were lost and why the Errors metric did not stop it. Then design the fraud-check path so that no claim loses its score silently, a claim that fails is set aside within minutes and someone is paged, and the claims that failed are scored after the fix without a second copy reaching payout or analytics. A complete answer has the loss mechanism with its timing (the three attempts and the discard), the errors a day the metric showed, your design with its settings (the visibility timeout and how you derived it, the receive limit, the alarms), the time from a bad claim's arrival to the page, and one failure the design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).