The Claims Change Stream That Stopped on a Friday Evening
The Claims Change Stream That Stopped on a Friday Evening
This situation follows the lesson Change Streams & the Transactional Outbox. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Northlane is an insurance company. Its claims service keeps every claim in a claims table on Amazon RDS for PostgreSQL, and writes an outbox row in the same transaction as every claim change. The outbox is insert-only and split into one partition per day. A Debezium connector reads the outbox through one logical replication slot, claims_slot, and publishes every row to the topic claims.events. Three consumers read the topic: the payout service (it pays approved claims), the search indexer (the claims search page) and the finance summary (a read model of the money reserved per claim, built from events).
| Setting | Value |
|---|---|
| Claim changes | 50 a second, the same by day and by night (assumption, so that every count is one multiplication) |
| Database | RDS for PostgreSQL, one writer, 1,000 GB of storage with 600 GB free (assumption) |
| WAL rate | 4 MB/s for the whole database, all tables together (assumption). 1 GB = 1,000 MB |
| Slot cap | max_slot_wal_keep_size = 150 GB |
| Heartbeat and alarm | The connector writes a heartbeat row every 10 s. An alarm fires when the heartbeat lag has been above 60 s for 5 minutes. The alarm posts to a chat channel that nobody reads at the weekend (assumption) |
| Outbox retention | One partition per day. At 00:00 on day D, a job drops the partition of day D − 4, so every row is kept for at least 3 full days after its day ends |
| Topic | claims.events on Amazon MSK, 6 partitions, key = claim ID, retention 168 hours (7 days). Modelling rule: a record is removed as soon as it is older than 168 hours |
| Republish tool | Reads outbox rows in seq order and publishes them at most 5,000 a second (a cap, because the broker also carries the live traffic); its margin is 10 minutes |
| Consumers | All three apply by version (skip, exactly next, park) and commit their offsets after the effect |
| Rebuild scan | The claims table holds 20,000,000 rows; a scan on the primary reads 10,000 rows a second (assumption) |
Synthesizing vector architecture diagram...
claims_slot pins the WAL from its confirmed position, up to the cap of 150 GB. The Debezium connector reads the outbox through the slot, confirms a heartbeat every 10 s, and publishes to claims.events, which keeps a record for 7 days. The payout service, the search indexer and the finance summary each read the topic at their own pace. So a change that a consumer has not read yet lives in the WAL (until the cap), in the outbox (until its daily partition is dropped) and in the topic (until retention), and the question for every outage is which of the three still has it.The failure. The week's events, in time order:
| When | What happens |
|---|---|
| The previous Sunday 12:00 | The finance summary is stopped for a rewrite. Its planned restart is Monday 12:00 |
| Friday 17:00 | The search indexer is stopped for a planned migration; its offsets are committed. Its planned restart is Monday 10:00 |
| Friday 18:00:00 | A configuration deploy gives the connector a wrong database password. The connector's task fails at once, and every automatic restart fails the same way (assumption). The last heartbeat it confirmed committed at 18:00:00 |
| Friday 18:06 | The heartbeat lag has been above 60 s for 5 minutes: the alarm posts to the chat channel. Nobody reads it |
| Monday 09:00 | An engineer reads the channel. The payout service has paid nothing since Friday 18:00. The claims table itself is correct |
Your task
Write the recovery plan for Monday 09:00 with every number: when the slot was lost and what would have happened without the cap; how many claim changes are missing from the topic, where they still exist and until when; the recovery steps in order and how long they take; and, for each of the two stopped consumers, whether it resumes or rebuilds, and why. End with the one failure this plan still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Claims Change Stream That Stopped on a Friday Evening
This situation follows the lesson Change Streams & the Transactional Outbox. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Northlane is an insurance company. Its claims service keeps every claim in a claims table on Amazon RDS for PostgreSQL, and writes an outbox row in the same transaction as every claim change. The outbox is insert-only and split into one partition per day. A Debezium connector reads the outbox through one logical replication slot, claims_slot, and publishes every row to the topic claims.events. Three consumers read the topic: the payout service (it pays approved claims), the search indexer (the claims search page) and the finance summary (a read model of the money reserved per claim, built from events).
| Setting | Value |
|---|---|
| Claim changes | 50 a second, the same by day and by night (assumption, so that every count is one multiplication) |
| Database | RDS for PostgreSQL, one writer, 1,000 GB of storage with 600 GB free (assumption) |
| WAL rate | 4 MB/s for the whole database, all tables together (assumption). 1 GB = 1,000 MB |
| Slot cap | max_slot_wal_keep_size = 150 GB |
| Heartbeat and alarm | The connector writes a heartbeat row every 10 s. An alarm fires when the heartbeat lag has been above 60 s for 5 minutes. The alarm posts to a chat channel that nobody reads at the weekend (assumption) |
| Outbox retention | One partition per day. At 00:00 on day D, a job drops the partition of day D − 4, so every row is kept for at least 3 full days after its day ends |
| Topic | claims.events on Amazon MSK, 6 partitions, key = claim ID, retention 168 hours (7 days). Modelling rule: a record is removed as soon as it is older than 168 hours |
| Republish tool | Reads outbox rows in seq order and publishes them at most 5,000 a second (a cap, because the broker also carries the live traffic); its margin is 10 minutes |
| Consumers | All three apply by version (skip, exactly next, park) and commit their offsets after the effect |
| Rebuild scan | The claims table holds 20,000,000 rows; a scan on the primary reads 10,000 rows a second (assumption) |
Synthesizing vector architecture diagram...
claims_slot pins the WAL from its confirmed position, up to the cap of 150 GB. The Debezium connector reads the outbox through the slot, confirms a heartbeat every 10 s, and publishes to claims.events, which keeps a record for 7 days. The payout service, the search indexer and the finance summary each read the topic at their own pace. So a change that a consumer has not read yet lives in the WAL (until the cap), in the outbox (until its daily partition is dropped) and in the topic (until retention), and the question for every outage is which of the three still has it.The failure. The week's events, in time order:
| When | What happens |
|---|---|
| The previous Sunday 12:00 | The finance summary is stopped for a rewrite. Its planned restart is Monday 12:00 |
| Friday 17:00 | The search indexer is stopped for a planned migration; its offsets are committed. Its planned restart is Monday 10:00 |
| Friday 18:00:00 | A configuration deploy gives the connector a wrong database password. The connector's task fails at once, and every automatic restart fails the same way (assumption). The last heartbeat it confirmed committed at 18:00:00 |
| Friday 18:06 | The heartbeat lag has been above 60 s for 5 minutes: the alarm posts to the chat channel. Nobody reads it |
| Monday 09:00 | An engineer reads the channel. The payout service has paid nothing since Friday 18:00. The claims table itself is correct |
Your task
Write the recovery plan for Monday 09:00 with every number: when the slot was lost and what would have happened without the cap; how many claim changes are missing from the topic, where they still exist and until when; the recovery steps in order and how long they take; and, for each of the two stopped consumers, whether it resumes or rebuilds, and why. End with the one failure this plan still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).