The Telemetry Topic That Lost Acknowledged Readings
The Telemetry Topic That Lost Acknowledged Readings
This situation follows the lesson Replication, Quorums & Read-Your-Writes. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Coldhaul is a freight company that moves refrigerated goods. Every truck carries a gateway that sends temperature and location readings to the topic telemetry on Amazon MSK (managed Apache Kafka). Food-safety rules require that a reading the gateway was told was accepted is never lost. Three consumers read the topic: a live map, a cold-chain alerting job, and an archive writer that copies every reading to Amazon S3. We follow one partition of the topic, P17.
| Setting | Value |
|---|---|
| P17's replicas | Three brokers in three Availability Zones: b1 (AZ a) is the leader; b3 (AZ b) and b5 (AZ c) are followers. Replication factor 3 |
| Rate | 1,000 messages a second on P17, 400 bytes each, the same all day (assumption). Offsets count from 0 at 10:00:00, so the message produced at second s has offset 1,000 × s |
| Acknowledgement | Producers send with acks=all and min.insync.replicas = 2. A message is acknowledged when every replica in the in-sync set has stored it. When the in-sync set has fewer than 2 replicas, the broker refuses the write and the producer gets a NotEnoughReplicas error |
| In-sync set | A follower that has not fetched, or has not caught up to the leader's log end, for 30 s (replica.lag.time.max.ms = 30,000) is removed from the in-sync set by the leader |
| Unclean election | unclean.leader.election.enable = true, MSK's default for a cluster without tiered storage: when no in-sync replica is alive, a replica that is not in sync may become the leader |
| Gateways | Each truck sends 5 messages a second of 400 bytes. When the broker refuses a write or does not answer, the gateway keeps the messages in a local spool and retries (assumption) |
| Modelling rules | (1) A replica that returns truncates its own log to the leader's log end before it follows again. (2) A consumer commits the offset of the last message it read. (3) Every repair time below is an assumption |
Synthesizing vector architecture diagram...
acks=all. b1 replicates every message to b3 and b5, and acknowledges a message to the gateway once every replica in the in-sync set has stored it; with min.insync.replicas = 2, that is at least one follower besides b1. The live map, the cold-chain alerting job and the archive writer all read P17 from the leader. So a reading is "accepted" when two of the three brokers hold it, and which two depends on who is in the in-sync set at that moment.The failure. The incident, in time order:
| Time | What happens |
|---|---|
| 10:00:18 | b5's storage volume stalls. b5 stores nothing after the message produced at 10:00:18. Nobody notices |
| 10:00:48 | 30 s later, b1 removes b5 from the in-sync set. The set is {b1, b3}, which is still 2, so producing continues and every new message is acknowledged once b1 and b3 have it |
| 10:04:00 | b1's host fails. b3 is in sync, so the cluster makes b3 the leader at once (a clean election). The in-sync set is {b3}, below 2: from now on the broker refuses writes, producers get NotEnoughReplicas, and the gateways spool |
| 10:04:20 | b3's host fails as well (both hosts came from one bad hardware batch; assumption). No in-sync replica is alive. The cluster elects b5 |
| 10:04:21 | The live map is back, showing readings up to 10:00:18. The alerting job logs an out-of-range offset and restarts from the end of the log |
| 10:30:00 | b3's host is repaired and b3 returns (assumption). Producing resumes |
| The next day | An auditor compares the archive in S3 with the topic: S3 holds readings from 10:00:18 to 10:04:00 that the topic no longer has |
Your task
Explain how a topic with three copies lost readings that were acknowledged, with the numbers: how many readings were lost, at what time the loss became possible, and what each setting did. Then say what you change, compute what the change would have cost during this same incident (who is unavailable, and for how long), and name the one failure the new design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Telemetry Topic That Lost Acknowledged Readings
This situation follows the lesson Replication, Quorums & Read-Your-Writes. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Coldhaul is a freight company that moves refrigerated goods. Every truck carries a gateway that sends temperature and location readings to the topic telemetry on Amazon MSK (managed Apache Kafka). Food-safety rules require that a reading the gateway was told was accepted is never lost. Three consumers read the topic: a live map, a cold-chain alerting job, and an archive writer that copies every reading to Amazon S3. We follow one partition of the topic, P17.
| Setting | Value |
|---|---|
| P17's replicas | Three brokers in three Availability Zones: b1 (AZ a) is the leader; b3 (AZ b) and b5 (AZ c) are followers. Replication factor 3 |
| Rate | 1,000 messages a second on P17, 400 bytes each, the same all day (assumption). Offsets count from 0 at 10:00:00, so the message produced at second s has offset 1,000 × s |
| Acknowledgement | Producers send with acks=all and min.insync.replicas = 2. A message is acknowledged when every replica in the in-sync set has stored it. When the in-sync set has fewer than 2 replicas, the broker refuses the write and the producer gets a NotEnoughReplicas error |
| In-sync set | A follower that has not fetched, or has not caught up to the leader's log end, for 30 s (replica.lag.time.max.ms = 30,000) is removed from the in-sync set by the leader |
| Unclean election | unclean.leader.election.enable = true, MSK's default for a cluster without tiered storage: when no in-sync replica is alive, a replica that is not in sync may become the leader |
| Gateways | Each truck sends 5 messages a second of 400 bytes. When the broker refuses a write or does not answer, the gateway keeps the messages in a local spool and retries (assumption) |
| Modelling rules | (1) A replica that returns truncates its own log to the leader's log end before it follows again. (2) A consumer commits the offset of the last message it read. (3) Every repair time below is an assumption |
Synthesizing vector architecture diagram...
acks=all. b1 replicates every message to b3 and b5, and acknowledges a message to the gateway once every replica in the in-sync set has stored it; with min.insync.replicas = 2, that is at least one follower besides b1. The live map, the cold-chain alerting job and the archive writer all read P17 from the leader. So a reading is "accepted" when two of the three brokers hold it, and which two depends on who is in the in-sync set at that moment.The failure. The incident, in time order:
| Time | What happens |
|---|---|
| 10:00:18 | b5's storage volume stalls. b5 stores nothing after the message produced at 10:00:18. Nobody notices |
| 10:00:48 | 30 s later, b1 removes b5 from the in-sync set. The set is {b1, b3}, which is still 2, so producing continues and every new message is acknowledged once b1 and b3 have it |
| 10:04:00 | b1's host fails. b3 is in sync, so the cluster makes b3 the leader at once (a clean election). The in-sync set is {b3}, below 2: from now on the broker refuses writes, producers get NotEnoughReplicas, and the gateways spool |
| 10:04:20 | b3's host fails as well (both hosts came from one bad hardware batch; assumption). No in-sync replica is alive. The cluster elects b5 |
| 10:04:21 | The live map is back, showing readings up to 10:00:18. The alerting job logs an out-of-range offset and restarts from the end of the log |
| 10:30:00 | b3's host is repaired and b3 returns (assumption). Producing resumes |
| The next day | An auditor compares the archive in S3 with the topic: S3 holds readings from 10:00:18 to 10:04:00 that the topic no longer has |
Your task
Explain how a topic with three copies lost readings that were acknowledged, with the numbers: how many readings were lost, at what time the loss became possible, and what each setting did. Then say what you change, compute what the change would have cost during this same incident (who is unavailable, and for how long), and name the one failure the new design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).