The Late Meter Readings That Overwrote a Closed Billing Block
The Late Meter Readings That Overwrote a Closed Billing Block
This situation follows the lesson Event Time, Watermarks & Checkpoints. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Gridly is an electricity distribution company. It runs the power lines of one city and serves 120,000 homes. Every home has a smart meter. Every 60 seconds the meter sends one reading: the meter's own clock time (the event time) and the energy used since the previous reading, in watt-hours (Wh; 1,000 Wh is one kilowatt-hour, the unit on an electricity bill). The readings travel over the mobile phone network to Gridly's ingest service, which appends them to an Amazon MSK (Managed Streaming for Apache Kafka) topic with four partitions, P1 to P4, keyed by meter id.
A feeder is one power line from a substation. Gridly has 120 feeders, and each serves about 1,000 homes. The grid operator buys from Gridly the energy of each feeder in 15-minute blocks, and pays from those numbers. The block named 09:00 covers 09:00:00 to 09:14:59.999. A stream job on Apache Flink sums the readings per feeder per block, by event time, and writes each block's total to an Amazon DynamoDB table feeder_blocks. The grid operator reads that table.
The team that built the job read the lesson Event Time, Watermarks & Checkpoints and built its checkpoint barriers (the section Barriers, step by step) and its idempotent sink (the sections Each writer tags its own attributes and The full flush, and the index that finds it). It did not build the watermark floor (the section The watermark floor), and it did not build the closed-row condition (the section Closed rows are final). Everything else follows that lesson's rules.
| Setting | This challenge | At real scale |
|---|---|---|
| Readings | Feeder F-12 has four meters, m1 to m4. Each sends one reading a minute at a fixed second: m1 at :05, m2 at :15, m3 at :25, m4 at :35. The load is constant (an assumption, to keep the arithmetic short): every reading of m1 is 110 Wh, of m2 90 Wh, of m3 130 Wh, of m4 70 Wh, so one minute of the four meters is 110 + 90 + 130 + 70 = 400 Wh | 120,000 meters, one reading each per 60 s: 120,000 ÷ 60 = 2,000 readings a second; about 1,000 meters per feeder |
| Partitions | m1 on P1, m2 on P2, m3 on P3, m4 on P4 (an assumption: a real partition carries thousands of meters) | 32 partitions |
| Read delay | The job reads a reading 0.3 s after the meter sends it, when the network is up | Same |
| Event time | The meter's clock. Assume every meter clock is correct | Clamped at ingest |
| Watermark per partition | Highest event time read − 20 s allowance − 1 ms, emitted every 200 ms of processing time, on the 200 ms marks of the wall clock | Same |
| Job watermark | Minimum over active partitions; never goes back | Same |
| Idleness | A partition with no records for 60 s is marked idle and left out of the minimum | Same |
| Windows | 15-minute tumbling windows by event time, keyed by feeder; allowed lateness 0; a late reading goes to a side output and into a late total per feeder and block | Same |
| Early firings | None. A block is written once, when it closes | Same |
| Sink | DynamoDB feeder_blocks, one item per (feeder, block). The window writes block_wh when the block closes; the late path writes late_wh. Both are absolute SETs. Each writer tags its own attributes with the id of the last checkpoint barrier its sink subtask passed (block_ckpt, late_ckpt). After every restore, before any new output, the job runs the full flush: every value tagged at or above the restored checkpoint id is removed, or rewritten from restored state if state holds it. No watermark floor. No closed-row condition. The energy the grid operator pays for is block_wh + late_wh | 120 feeders × 96 blocks a day = 11,520 items a day |
| Checkpoints | Every 60 s, on the minute, aligned, each complete 1 s after its barrier: checkpoint 121 at 09:16:00, checkpoint 122 at 09:17:00 | Same |
| Restart | 40 s from the crash to the job running again (an assumption) | Same |
| Before the story | At 09:14:00 the window for F-12, block 09:00, already holds the block's readings from 09:00:05 to 09:13:35: 14 minutes × 400 Wh = 5,600 Wh. P4's last reading so far, m4's 09:13:35, was read at 09:13:35.3 |
Synthesizing vector architecture diagram...
block_wh and the late path writes late_wh to DynamoDB feeder_blocks, each tagged with the last barrier passed, and the flush runs after a restore. The dashed arrow is the checkpoint: it saves the read positions, the open windows and the late totals, and nothing else.What happens. A meter that cannot reach the tower stores its readings and sends them all when the network is back. The tower is down from 09:14:30 to 09:17:00, so m4's readings of 09:14:35, 09:15:35 and 09:16:35 are all sent at 09:17:02, and the job reads them at 09:17:02.0, in the order the meter took them. At 09:17:30 the worker that runs the job dies. The last complete checkpoint is 122 (barrier at 09:17:00, complete at 09:17:01). The job is running again at 09:18:10, restored from checkpoint 122. Every reading from 09:13:35 to 09:18:05, for your trace:
| Reading | Partition | Event time | Read at | Wh | Block |
|---|---|---|---|---|---|
| m4 | P4 | 09:13:35 | 09:13:35.3 | 70 | 09:00 (already in the 5,600) |
| m1 | P1 | 09:14:05 | 09:14:05.3 | 110 | 09:00 |
| m2 | P2 | 09:14:15 | 09:14:15.3 | 90 | 09:00 |
| m3 | P3 | 09:14:25 | 09:14:25.3 | 130 | 09:00 |
| m4 | P4 | 09:14:35 | 09:17:02.0 (stored in the meter) | 70 | 09:00 |
| m1 | P1 | 09:15:05 | 09:15:05.3 | 110 | 09:15 |
| m2 | P2 | 09:15:15 | 09:15:15.3 | 90 | 09:15 |
| m3 | P3 | 09:15:25 | 09:15:25.3 | 130 | 09:15 |
| m4 | P4 | 09:15:35 | 09:17:02.0 (stored in the meter) | 70 | 09:15 |
| m1 | P1 | 09:16:05 | 09:16:05.3 | 110 | 09:15 |
| m2 | P2 | 09:16:15 | 09:16:15.3 | 90 | 09:15 |
| m3 | P3 | 09:16:25 | 09:16:25.3 | 130 | 09:15 |
| m4 | P4 | 09:16:35 | 09:17:02.0 (stored in the meter) | 70 | 09:15 |
| m1 | P1 | 09:17:05 | 09:17:05.3 | 110 | 09:15 |
| m2 | P2 | 09:17:15 | 09:17:15.3 | 90 | 09:15 |
| m3 | P3 | 09:17:25 | 09:17:25.3 | 130 | 09:15 |
| m4 | P4 | 09:17:35 | Waits in P4: the job is down from 09:17:30 to 09:18:10 | 70 | 09:15 |
| m1 | P1 | 09:18:05 | Waits in P1 | 110 | 09:15 |
After the restart, the job reads every waiting reading at once, in read order, 10 ms apart, before its first watermark emission at 09:18:10.2. Live readings continue every minute at the same seconds.
Your task
Predict the values of block_wh and late_wh for feeder F-12, block 09:00, in feeder_blocks at 09:19:00, with the arithmetic that leads to them: when the block closes and with what sum, what checkpoint 122 holds, what the flush removes at 09:18:10, and what the replay decides about m4's 09:14:35 reading. Then change the design so that the item is right, and name the one case your change still does not cover.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Late Meter Readings That Overwrote a Closed Billing Block
This situation follows the lesson Event Time, Watermarks & Checkpoints. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Gridly is an electricity distribution company. It runs the power lines of one city and serves 120,000 homes. Every home has a smart meter. Every 60 seconds the meter sends one reading: the meter's own clock time (the event time) and the energy used since the previous reading, in watt-hours (Wh; 1,000 Wh is one kilowatt-hour, the unit on an electricity bill). The readings travel over the mobile phone network to Gridly's ingest service, which appends them to an Amazon MSK (Managed Streaming for Apache Kafka) topic with four partitions, P1 to P4, keyed by meter id.
A feeder is one power line from a substation. Gridly has 120 feeders, and each serves about 1,000 homes. The grid operator buys from Gridly the energy of each feeder in 15-minute blocks, and pays from those numbers. The block named 09:00 covers 09:00:00 to 09:14:59.999. A stream job on Apache Flink sums the readings per feeder per block, by event time, and writes each block's total to an Amazon DynamoDB table feeder_blocks. The grid operator reads that table.
The team that built the job read the lesson Event Time, Watermarks & Checkpoints and built its checkpoint barriers (the section Barriers, step by step) and its idempotent sink (the sections Each writer tags its own attributes and The full flush, and the index that finds it). It did not build the watermark floor (the section The watermark floor), and it did not build the closed-row condition (the section Closed rows are final). Everything else follows that lesson's rules.
| Setting | This challenge | At real scale |
|---|---|---|
| Readings | Feeder F-12 has four meters, m1 to m4. Each sends one reading a minute at a fixed second: m1 at :05, m2 at :15, m3 at :25, m4 at :35. The load is constant (an assumption, to keep the arithmetic short): every reading of m1 is 110 Wh, of m2 90 Wh, of m3 130 Wh, of m4 70 Wh, so one minute of the four meters is 110 + 90 + 130 + 70 = 400 Wh | 120,000 meters, one reading each per 60 s: 120,000 ÷ 60 = 2,000 readings a second; about 1,000 meters per feeder |
| Partitions | m1 on P1, m2 on P2, m3 on P3, m4 on P4 (an assumption: a real partition carries thousands of meters) | 32 partitions |
| Read delay | The job reads a reading 0.3 s after the meter sends it, when the network is up | Same |
| Event time | The meter's clock. Assume every meter clock is correct | Clamped at ingest |
| Watermark per partition | Highest event time read − 20 s allowance − 1 ms, emitted every 200 ms of processing time, on the 200 ms marks of the wall clock | Same |
| Job watermark | Minimum over active partitions; never goes back | Same |
| Idleness | A partition with no records for 60 s is marked idle and left out of the minimum | Same |
| Windows | 15-minute tumbling windows by event time, keyed by feeder; allowed lateness 0; a late reading goes to a side output and into a late total per feeder and block | Same |
| Early firings | None. A block is written once, when it closes | Same |
| Sink | DynamoDB feeder_blocks, one item per (feeder, block). The window writes block_wh when the block closes; the late path writes late_wh. Both are absolute SETs. Each writer tags its own attributes with the id of the last checkpoint barrier its sink subtask passed (block_ckpt, late_ckpt). After every restore, before any new output, the job runs the full flush: every value tagged at or above the restored checkpoint id is removed, or rewritten from restored state if state holds it. No watermark floor. No closed-row condition. The energy the grid operator pays for is block_wh + late_wh | 120 feeders × 96 blocks a day = 11,520 items a day |
| Checkpoints | Every 60 s, on the minute, aligned, each complete 1 s after its barrier: checkpoint 121 at 09:16:00, checkpoint 122 at 09:17:00 | Same |
| Restart | 40 s from the crash to the job running again (an assumption) | Same |
| Before the story | At 09:14:00 the window for F-12, block 09:00, already holds the block's readings from 09:00:05 to 09:13:35: 14 minutes × 400 Wh = 5,600 Wh. P4's last reading so far, m4's 09:13:35, was read at 09:13:35.3 |
Synthesizing vector architecture diagram...
block_wh and the late path writes late_wh to DynamoDB feeder_blocks, each tagged with the last barrier passed, and the flush runs after a restore. The dashed arrow is the checkpoint: it saves the read positions, the open windows and the late totals, and nothing else.What happens. A meter that cannot reach the tower stores its readings and sends them all when the network is back. The tower is down from 09:14:30 to 09:17:00, so m4's readings of 09:14:35, 09:15:35 and 09:16:35 are all sent at 09:17:02, and the job reads them at 09:17:02.0, in the order the meter took them. At 09:17:30 the worker that runs the job dies. The last complete checkpoint is 122 (barrier at 09:17:00, complete at 09:17:01). The job is running again at 09:18:10, restored from checkpoint 122. Every reading from 09:13:35 to 09:18:05, for your trace:
| Reading | Partition | Event time | Read at | Wh | Block |
|---|---|---|---|---|---|
| m4 | P4 | 09:13:35 | 09:13:35.3 | 70 | 09:00 (already in the 5,600) |
| m1 | P1 | 09:14:05 | 09:14:05.3 | 110 | 09:00 |
| m2 | P2 | 09:14:15 | 09:14:15.3 | 90 | 09:00 |
| m3 | P3 | 09:14:25 | 09:14:25.3 | 130 | 09:00 |
| m4 | P4 | 09:14:35 | 09:17:02.0 (stored in the meter) | 70 | 09:00 |
| m1 | P1 | 09:15:05 | 09:15:05.3 | 110 | 09:15 |
| m2 | P2 | 09:15:15 | 09:15:15.3 | 90 | 09:15 |
| m3 | P3 | 09:15:25 | 09:15:25.3 | 130 | 09:15 |
| m4 | P4 | 09:15:35 | 09:17:02.0 (stored in the meter) | 70 | 09:15 |
| m1 | P1 | 09:16:05 | 09:16:05.3 | 110 | 09:15 |
| m2 | P2 | 09:16:15 | 09:16:15.3 | 90 | 09:15 |
| m3 | P3 | 09:16:25 | 09:16:25.3 | 130 | 09:15 |
| m4 | P4 | 09:16:35 | 09:17:02.0 (stored in the meter) | 70 | 09:15 |
| m1 | P1 | 09:17:05 | 09:17:05.3 | 110 | 09:15 |
| m2 | P2 | 09:17:15 | 09:17:15.3 | 90 | 09:15 |
| m3 | P3 | 09:17:25 | 09:17:25.3 | 130 | 09:15 |
| m4 | P4 | 09:17:35 | Waits in P4: the job is down from 09:17:30 to 09:18:10 | 70 | 09:15 |
| m1 | P1 | 09:18:05 | Waits in P1 | 110 | 09:15 |
After the restart, the job reads every waiting reading at once, in read order, 10 ms apart, before its first watermark emission at 09:18:10.2. Live readings continue every minute at the same seconds.
Your task
Predict the values of block_wh and late_wh for feeder F-12, block 09:00, in feeder_blocks at 09:19:00, with the arithmetic that leads to them: when the block closes and with what sum, what checkpoint 122 holds, what the flush removes at 09:18:10, and what the replay decides about m4's 09:14:35 reading. Then change the design so that the item is right, and name the one case your change still does not cover.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).