The Feed Export That Looped on Chunk 3
The Feed Export That Looped on Chunk 3
This situation follows the lesson Leases, Fencing Tokens & Distributed Locks. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Catalogo is a shopping-comparison service. Every night it exports each merchant's product catalog as a feed: a set of files that the merchant's partners download. One export job per merchant builds the feed in chunks. Each chunk is a sequence of batch writes to the feed store, one batch every second, and each batch write is a conditional write that carries the job's fencing token. Exactly one worker may build a feed at a time, so each job is guarded by a lease on a key in a lock service of three nodes that replicate through a consensus log, like etcd. Two workers, W and S, are started for each merchant. Both create a key under the job's election prefix; the key with the lower revision leads, and the other worker watches the leader's key and takes over when that key is deleted, as in the section An election has no orphan of the lesson Leases, Fencing Tokens & Distributed Locks.
The settings, with every value the problem needs:
| Setting | Value | Where it comes from |
|---|---|---|
| Lease length T | 8 s | The team's choice |
| Renewal interval | 2 s, a quarter of the lease | The team's choice |
| Holder's margin | 2 s. Stop time = the time the last successful renewal was sent + 8 − 2, on the holder's monotonic clock | The lesson's formula, in the section Renew and the holder's stop time |
| Fencing token | The key's create revision. The revision counter starts at 20: W's key is revision 21, S's key is revision 22. The counter rises by one per key create and per key delete; unrelated keys are left out (an assumption) | The lesson |
| Takeover timing | The waiter's watch fires 0.1 s after the holder's key is deleted. The new holder raises the fence 0.1 s later, and starts its chunk 0.1 s after the raise | Assumptions |
| Network and clocks | A keep-alive is applied at the moment it is sent (no delay). Every clock is correct and monotonic | Assumptions: this failure is not about clocks or the network |
| The feed store | fence (the highest token seen) and chunks_done. A chunk counts as done when its last batch write is accepted | The lesson |
| The merchant "Bricks & Co" | 4 chunks of 6, 6, 11 and 6 batches. One batch write a second, the first 1 s after the chunk starts; the next chunk starts 0.1 s after the last write | Measured in the last export |
| The worker's code | It checks its stop time once, at the start of each chunk. Its heartbeat thread lives in the same process. During a chunk, the chunk's work takes all the CPU time, so the heartbeat thread runs only between chunks: a starved thread | Found in the incident |
| A waiter's key | Renewed every 2 s while the worker waits. A waiting worker is idle, so its heartbeat thread runs | The same code, idle |
| A rejected holder | Drops the job at once and re-campaigns 0.1 s later with a new key | The worker's code |
Synthesizing vector architecture diagram...
The failure. Small merchants export without a problem. The export of "Bricks & Co" never finishes. The log shows W writing chunk 3, then W's writes rejected ("token 21, fence 22"), then S writing chunk 3 from its first batch, then S's writes rejected ("token 22, fence 24"), then W writing chunk 3 from its first batch again, for the whole night. No worker crashed and no worker paused. Both were busy the whole time.
Your task
Explain, with times, why the export of "Bricks & Co" loops on chunk 3, name the rule in the lesson that the worker's code breaks, and count the writes that break it. Then redesign the worker so that the 11-batch chunk completes with the same 8-second lease: say what changes in the code, show with the stop-time formula why it is safe, give the earliest and latest takeover after a real crash of the holder under the new design (with and without the lock service's leader failing at the same moment), and the one failure the new design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Feed Export That Looped on Chunk 3
This situation follows the lesson Leases, Fencing Tokens & Distributed Locks. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Catalogo is a shopping-comparison service. Every night it exports each merchant's product catalog as a feed: a set of files that the merchant's partners download. One export job per merchant builds the feed in chunks. Each chunk is a sequence of batch writes to the feed store, one batch every second, and each batch write is a conditional write that carries the job's fencing token. Exactly one worker may build a feed at a time, so each job is guarded by a lease on a key in a lock service of three nodes that replicate through a consensus log, like etcd. Two workers, W and S, are started for each merchant. Both create a key under the job's election prefix; the key with the lower revision leads, and the other worker watches the leader's key and takes over when that key is deleted, as in the section An election has no orphan of the lesson Leases, Fencing Tokens & Distributed Locks.
The settings, with every value the problem needs:
| Setting | Value | Where it comes from |
|---|---|---|
| Lease length T | 8 s | The team's choice |
| Renewal interval | 2 s, a quarter of the lease | The team's choice |
| Holder's margin | 2 s. Stop time = the time the last successful renewal was sent + 8 − 2, on the holder's monotonic clock | The lesson's formula, in the section Renew and the holder's stop time |
| Fencing token | The key's create revision. The revision counter starts at 20: W's key is revision 21, S's key is revision 22. The counter rises by one per key create and per key delete; unrelated keys are left out (an assumption) | The lesson |
| Takeover timing | The waiter's watch fires 0.1 s after the holder's key is deleted. The new holder raises the fence 0.1 s later, and starts its chunk 0.1 s after the raise | Assumptions |
| Network and clocks | A keep-alive is applied at the moment it is sent (no delay). Every clock is correct and monotonic | Assumptions: this failure is not about clocks or the network |
| The feed store | fence (the highest token seen) and chunks_done. A chunk counts as done when its last batch write is accepted | The lesson |
| The merchant "Bricks & Co" | 4 chunks of 6, 6, 11 and 6 batches. One batch write a second, the first 1 s after the chunk starts; the next chunk starts 0.1 s after the last write | Measured in the last export |
| The worker's code | It checks its stop time once, at the start of each chunk. Its heartbeat thread lives in the same process. During a chunk, the chunk's work takes all the CPU time, so the heartbeat thread runs only between chunks: a starved thread | Found in the incident |
| A waiter's key | Renewed every 2 s while the worker waits. A waiting worker is idle, so its heartbeat thread runs | The same code, idle |
| A rejected holder | Drops the job at once and re-campaigns 0.1 s later with a new key | The worker's code |
Synthesizing vector architecture diagram...
The failure. Small merchants export without a problem. The export of "Bricks & Co" never finishes. The log shows W writing chunk 3, then W's writes rejected ("token 21, fence 22"), then S writing chunk 3 from its first batch, then S's writes rejected ("token 22, fence 24"), then W writing chunk 3 from its first batch again, for the whole night. No worker crashed and no worker paused. Both were busy the whole time.
Your task
Explain, with times, why the export of "Bricks & Co" loops on chunk 3, name the rule in the lesson that the worker's code breaks, and count the writes that break it. Then redesign the worker so that the 11-batch chunk completes with the same 8-second lease: say what changes in the code, show with the stop-time formula why it is safe, give the earliest and latest takeover after a real crash of the holder under the new design (with and without the lock service's leader failing at the same moment), and the one failure the new design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).