The GC Pause That Corrupted Shared Storage
The GC Pause That Corrupted Shared Storage
Your distributed media pipeline processes 4K video frames where exactly one worker instance must hold an exclusive lock on an asset to render and write the final compiled video to S3. The team uses a Redis-based distributed lock (SET resource_name my_random_token NX PX 10000) with a 10-second lease time. During high cluster memory pressure, worker node-A acquires the lock, prepares to write to storage, but enters a 14-second Stop-the-World garbage collection (or OS paging) pause. Redis's 10-second TTL expires; worker node-B detects the lock is free, acquires it, renders frame updates, and writes version 2 of the video to S3. Worker node-A wakes from its GC pause, completely oblivious that its lease expired, and blindly proceeds to write its outdated version 1 frames over node-B's data, silently corrupting the customer's video export. You are asked to redesign the mutual exclusion architecture so that out-of-order writes from expired lock holders are mathematically rejected by the storage tier.
The GC Pause That Corrupted Shared Storage
Your distributed media pipeline processes 4K video frames where exactly one worker instance must hold an exclusive lock on an asset to render and write the final compiled video to S3. The team uses a Redis-based distributed lock (SET resource_name my_random_token NX PX 10000) with a 10-second lease time. During high cluster memory pressure, worker node-A acquires the lock, prepares to write to storage, but enters a 14-second Stop-the-World garbage collection (or OS paging) pause. Redis's 10-second TTL expires; worker node-B detects the lock is free, acquires it, renders frame updates, and writes version 2 of the video to S3. Worker node-A wakes from its GC pause, completely oblivious that its lease expired, and blindly proceeds to write its outdated version 1 frames over node-B's data, silently corrupting the customer's video export. You are asked to redesign the mutual exclusion architecture so that out-of-order writes from expired lock holders are mathematically rejected by the storage tier.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).