The NTP Sync That Generated Duplicate Order IDs
The NTP Sync That Generated Duplicate Order IDs
Your global financial checkout service relies on a 64-bit distributed ID generator inspired by Twitter Snowflake: 1 bit unused, 41 bits millisecond timestamp, 10 bits machine/worker ID, and 12 bits sequence number (up to 4,096 IDs per millisecond per worker node). The system generates 180,000 unique transaction IDs per second across 32 generator nodes. At 03:00 UTC on Sunday, the datacenter network time daemon (NTP) adjusts for a 280-millisecond server clock drift on worker node 05, jumping the operating system clock backward from 1700000000280 to 1700000000000. Over the next quarter-second, node 05 generates fresh transaction IDs using the exact same timestamp millisecond values it had already issued 280ms earlier, with sequence numbers resetting to 0. Database unique constraint violations explode in the primary order database (PRIMARY KEY (order_id)), dropping 8,500 customer transactions. You are asked to redesign the ID generation service to be strictly resilient against backwards clock adjustments, leap seconds, and high-throughput sequence exhaustion.
The NTP Sync That Generated Duplicate Order IDs
Your global financial checkout service relies on a 64-bit distributed ID generator inspired by Twitter Snowflake: 1 bit unused, 41 bits millisecond timestamp, 10 bits machine/worker ID, and 12 bits sequence number (up to 4,096 IDs per millisecond per worker node). The system generates 180,000 unique transaction IDs per second across 32 generator nodes. At 03:00 UTC on Sunday, the datacenter network time daemon (NTP) adjusts for a 280-millisecond server clock drift on worker node 05, jumping the operating system clock backward from 1700000000280 to 1700000000000. Over the next quarter-second, node 05 generates fresh transaction IDs using the exact same timestamp millisecond values it had already issued 280ms earlier, with sequence numbers resetting to 0. Database unique constraint violations explode in the primary order database (PRIMARY KEY (order_id)), dropping 8,500 customer transactions. You are asked to redesign the ID generation service to be strictly resilient against backwards clock adjustments, leap seconds, and high-throughput sequence exhaustion.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).