The Time-Series Database That Froze on Flush
The Time-Series Database That Froze on Flush
Your IoT telemetry platform ingests 250,000 sensor metric points per second into an LSM-tree based storage engine (RocksDB/Cassandra). For 55 minutes out of every hour, write latency is phenomenal: incoming writes append sequentially to an append-only Write-Ahead Log (WAL) on NVMe SSDs and insert into an in-memory MemTable (skiplist), completing in under 0.8ms. However, at predictable intervals, the engine abruptly enters a 12-second "write stall" freeze where all client writes time out, ingress queues back up, and p99.9 latency spikes from 2ms to 12,000ms. Disk metrics reveal that background Leveled Compaction cannot merge Level-0 SSTables to Level-1 as fast as the MemTables are flushing to disk. When the number of uncompacted Level-0 files hits the safety threshold, the engine halts all new client writes to prevent unbounded read amplification and memory exhaustion. You are tasked with diagnosing the I/O write amplification, redesigning the LSM compaction strategy, and stabilizing p99 write latency.
The Time-Series Database That Froze on Flush
Your IoT telemetry platform ingests 250,000 sensor metric points per second into an LSM-tree based storage engine (RocksDB/Cassandra). For 55 minutes out of every hour, write latency is phenomenal: incoming writes append sequentially to an append-only Write-Ahead Log (WAL) on NVMe SSDs and insert into an in-memory MemTable (skiplist), completing in under 0.8ms. However, at predictable intervals, the engine abruptly enters a 12-second "write stall" freeze where all client writes time out, ingress queues back up, and p99.9 latency spikes from 2ms to 12,000ms. Disk metrics reveal that background Leveled Compaction cannot merge Level-0 SSTables to Level-1 as fast as the MemTables are flushing to disk. When the number of uncompacted Level-0 files hits the safety threshold, the engine halts all new client writes to prevent unbounded read amplification and memory exhaustion. You are tasked with diagnosing the I/O write amplification, redesigning the LSM compaction strategy, and stabilizing p99 write latency.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).