The Photo Index That Kept Deleted Records on Disk
The Photo Index That Kept Deleted Records on Disk
This situation follows the lesson LSM-Trees & Compaction. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Pictly is a photo-sharing service. For every photo it keeps one record in a tag index: the key is the photo id (photo:01, photo:02, ...), and the value is the photo's list of tags, about 1 KB. The tag index is a log-structured merge-tree (LSM tree) store that Pictly runs itself on one EC2 instance (a virtual machine on AWS), with leveled compaction, on one EBS gp3 volume (AWS's general-purpose SSD block volume, attached over the network). When a user deletes a photo, the app sends DELETE photo:N to the tag index.
Every night a backup job opens a snapshot (a read at a fixed sequence number) and reads the whole index through it for 4 hours, then releases it.
The new requirement. Pictly's legal team now requires two things for every deleted photo. First, no read may return its record after the delete is acknowledged. Second, the record's bytes must be gone from the index's disk within 7 days of the delete. The team asks whether the store already does this, and if not, what to run.
We study the rule on a tiny copy of the index with the same settings as the tiny store that the lesson LSM-Trees & Compaction follows, then size the answer for the real index. The settings:
| Setting | Tiny index | Pictly's real index |
|---|---|---|
| MemTable capacity | 4 entries | 64 MB (RocksDB's default) |
| Level 0 compaction trigger; stop point | 2 files; not reached | 4 files; writes stop at 36 files (RocksDB's defaults) |
| Level size limit | Level 1 up to 2 files, Level 2 up to 4 | Each level about 10 times the one above |
| Compaction output file | 4 entries; one key's versions never split; the file that overlaps least in the next level is picked first, ties by key order | About 64 MB |
| Live data | 23 photos after the delete | 500 GB (an assumption) |
| Space on disk with leveled compaction | Counted in entries | About 1.1 × 500 = 550 GB (the lesson's figure for leveled compaction) |
| New tag writes | 24 writes, listed below | 2 MB/s (an assumption) |
| Write amplification | Not modelled | 10 (an assumption; the lesson's typical value) |
| Volume | Not modelled | gp3 baseline: 3,000 IOPS and 125 MiB/s, which is 131 MB/s, shared by every read and write (AWS documentation) |
| Reads | Not modelled | Served mostly from the block cache; their disk I/O is small, and we leave it out (an assumption) |
| Deleted photos in the first week of the rule | 1 (photo:04) | 40,000,000 photos of 1 KB each; a tombstone is about 40 bytes (key plus header, an assumption) |
| Backup snapshot | Opened right after write 16, held open | Opened nightly, held for 4 hours |
The tiny index's 24 writes: writes 1 to 16 are PUT photo:01 to PUT photo:16, one photo each, in order. Write 17 is DELETE photo:04. Writes 18 to 20 are PUT photo:17, PUT photo:18 and PUT photo:19. Writes 21 to 24 are PUT photo:20 to PUT photo:23. The backup's snapshot, S16, is opened right after write 16 and stays open until the backup finishes.
Synthesizing vector architecture diagram...
Your task
Using the tiny index, say when photo:04's record becomes unreadable and when its bytes leave the disk: which file holds the tombstone and which holds the old value after write 24, and which conditions a compaction must meet to drop both. Then design the procedure that makes the 7-day rule true for Pictly's real index: what it runs, how often, when relative to the backup, how many bytes it reads and writes, how long it takes on the gp3 baseline, and the one way it still fails.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Photo Index That Kept Deleted Records on Disk
This situation follows the lesson LSM-Trees & Compaction. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Pictly is a photo-sharing service. For every photo it keeps one record in a tag index: the key is the photo id (photo:01, photo:02, ...), and the value is the photo's list of tags, about 1 KB. The tag index is a log-structured merge-tree (LSM tree) store that Pictly runs itself on one EC2 instance (a virtual machine on AWS), with leveled compaction, on one EBS gp3 volume (AWS's general-purpose SSD block volume, attached over the network). When a user deletes a photo, the app sends DELETE photo:N to the tag index.
Every night a backup job opens a snapshot (a read at a fixed sequence number) and reads the whole index through it for 4 hours, then releases it.
The new requirement. Pictly's legal team now requires two things for every deleted photo. First, no read may return its record after the delete is acknowledged. Second, the record's bytes must be gone from the index's disk within 7 days of the delete. The team asks whether the store already does this, and if not, what to run.
We study the rule on a tiny copy of the index with the same settings as the tiny store that the lesson LSM-Trees & Compaction follows, then size the answer for the real index. The settings:
| Setting | Tiny index | Pictly's real index |
|---|---|---|
| MemTable capacity | 4 entries | 64 MB (RocksDB's default) |
| Level 0 compaction trigger; stop point | 2 files; not reached | 4 files; writes stop at 36 files (RocksDB's defaults) |
| Level size limit | Level 1 up to 2 files, Level 2 up to 4 | Each level about 10 times the one above |
| Compaction output file | 4 entries; one key's versions never split; the file that overlaps least in the next level is picked first, ties by key order | About 64 MB |
| Live data | 23 photos after the delete | 500 GB (an assumption) |
| Space on disk with leveled compaction | Counted in entries | About 1.1 × 500 = 550 GB (the lesson's figure for leveled compaction) |
| New tag writes | 24 writes, listed below | 2 MB/s (an assumption) |
| Write amplification | Not modelled | 10 (an assumption; the lesson's typical value) |
| Volume | Not modelled | gp3 baseline: 3,000 IOPS and 125 MiB/s, which is 131 MB/s, shared by every read and write (AWS documentation) |
| Reads | Not modelled | Served mostly from the block cache; their disk I/O is small, and we leave it out (an assumption) |
| Deleted photos in the first week of the rule | 1 (photo:04) | 40,000,000 photos of 1 KB each; a tombstone is about 40 bytes (key plus header, an assumption) |
| Backup snapshot | Opened right after write 16, held open | Opened nightly, held for 4 hours |
The tiny index's 24 writes: writes 1 to 16 are PUT photo:01 to PUT photo:16, one photo each, in order. Write 17 is DELETE photo:04. Writes 18 to 20 are PUT photo:17, PUT photo:18 and PUT photo:19. Writes 21 to 24 are PUT photo:20 to PUT photo:23. The backup's snapshot, S16, is opened right after write 16 and stays open until the backup finishes.
Synthesizing vector architecture diagram...
Your task
Using the tiny index, say when photo:04's record becomes unreadable and when its bytes leave the disk: which file holds the tombstone and which holds the old value after write 24, and which conditions a compaction must meet to drop both. Then design the procedure that makes the 7-day rule true for Pictly's real index: what it runs, how often, when relative to the backup, how many bytes it reads and writes, how long it takes on the gp3 baseline, and the one way it still fails.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).