Design a File Sync and Storage Service (Google Drive)

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is File Sync?

You Already Know One: the Same Folder on Every Device

You save a spreadsheet on your laptop at the office. On the train home you open your phone, and the new version is already there. You edit a slide on the plane with no network; when you land, the laptop quietly sends the change up, and your desktop at home picks it up. It feels like one folder that lives on every device at once.

A file sync service makes that illusion. Each device keeps a real copy of the folder on its own disk. A small program on each device (the sync client) watches for local changes and sends them to the server, and it asks the server for changes made elsewhere and applies them locally. The server is the meeting point: it holds the authoritative list of files and a copy of every file's bytes.

A few words we'll use all page:

What Makes It Hard

• Files are big and change a little at a time. A 2 GB video project or a 40 MB spreadsheet gets saved dozens of times a day, and each save changes a few kilobytes. Sending the whole file every time wastes the user's upload link and our bandwidth.
• Devices go offline. A laptop sleeps for a weekend, a phone loses signal in a tunnel. When they come back, they must catch up exactly, without re-reading everything.
• Two people (or two devices) edit the same file. Both were offline, both saved, both come online. We must never silently throw one of those edits away.
• The file tree is metadata that must be exactly right. Renames, moves and deletes must never produce two files with the same name in one folder, a folder inside itself, or a file that points to bytes we deleted.

The Question the Whole Loop Answers

How do we keep every device's copy in agreement, send the fewest bytes, and never silently lose someone's edit?

The answer grows every round:

• Round 1: split the namespace (a database) from the bytes (content-addressed chunks in S3), give every namespace a change log with a cursor, make commits conditional on the base revision, and save a conflicted copy instead of overwriting.
• Round 2: cut chunks by content so an insert only changes nearby chunks, store each chunk once across all users (with proof of possession), push "something changed" nudges over WebSockets, serve immutable chunks from a CDN, and garbage-collect chunks safely.
• Round 3: add organizations: inherited permissions, customer-held keys (which limit dedup), legal holds, data residency, ransomware rollback and audit logs.

How the bytes are stored underneath (replication, erasure coding, durability math) is the subject of the S3-like object storage loop. Here we use Amazon S3 as the byte store and link there instead of re-teaching it.

---

Round 1 · Mid-level · "Sync for a 100K-User Startup"

~35 min · SDE II (L5) · 1 region, 3 AZs · 100K users, 200K devices · 1M commits/day, 35/s at peak · about 600 TB stored · changes visible within 1 min · 99.9%

R1.1 Establish Design Scope

The interviewer says: "We're a startup building a Dropbox-style app: a folder on your desktop and an app on your phone that stay in sync. Design the backend." Before drawing anything, we ask.

Out of scope for this round: sharing, cross-user dedup, real-time push, a web editor.

R1.2 Functional Requirements, Derived Step by Step

Not yet: sharing, dedup across users, real-time push.

R1.3 Non-Functional Requirements: the Questions

We name each quality in words first; the numbers come in R1.7.

• Never lose data. Once the app says "saved to the cloud", the file must survive any single server, disk or AZ failure, and no concurrent edit may silently erase it. This is the correctness bar of the round.
• Bandwidth on slow links. Many users sit behind a 10 Mbps home upload link or a phone connection. A small edit must not cost a big upload.
• Eventual agreement. After everyone stops editing and every device is online, every device must end up with exactly the same tree and the same bytes.
• Availability. If the service is down, the client keeps working offline and catches up later, so we can be a little less strict than a payment system; 99.9% is the target.

R1.4 The API

Chunks travel directly between the client and S3 using pre-signed URLs: an S3 URL that our server signs with its credentials, valid for a short time and for exactly one operation on one key. Our API servers never touch file bytes; they only decide who may upload or download what.

1. Start an upload: "here are my chunks, which do you need?"

httpCopy
POST /v1/uploads HTTP/1.1
Host: api.syncapp.example
Authorization: Bearer <device token>
Content-Type: application/json

{ "node_id": "n_8f21", "base_rev": 6, "size": 9437184,
  "chunks": ["3f9a...c1", "b07e...42", "77d1...9e"] }

The file is 9,437,184 bytes (9 MiB): two full 4 MiB chunks and a last chunk of 1 MiB. The hashes are SHA-256 in hex, shortened here. The server looks up which of this user's chunks it already stores:

jsonCopy
{ "missing": [
    { "index": 2, "hash": "77d1...9e",
      "put_url": "https://syncapp-chunks.s3.us-east-1.amazonaws.com/u/u_1001/77d1...9e?X-Amz-Signature=...",
      "required_headers": { "x-amz-checksum-sha256": "d9FxT...=" },
      "expires_at": "2026-09-28T09:15:00Z" } ] }

2. Upload each missing chunk straight to S3 with a PUT to its URL. The x-amz-checksum-sha256 header is one of the signed headers: the client must send exactly that value, and S3 computes the SHA-256 of the bytes it receives and rejects the upload if they don't match. So a chunk stored under the name 77d1...9e really does contain the bytes whose hash is 77d1...9e; a buggy or malicious client can't put the wrong bytes under a hash.

3. Commit: "make these chunks revision 7, on top of revision 6"

httpCopy
POST /v1/commits HTTP/1.1
Authorization: Bearer <device token>
Content-Type: application/json

{ "commit_id": "c_2b9d0e71", "node_id": "n_8f21", "base_rev": 6,
  "size": 9437184, "chunks": ["3f9a...c1", "b07e...42", "77d1...9e"],
  "client_mtime": "2026-09-28T09:02:11Z" }

jsonCopy
{ "node_id": "n_8f21", "rev": 7, "seq": 48214 }

commit_id is a random ID the client makes once per commit and reuses on every retry, so a retried commit can't create two revisions. client_mtime is stored for display only; it never decides anything (R1.9).

4. Catch up: "what changed since my bookmark?"

httpCopy
GET /v1/changes?cursor=48213&limit=500 HTTP/1.1
Authorization: Bearer <device token>

jsonCopy
{ "cursor": 48214, "has_more": false,
  "changes": [
    { "seq": 48214, "op": "UPSERT", "node_id": "n_8f21", "parent_id": "n_docs",
      "name": "q3-plan.docx", "rev": 7, "size": 9437184,
      "chunks": ["3f9a...c1", "b07e...42", "77d1...9e"] } ] }

5. Download a chunk

httpCopy
GET /v1/files/n_8f21/revisions/7/chunks/2 HTTP/1.1
Authorization: Bearer <device token>

The server checks that this user can read that file, then answers 302 Found with a pre-signed S3 GET URL valid for 15 minutes. The client already knows the hash from the manifest, and checks the downloaded bytes against it.

6. Restore and delete

• POST /v1/files/n_8f21/restore with { "rev": 5, "commit_id": "..." } creates revision 8 whose manifest is revision 5's. No bytes move.
• DELETE /v1/files/n_8f21?base_rev=7 moves the file to the trash (a soft delete, step 1.6).

Status codes

Recap

• Bytes go straight between client and S3 by pre-signed URL; S3 checks each chunk's SHA-256.
• Upload is three steps: "which chunks do you need?", PUT the missing ones, then a commit that names its base revision and carries a commit_id.
• Devices catch up with changes(cursor); 409 means conflict or missing chunks; 410 means resync.

R1.5 Design Evolution: From Whole Files to Chunks, Revisions and a Change Log

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

The client uploads the whole file to PUT /files/<path> on every save. The server writes it to S3 under the user's path, like u_1001/Documents/q3-plan.docx, overwriting the old object. Every minute, each device lists the user's whole tree and downloads whatever looks different.

It works for a demo. The weaknesses show up as soon as files get big, edits get frequent, and two devices disagree.

Step 1.1: A 2 GB Upload Fails at 90%

Step 1.2: Where Do Files and Folders Live?

Step 1.3: Saving a Small Edit Re-uploads the Whole File

Step 1.4: How Does My Phone Know What Changed?

Primitive: Change Data Capture and the Outbox Pattern

Step 1.5: Two Devices Edited the Same File Offline

Step 1.6: I Deleted a File and Want It Back

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

File bytes never pass through our servers: clients move them to and from S3 with pre-signed URLs. Everything the API does is a small metadata transaction in Aurora.

Tables

sqlCopy
CREATE TABLE namespaces (
  ns_id      BIGINT PRIMARY KEY,           -- one per user this round
  owner_id   BIGINT NOT NULL,
  seq        BIGINT NOT NULL DEFAULT 0,    -- last journal number used
  min_seq    BIGINT NOT NULL DEFAULT 0,    -- oldest journal row still kept
  used_bytes BIGINT NOT NULL DEFAULT 0
);

CREATE TABLE nodes (
  node_id    BIGINT PRIMARY KEY,
  ns_id      BIGINT NOT NULL REFERENCES namespaces,
  parent_id  BIGINT,                       -- NULL for the root folder
  name       TEXT   NOT NULL,
  is_folder  BOOLEAN NOT NULL,
  rev        INT    NOT NULL DEFAULT 0,    -- current revision (files)
  deleted_at TIMESTAMPTZ,
  UNIQUE (ns_id, parent_id, name)          -- two live files can't share a name (see note)
);

CREATE TABLE revisions (
  node_id      BIGINT NOT NULL,
  rev          INT    NOT NULL,
  size_bytes   BIGINT NOT NULL,
  chunks       BYTEA[] NOT NULL,           -- 32-byte SHA-256 hashes, in order
  commit_id    UUID   NOT NULL UNIQUE,     -- idempotency: one revision per commit_id
  device_id    BIGINT NOT NULL,
  client_mtime TIMESTAMPTZ,                -- display only
  committed_at TIMESTAMPTZ NOT NULL DEFAULT now(),   -- server clock
  PRIMARY KEY (node_id, rev)
);

CREATE TABLE journal (
  ns_id   BIGINT NOT NULL,
  seq     BIGINT NOT NULL,
  node_id BIGINT NOT NULL,
  op      TEXT   NOT NULL,                 -- UPSERT, MOVE, DELETE, RESTORE
  rev     INT,
  PRIMARY KEY (ns_id, seq)
);

CREATE TABLE chunks (
  owner_id   BIGINT NOT NULL,
  hash       BYTEA  NOT NULL,              -- SHA-256
  size_bytes INT    NOT NULL,
  ref_count  INT    NOT NULL,
  zero_since TIMESTAMPTZ,                  -- set when ref_count reaches 0
  PRIMARY KEY (owner_id, hash)
);

A note on the unique name constraint: deleted_at rows would block reusing a name, so in practice the constraint is a partial unique index over rows WHERE deleted_at IS NULL, and PostgreSQL treats NULL parents as distinct, so the root folder's children use the root's node ID as parent_id rather than NULL.

The commit transaction (READ COMMITTED isolation, one short transaction):

1. UPDATE namespaces SET seq = seq + 1 WHERE ns_id = $ns RETURNING seq — this takes the namespace row lock first, so all commits and sweeps in one namespace run one at a time, and gives us the journal number.
2. If a revision with this commit_id already exists, this is a retry: roll back and return the stored result.
3. SELECT rev FROM nodes WHERE node_id = $n FOR UPDATE; if rev <> base_rev, roll back and return 409 STALE_BASE.
4. Check that every chunk in the manifest has a chunks row, creating rows for the chunks this commit uploaded after a HEAD on S3 confirms each exists (S3 reads are strongly consistent after a write). A missing one → 409 MISSING_CHUNKS.
5. Insert the revision, set nodes.rev = base_rev + 1, insert the journal row with the new seq, increment ref_count for the manifest's chunks, update used_bytes.
6. Commit.

Tracing an edit on the laptop appearing on the desktop

Synthesizing vector architecture diagram...

Only 1 MiB moved in each direction. The desktop found the change on its next 30-second poll, 17 seconds after the commit here.

Tracing a conflict (numbered steps; both devices started from revision 6)

1. 18:40:05: the home desktop commits budget.xlsx with base_rev = 6. The transaction finds rev = 6, writes revision 7, journal seq 50,112.
2. 18:55:30: the laptop comes back online and runs POST /v1/uploads with base_rev = 6; the server doesn't check the base here, it only says which chunks are missing. The laptop uploads 2 chunks.
3. 18:55:33: the laptop's commit with base_rev = 6 finds rev = 7. The transaction rolls back: 409 STALE_BASE, current_rev 7.
4. The laptop renames its local copy to budget (Maria's conflicted copy 2026-09-28).xlsx and commits it as a new node (revision 1, seq 50,113). Its 2 new chunks are already uploaded, so this commit moves no bytes.
5. It pulls changes(50111): revision 7 of budget.xlsx (seq 50,112) and its own new file (seq 50,113). It downloads revision 7's changed chunks and writes budget.xlsx.
6. The desktop's next poll picks up seq 50,113 and downloads the conflicted copy. Both devices now show both files.

R1.7 Numbers

Targets

Traffic

Storage

Metadata (we assume 1,000 files and folders per user)

Propagation chain, worst case for a small edit (every step is dependent, so they add):

A 2 GB new file takes 27 minutes to upload on a 10 Mbps link, which is simply the user's link; the minute target is for edits.

Monthly cost (us-east-1 on-demand list prices, 730 hours a month, ignoring free tiers; check the AWS Pricing Calculator before quoting):

About $0.21 per user a month. Storage is 61% of the bill and data transfer out is 34%; the servers are almost free. Every later round keeps that shape.

R1.8 Trade-Offs

Fixed-size vs content-defined chunks

For a startup, fixed chunks are simple and good enough; the insert problem becomes worth solving when bandwidth and storage are big bills (Round 2).

Polling vs push

Relational vs key-value namespace

At this size the relational database wins on simplicity. Round 2's 35,000 commits a second and hundreds of terabytes of metadata change the answer.

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Splits the namespace (a transactional database) from the bytes (content-addressed chunks in object storage), and states the write order: bytes first.
• Uploads in chunks with per-chunk retry, and sends only chunks whose hash is new.
• Uses a change log with a cursor instead of rescanning, and compacts it with a resync path.
• Makes every commit conditional on a base revision, and turns conflicts into conflicted copies rather than merges or last-writer-wins.
• Keeps version history with soft deletes, and frees chunks only when nothing references them.
• Never trusts client clocks.

Follow-up questions

1. "Why not let the API servers receive the file bytes and write them to S3?" Answer: every byte would cross our servers twice (in and out), our fleet would need to scale with bandwidth, not requests, and a 10 GB upload would tie up a server for an hour. Pre-signed URLs let S3 carry the bytes while we keep control: each URL names one key, one operation, a checksum and an expiry.

2. "Two chunks from different files of the same user have the same hash. Is that a problem?" Answer: it's a feature: they're the same bytes, stored once under u/<user>/<hash>, and the ref_count counts both references. SHA-256 collisions between different contents are not a practical concern; nobody has ever produced one.

3. "A user's disk fills up locally and the client can't write a downloaded file. What happens to the cursor?" Answer: the cursor only moves after a change is applied locally. The client stops at the change it couldn't apply, reports it, and retries later; the server doesn't care how far behind a device is, up to the 30-day log.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "100M Users, 250M Devices, 1B Changes a Day"

~40 min · Senior SDE (L6) · 1 region, 3 AZs, plus a CDN · 500M users, 100M daily, 250M connected devices · 1B commits/day, 34.7K/s at peak · 2.14 EB stored after dedup · commit P95 < 100 ms · push P95 < 500 ms · 99.99%

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built sync for a 100,000-user startup: about 1 million commits a day, 35 a second at peak, 600 TB in S3. The namespace (folders, files, revisions, manifests) lives in Aurora PostgreSQL; the bytes are fixed 4 MiB chunks in S3, named by their SHA-256 hash under each user's prefix and uploaded straight from the client with pre-signed URLs that make S3 check the checksum. Bytes go first, metadata second. A commit is one transaction: take the namespace lock, check the file is still at the base revision, write the revision, a journal row with the next sequence number, and exact reference counts. A stale base gets 409 and the client saves a conflicted copy; nothing is ever merged. Devices poll changes(cursor) every 30 seconds; the journal keeps 30 days and older cursors do a full resync. Deletes are soft, history is 30 days, and a sweeper under the same namespace lock frees chunks whose count stayed at zero for 7 days. About $21.4K a month. Open costs: an insert shifts every fixed chunk, the same file is stored once per user, polling doesn't scale to push speed, and one Aurora writer won't hold the next round."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: metadata in one transactional database, bytes in S3, and a journal every device reads with its cursor.

Round 1 step summary

Open costs: fixed chunks; per-user copies of identical files; polling; one relational writer.

R2.1 The Scope Raise

Interviewer: "We're now a consumer service the size of Google Drive or Dropbox: 500 million accounts, 100 million people using it every day, 250 million devices connected at once. A billion saves a day. Edits have to show up on your other devices in half a second. Millions of people store the same installers and email attachments, and storage is our biggest bill; it has to come down. Folders can be shared with thousands of people. Last month a company shared a 10 GB video with 10,000 employees at 9 a.m. and our origin fell over."

Scope change

The "Not yet" list from R1.2 comes back: sharing, cross-user dedup and real-time push are all in scope now.

A note on the availability target: we promise 99.99% (4.4 minutes a month), not "five nines". 99.999% allows 5.26 minutes a year, less than one bad deploy.

R2.2 What Breaks in the Round 1 Design

The order we fix it in: chunking first (2.1), because it decides what a chunk is; then dedup across users (2.2), which changes chunk names; then push (2.3) and the CDN (2.4), which move the bytes and the news; then moves at scale (2.5); and last the garbage collector (2.6), which must be safe against everything before it.

R2.3 New Requirements and API Additions

Negotiation now returns three kinds of answers per chunk

httpCopy
POST /v1/uploads HTTP/1.1
Authorization: Bearer <device token>
Content-Type: application/json

{ "ns_id": "ns_7Qm", "node_id": "n_8f21", "base_rev": 14,
  "new_chunks": [
    { "hash": "e19c...5a", "size": 3981204 },
    { "hash": "c41e...07", "size": 4012331 },
    { "hash": "a0d3...b8", "size": 2210876, "from": { "node_id": "n_3c10", "rev": 2 } } ] }

The client sends only hashes that are not in its base revision's manifest: it already knows the server has those. from says "this chunk is also in another file of mine, n_3c10 revision 2"; the server checks that revision lists that hash and that the user can read it.

jsonCopy
{ "session": "us_4c1e77",
  "upload": [ { "hash": "e19c...5a",
                "put_url": "https://drive-chunks.s3.us-east-1.amazonaws.com/e19c/e19c...5a.1?X-Amz-Signature=...",
                "required_headers": { "x-amz-checksum-sha256": "4ZxQ...=" },
                "expires_at": "2026-09-28T09:15:00Z" } ],
  "prove":  [ { "hash": "c41e...07",
                "challenge": { "user": "u_1001", "hash": "c41e...07", "day": "2026-09-28",
                               "ranges": [[1048576, 1024], [2621440, 1024], [3801088, 1024]],
                               "expires_at": "2026-09-28T09:15:00Z", "sig": "..." } } ],
  "have":   [ "a0d3...b8" ] }

• upload: nobody has this chunk. .1 at the end of the key is the chunk's generation (step 2.6).
• prove: another user has it, and it's popular enough to skip the upload; prove you hold the bytes (step 2.2). Each range is [offset, length], all inside the chunk's 4,012,331 bytes.
• have: you already own it (it's in a file you can read).

Commit carries the full manifest and the proofs

jsonCopy
{ "commit_id": "c_91e0aa3f", "session": "us_4c1e77", "ns_id": "ns_7Qm", "node_id": "n_8f21", "base_rev": 14,
  "size": 41890112,
  "manifest": ["7a11...0c", "e19c...5a", "c41e...07", "a0d3...b8", "..."],
  "proofs": [ { "hash": "c41e...07", "challenge": { "...": "echoed as received" },
                "proof": "5be0...9d" } ] }

proof is HMAC-SHA256, keyed with the challenge's sig (a value the client can't compute itself), over the bytes of the three ranges joined in order. Response: { "rev": 15, "seq": 912004411 }.

The commit checks ownership of every hash. The negotiation stored an upload_sessions item (us_4c1e77: user, node, base revision, the hashes granted for upload, the hashes sent a challenge, the from references it accepted, and an expiry our servers check with their own clock). At commit, each hash in manifest must be one of: (a) in the base revision's manifest; (b) granted for upload in this session (and now verified in S3); (c) covered by a proof verified in this commit; (d) in a from revision the user can read. Any other hash gets 403 NOT_OWNED. Without this check a client could skip the negotiation and commit a manifest of hashes it merely knows.

The push channel (one WebSocket per device, wss://push.drive.example/v1/connect)

jsonCopy
{ "type": "hello", "device_id": "d_77",
  "cursors": { "ns_7Qm": 912004410, "ns_team9": 1022 } }

jsonCopy
{ "type": "nudge", "ns": "ns_7Qm", "seq": 912004411, "pull_within_ms": 0 }

A nudge says only "namespace ns_7Qm has reached seq 912,004,411". It carries no file data; the device then calls GET /v1/changes?cursor=ns_7Qm:912004410,ns_team9:1022, which returns changes for each namespace it mounts. pull_within_ms lets the server spread a big fan-out (step 2.4): the device waits a random time up to that value before pulling.

Shared folders and membership

• POST /v1/shared-folders with { "node_id": "n_proj" } turns a folder into its own namespace (a background job, step 2.5).
• POST /v1/namespaces/ns_team9/members with { "user_id": "u_2044", "role": "editor" }; the member's root gets a mount node pointing at ns_team9, and their devices add ns_team9 to their cursors.

Moves

httpCopy
POST /v1/nodes/n_proj/move HTTP/1.1
Content-Type: application/json

{ "commit_id": "c_5d2e19b0", "new_parent_id": "n_archive", "new_name": "Project X" }

200 for a move inside one namespace; 409 CYCLE if n_archive is inside n_proj; 202 Accepted with a job ID for a move into a different namespace (step 2.5).

Recap

• Clients ask only about chunks new to the file; the server answers upload, prove or have.
• Proofs are HMACs over server-chosen byte ranges; commits carry the full manifest, and every hash must be owned (base revision, this session's grants or proofs, or a readable from), else 403 NOT_OWNED.
• One WebSocket per device carries tiny nudges; the device pulls changes per namespace.
• Shared folders are namespaces with members; moves are small unless they cross namespaces.

R2.4 Design Evolution: Content-Defined Chunks, One Copy of Everything, and Push

Step 2.1: Inserting One Sentence Near the Start Re-uploads the Whole File

Step 2.2: Millions of Users Upload the Same Installer

Primitive: Bloom Filters and Counting Filters · Drill: The Web Crawler That Forgot Its History. If the filter were used and filled past its design size, its false-positive rate would climb (at 15%, 15% of truly new chunks would cost an extra index read, and a rebuild or resizing of the sub-filter would be due). And a Valkey set of all 511 billion SHA-256 hashes at about 80 bytes per member with overhead would need about 41 TB of RAM against the filter's 639 GB: exact membership at 64 times the memory, which is why an exact index belongs in DynamoDB, not in a cache.

Step 2.3: Polling 250 Million Devices

Primitives: WebSocket, SSE and Long Polling · Change Data Capture and the Outbox Pattern · Drills: The Gateway Restart That DDOSed the Chat Fleet · The Dual-Write That Broke Search Consistency. Why the stream and not a second write, or a poller: a service that writes DynamoDB and then publishes to a queue can crash between the two and silently drop the nudge; and an application thread polling a "to publish" table every 500 ms adds a query load that never stops, a 500 ms latency floor, and an index on unpublished rows, while the table's own change stream is ordered per item, kept 24 hours, and read by Lambda at no request charge.

Step 2.4: 10,000 People Download the Same 10 GB at 9 a.m.

Primitive: Distributed Cache Patterns and Eviction · Drill: The Image Service That Made Every Origin Server Sweat. Content-addressed names are the answer to its "how long until viewers stop seeing the old image?": zero, because a new version is a new URL; and the CDN-versus-bigger-origin trade-off is the egress price, request collapsing and latency argument above.

Step 2.5: Moving a Folder With 1 Million Files

Primitive: Database Isolation Levels, ACID and Concurrency Anomalies · Drill: The On-Call Doctor Anomaly. Why not run everything at strict serializable isolation instead? In a relational database, serializable transactions abort more often under contention and have to be retried, and at 35,000 commits a second we'd pay that on every commit to protect the rare move; we pay for the extra checks only on moves.

Step 2.6: Chunks Nobody References Any More

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The write path: metadata commits in DynamoDB, bytes straight into S3; nudges leave by the fast path and again through the table's own stream.

Synthesizing vector architecture diagram...

The read path: the manifest comes from the API and is never cached; chunks come from the nearest cache that has them, and at most about once from S3.

Tables (DynamoDB, on-demand)

The seq and rev sort keys are Numbers, so they sort as numbers.

The commit transaction. First the handler checks ownership: every hash in the manifest must be in the base revision, granted or proven in this upload session (upload_sessions), or in a readable from revision; any other hash is 403 NOT_OWNED. Then it reads namespaces.seq (say s), then runs one TransactWriteItems: update the namespace to s + 1 if seq = s; put journal (ns, s + 1) if it doesn't exist; update the node to rev = base_rev + 1 if rev = base_rev; put the revision; put commits (commit_id) if it doesn't exist. If the commits condition is what failed, it's a retry: read the stored result, return it, and re-send the nudge. If rev failed: 409 STALE_BASE. If seq failed: another commit in this namespace won the race; read seq again and retry. Commits to one namespace are serialized through its head item, which caps a single namespace at a few hundred commits a second: plenty for a person, and a known limit for the busiest shared folders.

Tracing an edit with one new chunk

Synthesizing vector architecture diagram...

One chunk of about 4 MB moved each way for a 40 MB file: 90% saved. The stream backstop sends the same nudge a moment later; the desktop sees seq already applied and does nothing.

Tracing the shared-file herd (numbered steps)

1. 08:59:40: the town-hall video (10 GB, 2,500 chunks after CDC) is committed to the shared folder ns_allhands (10,000 members). One journal row, seq 1,204.
2. The router reads the members from its cache: 10,000 users, about 25,000 devices, 18,000 of them online. It sends 18,000 nudges with pull_within_ms = 300000.
3. Each client waits a random 0–300 s, then pulls changes, gets the manifest and signed URLs, and fetches chunks with 4 parallel downloads. So about 60 new clients start each second instead of 18,000 at once.
4. The first requests for each chunk miss at their edge, then at the regional edge cache, then at Origin Shield, which fetches from S3 once per chunk: about 2,500 GETs on S3 in total. Everything after is served from cache.
5. On the office LAN, clients that already hold chunks serve them to their neighbors; the internet link carries far less than 100 TB.

Tracing a folder move (numbered steps)

1. The desktop moves Projects (n_proj) into Archive (n_arch), both in ns_7Qm.
2. The API walks n_arch's ancestors: n_arch → n_root. n_proj isn't among them.
3. It reads seq = 912,004,500 and runs one transaction: update n_proj.parent_id = n_arch if rev matches; ConditionCheck n_arch.parent_id = n_root; delete the old names item and put the new one if absent; bump seq to 912,004,501 if it's still 912,004,500; put the journal row MOVE; put the commits record. Seven items.
4. Other devices pull one MOVE change and rename one directory locally. None of the 1M files inside was touched.

Losing an AZ. The API, routers and gateways run in three AZs; DynamoDB, S3 and Lambda are regional. Devices on the lost AZ's gateways reconnect to the others with jittered backoff and resume tickets (R2.6 sizes the survivors).

R2.6 Numbers and Cost

Traffic

Delta sync saves nothing on a file smaller than one chunk: it is always sent whole. That's why the 40 MB average is taken over edited files, which skew large.

Storage and dedup

Metadata (we assume 1,000 files and folders per account)

If we kept every revision for 3 years instead of 30 days, revisions alone would be 1.095 trillion × 500 B = 547.5 TB; the 30-day history is what keeps all tables together near 475 TB.

DynamoDB writes per commit. Transactional writes cost 2 units per item (up to 1 KB): namespace, journal, node, revision and commit record = 5 × 2 = 10; plus the new chunk's PENDING and LIVE updates, 2 more: 12 write units per commit. At peak that's 34,722 × 12 ≈ 417,000 write units a second, about 69,000 on each of the busiest tables: above the default on-demand limit of 40,000 per table, so we raise those quotas (they're adjustable) and pre-warm the tables before launch.

Gateways, with per-AZ rounding. We assume one 4 vCPU / 16 GB Fargate task holds 50,000 idle WebSockets (a figure to load-test). 250M ÷ 50,000 = 5,000 tasks; spread over 3 AZs that's 1,666.7 per AZ, rounded up to 1,667 per AZ, 5,001 tasks. If an AZ is lost, 3,334 tasks carry 250M ÷ 3,334 ≈ 75,000 connections each for the minutes Fargate needs to add tasks. That's 16 GB ÷ 75,000 ≈ 213 KB of memory per connection, an emergency ceiling we must load-test, not a normal operating point.

Latency budgets (P95; dependent steps add, parallel steps count once, as their maximum)

The stream backstop is slower: DynamoDB Streams' own delay (AWS publishes no figure, so we measure it) plus up to 250 ms, because Lambda polls each stream shard 4 times a second, plus the same 84 ms of routing and delivery. It only decides the latency when the fast path was lost.

Availability. 99.99% of a 30.4-day month is 43,776 min × 0.0001 ≈ 4.4 minutes.

Monthly cost (us-east-1 list prices, decimal GB, 730 hours; at this size real contracts are negotiated, so read these as proportions):

About $0.23 per daily user a month, or $0.045 per account. Storage is 76% of the bill and CDN egress 16%; everything else together is under 8%. Two choices dominate the total: dedup (without it, storage would be 3.5 times larger) and tiering (S3 Standard would add about $28M). At this scale the next big lever is to stop renting object storage and build our own, which is what the S3-like object storage loop designs.

R2.7 Trade-Offs

Chunk size

CDC vs fixed chunks

Cross-user dedup vs privacy

The last row is Round 3's problem: dedup across users and keys held per customer don't mix.

Push vs poll, and whose gateway

R2.8 Failure Modes

Primitive: Circuit Breaker, Bulkhead and Fault Tolerance

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer shows

• Explains why fixed chunks break on inserts, and describes content-defined chunking correctly (a rolling hash picks cut points; Gear-based FastCDC is the modern choice, Rabin the older one), with min / average / max sizes.
• Dedups across users and immediately raises proof of possession and the existence side channel.
• Does the arithmetic before adding a Bloom filter or a cache, and knows a filter's "maybe" must be confirmed.
• Pushes nudges and pulls data; keeps a stream backstop, re-sends on duplicate retries, and handles reconnect storms.
• Serves immutable chunks from a CDN and notices the customer's own link is the real limit.
• Makes moves one update, and sees write skew in concurrent cycle checks.
• Builds a GC that can't delete a chunk a commit is about to use.

Follow-up questions

1. "Why not merge concurrent edits to text files automatically?" Answer: a three-way merge of plain text is possible when both edits touch different lines, but the sync client can't know a file is plain text in a format where merging is safe (a .csv read by a program may break on a merged line), and a wrong merge is silent data corruption. Conflicted copies are loud and safe. Real-time co-editing belongs in an editor that understands the document, not in file sync.

2. "A device has been offline for 45 days and has local edits." Answer: its cursor is below min_seq: 410. It resyncs: lists the tree at the current seq, compares hashes, downloads what changed, and commits its local edits with their original base revisions, so any that went stale become conflicted copies. Nothing it wrote offline is lost.

3. "How would you know dedup is really 3.5×?" Answer: it's measured, not assumed: logical bytes over stored bytes, per day of uploads and in total, from the quota counters and S3 Storage Lens. If it's lower, storage cost rises in proportion; the plan should show the cost at 2× and 5× too.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Enterprise: Sharing, Keys, Laws, Ransomware"

~45 min · Principal (L7) · 4 regions: a home and a standby in the US and in the EU · 20M seats in 20,000 organizations · 180M commits/day · 1 EB logical, 625 PB stored · tenant isolation · residency · point-in-time rollback · survive a region loss

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run a consumer sync service for 500 million accounts from one region: 250 million connected devices, a billion commits a day, 34,700 a second at peak. Clients cut files with FastCDC, a Gear rolling hash with 1, 4 and 8 MB bounds, so an insert changes only nearby chunks and an edit uploads about 4 MB of a 40 MB file. Chunks are stored once for everyone, 7.5 EB becomes about 2.14 EB, under a global chunk index in DynamoDB; skipping an upload needs a proof of possession, and only after a random number of users have uploaded the chunk, so the answer doesn't reveal private files. Metadata is DynamoDB: a commit is one transaction on the namespace head, the node, the revision, the journal and an idempotency record, conditional on the base revision; stale bases become conflicted copies. Devices hold WebSockets to 5,001 gateway tasks and get tiny nudges, sent right after the commit and again from the journal's change stream; they pull changes by cursor. Chunks are served from CloudFront with year-long caching and Origin Shield. Moves are one update with cycle checks as condition checks. GC is mark, doom, resurrect, second mark, sweep, with a generation in every key and a versioned bucket as a safety net. About $22.7 million a month, three quarters of it storage. Open costs: no organizations or permissions beyond folder membership, our keys only, one region, and nothing between a ransomware infection and every synced device."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: metadata transactions, bytes stored once in S3, nudges pushed, chunks pulled through the CDN.

Round 2 step summary

Open costs: permissions; customer keys; legal holds; residency; ransomware; audit.

R3.1 The Scope Raise

Interviewer: "We're launching the enterprise edition. Customers are organizations, some with 400,000 employees and millions of shared files in drives that are twelve folders deep. Their security teams want to hold their own encryption keys. Their legal teams put holds on employees' files during lawsuits. European customers' data must stay in the EU. Last quarter a customer's laptop got ransomware, encrypted 50,000 files, and our sync faithfully spread the damage to every device. And admins need an audit log of every access."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Permissions on a folder

httpCopy
PUT /v1/nodes/n_board/permissions HTTP/1.1
Authorization: Bearer <admin or manager token>
Content-Type: application/json

{ "base_acl_ver": 7, "inherit": false,
  "entries": [ { "principal": "group:board@acme.example", "role": "editor" },
               { "principal": "user:cfo@acme.example",    "role": "manager" } ],
  "link": null }

inherit: false makes n_board a restricted folder: permissions from its ancestors stop here. base_acl_ver makes the change conditional, like a commit: two admins editing the same folder's permissions can't overwrite each other.

Tenant key configuration

jsonCopy
{ "mode": "customer_key",
  "kms_key_arn":     "arn:aws:kms:eu-central-1:444455556666:key/mrk-1a2b3c4d5e6f47a8b9c0d1e2f3a4b5c6",
  "replica_key_arn": "arn:aws:kms:eu-west-1:444455556666:key/mrk-1a2b3c4d5e6f47a8b9c0d1e2f3a4b5c6" }

The key lives in the customer's AWS account (444455556666) as a multi-Region key (the mrk- prefix), with a replica in the standby region. Their key policy grants our service roles GenerateDataKey and Decrypt. The replica in eu-west-1 is a separate key resource with its own state and its own policy, so cutting us off means disabling (or removing our grant from) both keys; our revocation check reads the key state in both regions. Multi-Region keys can't be created in an external key store, so tenants who keep keys outside AWS register a separate key in each region.

Legal holds

httpCopy
POST /v1/admin/holds HTTP/1.1
Content-Type: application/json

{ "hold_id": "h_2291", "matter": "Case 2026-114",
  "scope": { "users": ["u_4410"], "shared_drives": ["ns_legal7"] } }

DELETE /v1/admin/holds/h_2291 releases it. Holds have no expiry: only a release ends one.

Point-in-time restore for a drive

jsonCopy
{ "scope": { "ns": "ns_fin" }, "to": "2026-09-28T08:55:00Z",
  "only_changes_by": { "device_id": "d_77" }, "dry_run": true }

POST /v1/admin/restores with dry_run: true returns counts (files to restore, files also changed by others since then, deletes to undo); without it, it starts the job.

An audit event

jsonCopy
{ "event_id": "ev_01J8Z6Q4M9", "tenant": "t_acme", "time": "2026-09-28T09:14:02.118Z",
  "actor": "user:bob@acme.example", "device": "d_4410", "ip": "203.0.113.24",
  "action": "download_url_issued", "node": "n_deck", "rev": 12, "decision": "allow",
  "via": "group:board@acme.example on n_board" }

Customers read theirs with GET /v1/admin/audit?from=...&to=...&actor=..., or receive daily files in their own S3 bucket.

Recap

• Roles with inheritance and restricted folders; permission changes are conditional on a version.
• Keys per tenant, optionally in the customer's own account, multi-Region within a jurisdiction.
• Holds have no expiry; restores are dry-run first; audit events are structured, per tenant, with the reason for each decision.

R3.4 Design Evolution: Organizations, Keys, Law and Attackers

Step 3.1: Who Can See This File?

Step 3.2: Customers Want Their Own Keys

Step 3.3: A Legal Hold on an Employee's Files

Step 3.4: EU Data Stays in the EU

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active · Drill: The Booking That Existed in Frankfurt but Not in Virginia. Its first question is exactly the active-active collision above (two regions accepting writes to the same record during replication lag), which one owner region per tenant prevents; its second (why not replicate synchronously to every region) is answered by the latency of a cross-region round trip on every commit, by a commit failing whenever any region is down, and by multi-Region strong consistency's three-region, no-transaction limits.

Step 3.5: Ransomware Encrypted and Synced 50,000 Files

Step 3.6: Every Access Must Be Auditable

Primitive: Change Data Capture and the Outbox Pattern

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Content never crosses the jurisdiction line; only the tenant map (IDs, regions, epochs) is global.

Synthesizing vector architecture diagram...

Each regional stack adds four things to Round 2: a permission service fed from the stream, override tables (holds and device holds), a detector, and an audit pipeline.

Tracing a shared-drive read with inherited permissions

Synthesizing vector architecture diagram...

The check never reads Finance's entries: the restricted Board folder cut inheritance, so the finance group can't see the deck even though it can see Finance.

Tracing a ransomware rollback (numbered steps)

1. 09:02:10: Tom's laptop (d_77) starts rewriting files in ns_fin, about 40 a second.
2. 09:07:10: the detector's 5-minute window closes with 12,000 rewrites (40/s × 300 s), 96% of sampled chunks random-looking and mismatched. It writes a device hold for d_77. By then, 12,000 files have been rewritten and pushed.
3. 09:07:11: d_77's next commit gets 423 Locked. The admin is paged.
4. 09:15: the admin runs a dry run to 09:02:05, limited to d_77: 12,000 files to restore, 14 also edited by others since, 0 deletes.
5. 09:16: the job commits 12,000 restores in 600 transactions of 20 and puts the pre-attack versions of the 14 shared-edit files next to their current ones as copies. Two seconds of writes at 300 transactions a second.
6. 09:16 onward: nudges go out; devices pull the rollback and fetch old chunks. The encrypted revisions remain in history for 90 days, then are purged normally.

(The detector here stopped the device after 12,000 files, not the interviewer's 50,000: a lower threshold stops it sooner but raises false positives. Either way the rollback is the same metadata operation.)

Tracing a legal hold (numbered steps)

1. Legal places h_2291 on user u_4410 (Dana) and shared drive ns_legal7. The API writes the hold and an audit event.
2. A job marks Dana's owned files' revisions and all of ns_legal7's as held_by: [h_2291] and starts an S3 Batch Operations job that sets Object Lock legal hold on the chunk versions they reference.
3. Dana deletes Contracts/ and empties the trash. Her view no longer shows it; the purge job checks the holds table itself (not only held_by, which a new revision may not carry yet), finds h_2291 covers Dana, and keeps the revisions.
4. The weekly mark finds those revisions, so their chunks stay LIVE. Even a buggy sweep couldn't destroy them: its delete would only add a delete marker, and no one, lifecycle included, can remove a version under legal hold.
5. Months later, legal releases the hold. A job clears held_by for h_2291, removes Object Lock legal holds from chunk versions no other hold references, and the normal purge then deletes what Dana deleted.

Deleting data really means a sequence of windows. When a user permanently deletes a file (with no hold), its bytes disappear from our systems only after: the trash (30 days, unless emptied), the 90-day version history, up to about 21 days of GC quarantine and marks, 7 days as a non-current S3 version, and the same in the standby region (which needs delete-marker replication turned on and its own lifecycle rule, or the standby copy would outlive the original); for US tenants, CDN edge caches can keep a chunk for up to their cache lifetime (1 day for customer-key tenants, otherwise up to a year) unless we invalidate it, although without a valid signed URL no one can fetch it; its name and metadata stay in DynamoDB point-in-time backups for up to 35 days, and audit events that mention it stay for the contract period. Anything we can restore from, we must count. When a customer deletes their KMS key (after a waiting period of 7 to 30 days), everything encrypted under it, in every copy and backup, becomes unreadable at once: that is the fastest real deletion we offer. Local copies on devices are outside all of this.

R3.6 Numbers and Cost

Tenants and data per jurisdiction (assumptions from the scope raise: 50 GB per seat including 90-day history, 1.6× dedup inside tenants, 9 commits per seat per day)

With global dedup (3.5×) the same 1 EB would be 286 PB. Tenant keys cost us 339 PB of storage.

Permission service memory. The largest tenant needs about 11.5 GB per copy (step 3.1). Across all tenants we assume about 100 folders per seat: 2B folders × 200 B = 400 GB, plus explicit entries, about 1 TB per copy, 3 TB with three copies, sharded by tenant over both jurisdictions.

KMS requests (per commit: 1 GenerateDataKey for the new chunk; Decrypts for downloads: in the US 2, since a 1-day CDN cache misses about half of 4 downloads, in the EU all 4, since EU chunks are served from S3 directly; 1 Decrypt and 1 GenerateDataKey for replication to the standby)

With Bucket Keys, KMS requests fall by up to 99%, to under $1K a month, and eu-central-1's quota stops mattering. For tenants who opt out of Bucket Keys, their share counts against eu-central-1's 20,000 a second, which we watch and raise before it matters.

Audit volume. 20M seats × 60% active × 200 events a day = 2.4B events, about 1.2 TB a day raw, about 150 GB compressed; a year is about 55 TB, under Object Lock.

Failover timing (dependent steps add):

RPO: metadata, usually about a second; chunks, 15 minutes for 99.99% of objects under Replication Time Control, and in practice recovered from the committing devices.

Monthly cost (list prices, decimal GB, 730 hours; us-east-1 and us-west-2 at us-east-1 prices; EU S3 Intelligent-Tiering at Frankfurt's prices, $0.0225 / $0.0135 / $0.005 per GB for the frequent, infrequent and archive instant tiers, which with the same 10/20/70 mix is $0.00845/GB, 14% above us-east-1; Glacier Instant Retrieval in eu-west-1 at $0.004; other EU lines at us-east-1 prices; check each region's prices in the calculator)

About $0.59 per seat a month. The EU pair costs 70% as much as the US pair for 54% as many seats, because residency takes away the CDN's cheaper egress. The two lines architects argue about: tenant-scoped dedup costs about $4.0M a month (220.5 PB extra in the US × $0.0074 ≈ $1.63M and 118.75 PB in the EU × $0.00845 ≈ $1.00M at home, plus the standby copies, 339 PB × $0.004 ≈ $1.36M), 34% of the bill, and the standby copy plus replication costs about $3.4M a month, the price of surviving a region inside each jurisdiction. Everything else (compute, KMS, audit, metadata) is under 5%.

R3.7 Trade-Offs

Dedup vs tenant keys

The consumer service keeps global dedup (Round 2); the enterprise edition pays for isolation.

Precomputed permissions vs check on read

Retention vs cost. Each day of version history keeps about one day of new chunk data: 720 TB. At home Intelligent-Tiering plus the standby copy, about $0.0118/GB a month (the US and EU blends weighted by their new data):

90 days is our choice because ransomware and accidental deletes are often found weeks later; tenants can buy longer.

Closing the loop. Round 1 answered "how do devices agree?" with a journal, a cursor and conditional commits. Round 2 kept exactly those and changed what a chunk is, who stores it and how the news travels. Round 3 added organizations, keys, laws and attackers, and still every change in the system, a restore, a rollback, a move, a revoked permission, is a commit on a base revision, recorded in a journal that every device and every index reads in order. One ordered log per namespace, conditional commits on top of immutable chunks is the invariant that survived all three rounds.

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per tenant and per region OPS 8 · REL 6

Ransomware rollback procedure SEC 10

1. Confirm the device hold is in place (the detector writes it; command 2 writes it by hand).
2. With the tenant's admin, choose the rollback time: the detector's first suspicious commit minus a few seconds.
3. Run the restore as a dry run; review files also edited by others since (they'll get copies, not overwrites).
4. Run it. Watch the rollback's transactions and the devices' pull traffic.
5. The device is cleaned or reimaged by the customer; only then does the admin clear its hold.

GC pause procedure (a suspected GC bug) REL 9

1. Set the pause flag (command 3); mark, doom and sweep jobs check it before each batch.
2. Count what the last sweep deleted; non-current versions from the last 7 days can be restored by removing their delete markers.
3. Fix, re-run a mark in dry-run mode, compare, and only then clear the flag.

Regional failover procedure REL 13

1. Confirm it's the region: the standby's canaries, several services and AWS Health agree; Route 53's health-check status (command 7) is a hint only, since its API may be unavailable when us-east-1 is impaired.
2. Check replica state and chunk replication lag (commands 4 and 5) to estimate what may be pending.
3. From the standby region, flip the ARC routing control and each affected tenant's owner with a new epoch (command 8). Confirm commits succeed in the standby.
4. Start the "pending chunks" job: it asks committing devices to re-upload missing chunks.
5. Fail back later, tenant by tenant at quiet hours, after replication catches up; run the reconciliation report.

Go deeper: CLI playbook

Plain commands an on-call engineer runs one at a time. Replace names, times and IDs with real ones.

textCopy
# 1. Alarms firing for the sync service in a region
aws cloudwatch describe-alarms --region eu-central-1 --state-value ALARM --alarm-name-prefix drive-

# 2. Put a device hold by hand
aws dynamodb put-item --region eu-central-1 --table-name device_holds --item '{"device_id":{"S":"d_77"},"tenant_id":{"S":"t_acme"},"reason":{"S":"mass encryption"},"created_at":{"S":"2026-09-28T09:07:10Z"}}'

# 3. Pause garbage collection
aws ssm put-parameter --region eu-central-1 --name /drive/gc/paused --value true --type String --overwrite

# 4. Replica status of the nodes table, seen from the standby
aws dynamodb describe-table --region eu-west-1 --table-name nodes --query "Table.Replicas"

# 5. Chunk replication latency over the last hour
aws cloudwatch get-metric-statistics --region eu-central-1 --namespace AWS/S3 --metric-name ReplicationLatency --dimensions Name=SourceBucket,Value=drive-chunks-euc1 Name=DestinationBucket,Value=drive-chunks-euw1 Name=RuleId,Value=all-chunks --start-time 2026-09-28T08:00:00Z --end-time 2026-09-28T09:00:00Z --period 300 --statistics Maximum

# 6. Legal hold on one chunk version by hand
aws s3api put-object-legal-hold --bucket drive-chunks-euc1 --key t/t_acme/9c1f/9c1f...e2.1 --legal-hold Status=ON

# 7. Health check status for a home region's endpoint (a hint only: may fail when us-east-1 is impaired)
aws route53 get-health-check-status --health-check-id 0a1b2c3d-4e5f-6789-abcd-ef0123456789

# 8. Flip a tenant to the standby with a new owner epoch, run in the standby region
aws dynamodb update-item --region eu-west-1 --table-name tenant-map --key '{"tenant_id":{"S":"t_acme"}}' --update-expression "SET owner_region = :r, owner_epoch = owner_epoch + :one" --condition-expression "owner_region = :old" --expression-attribute-values '{":r":{"S":"eu-west-1"},":one":{"N":"1"},":old":{"S":"eu-central-1"}}'

# 9. State of a tenant's key
aws kms describe-key --region eu-central-1 --key-id arn:aws:kms:eu-central-1:444455556666:key/mrk-1a2b3c4d5e6f47a8b9c0d1e2f3a4b5c6 --query "KeyMetadata.KeyState"

# 10. Drop a US tenant's cached chunks from the CDN after a key revocation (EU tenants aren't on the CDN)
aws cloudfront create-invalidation --distribution-id E2DRIVEEXAMPLE --paths "/t/t_globex/*"

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Keeps permission checks fast without rewriting subtrees, and makes revocation fail closed.
• Explains exactly what customer-held keys promise (revocable, visible) and what they cost (dedup across tenants), and why convergent encryption isn't the answer.
• Treats legal holds and device holds as override layers that every deletion or commit path consults, with an independent second layer.
• Designs residency as home and standby within a jurisdiction, one writer per tenant, and failover driven from the standby; names the split-brain window and how it's reconciled.
• Rolls back ransomware through metadata, not bytes, because chunks are immutable and history is kept.
• States deletion honestly, as a list of windows including backups, caches and devices.

Follow-up questions

1. "A customer revokes their key in the middle of the day. What exactly happens?" Answer: once the customer has disabled both the primary key and its replica in the standby region (each has its own state; disabling only one leaves the other region readable), S3 can no longer decrypt their objects: downloads fail, uploads fail. That's usually quick, but with Bucket Keys S3 may keep using a cached bucket-level key for a short while, so it isn't instant. For a US tenant, CDN copies live up to a day unless we invalidate them, which we do. Devices keep their local files. Metadata fields we encrypted with the tenant's data key become unreadable once our 5-minute cache of the data key expires. If they re-enable the key, everything works again; if they delete it after the waiting period, the data is gone for good.

2. "Why not detect ransomware on the device instead?" Answer: we should, too, but the device is the thing that's compromised, and it can be told to lie. The server sees every commit whatever the device claims, so the server-side detector and device holds are the backstop that can't be switched off by the attacker.

3. "A 400,000-seat tenant wants its own isolated stack." Answer: the design already partitions by tenant everywhere (keys, chunk prefixes, permission shards, audit prefixes), so a dedicated cell is the same stack with one tenant in its tenant map entry. It costs a fixed overhead per cell, which is why it's a paid option, not the default.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening any round: "The file tree is metadata that must be exactly right, so every change is a conditional commit on a base revision, logged in an ordered journal that devices follow with a cursor; the bytes are immutable chunks named by their hash, uploaded first and never overwritten, and when two edits collide we keep both as a conflicted copy."
• When scale arrives: "I'll cut chunks by content so edits move one chunk, store each chunk once with proofs before anyone skips an upload, push tiny nudges and let devices pull, serve chunks from a CDN because they never change, and delete a chunk only after it has been unreferenced across two marks."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels