Design S3-Like Object Storage

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is Object Storage?

You Already Use One: a Coat Check

You hand your coat to the attendant and get a ticket. Later you hand back the ticket and get exactly the same coat. You never learn which rack it hung on. You can't alter the coat while it's checked; if you want a different coat stored, you check a new one under a new ticket.

Object storage works the same way for files. You upload a file (an object) under a name made of a bucket (a named container, like acme-uploads) and a key (the name inside the bucket, like invoices/2026/09/inv-4411.pdf). Later you ask for that bucket and key and get exactly the same bytes back. Amazon S3, Google Cloud Storage and Azure Blob Storage all work this way.

Three things make it different from a file system on your laptop:

• No editing in place. An object is written whole and never modified. To change it, you upload a new one under the same key, and it replaces the old one.
• No real folders. invoices/2026/09/ is just the start of some keys. "Folders" are an illusion made by listing keys that share a prefix.
• Access over HTTP. Every operation is an HTTP request: PUT to store, GET to read, DELETE to remove.

What Makes It Hard

• Numbers. Billions of objects, sized from a few bytes to terabytes, in the same system.
• Disks die every day. At a few thousand drives, several fail every month. The promise is still that a stored file is never lost.
• Two very different workloads. Finding an object by name is a small, hot, database-like lookup. Moving its bytes is a huge, streaming, disk-and-network job. One system does both badly.
• Cost. Storage is sold by the gigabyte-month. Every extra copy we keep is money.

The Question the Whole Loop Answers

How do we store any amount of bytes so they are never lost, always findable, and cheap?

The answer gets sharper every round:

• Round 1: split the names (metadata) from the bytes (data), and keep three copies of the bytes.
• Round 2: replace most of those copies with erasure coding, and pay for it in repair traffic, tail latency and small-object tricks.
• Round 3: add geography, law (data that must not be deleted) and money (archive tiers and metering), and put a number on "never lost".

---

Round 1 · Mid-level · "File Storage for One Company's Apps"

~35 min · SDE II (L5) · 1 region, 3 AZs · ~50M objects, ~100 TB · ~450 requests/s peak · 99.9%

R1.1 Establish Design Scope

The interviewer says: "Our apps let users upload photos and documents. Design the service that stores them." Before we draw anything, we ask questions and say what each answer changes.

Out of scope for this round:

• Versions. Keeping old copies of overwritten files needs version IDs and delete markers. Nobody asked yet.
• Storage tiers. Moving old files to cheaper storage needs lifecycle rules.
• Other regions. One AWS region, with its three Availability Zones (AZs: separate data centers a few milliseconds apart).

R1.2 Functional Requirements, Derived Step by Step

We read the problem one phrase at a time and turn each phrase into an operation:

Not yet: versions, storage classes, lifecycle rules, replication to another region.

R1.3 Non-Functional Requirements: the Questions

We state each quality in words first, in the order we'd defend it. The numbers come in R1.7.

• Durability first: an acknowledged upload is never lost. A user who saw "uploaded" must never see "file not found" because a disk died. Losing data is worse than being down.
• Then availability. Uploads and downloads keep working through a node failure and a database failover.
• Latency for small objects. An avatar or a thumbnail sits on a page load. The time to the first byte matters.
• Throughput for big objects. A 10 GB export doesn't care about the first byte; it cares about megabytes per second.
• Read-after-write. After a successful PUT or DELETE, every following GET, HEAD and list sees it. No "wait a few seconds".

Why durability and availability are different. Availability is "can I read it right now?". Durability is "does it still exist?". A service can be down for an hour (bad availability) and lose nothing (perfect durability). The reverse is the disaster: it answers every request quickly and one file in a million is quietly gone. We design for durability first.

R1.4 The API

We copy S3's API shape, because every SDK and tool already speaks it.

Upload a small file, with a checksum

httpCopy
PUT /acme-uploads/avatars/u_1001.jpg HTTP/1.1
Host: storage.internal.acme.com
Authorization: AWS4-HMAC-SHA256 Credential=AKIDAPP01/20260926/us-east-1/objstore/aws4_request, SignedHeaders=host;x-amz-checksum-sha256;x-amz-content-sha256;x-amz-date, Signature=5f2c...
x-amz-date: 20260926T101500Z
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-checksum-sha256: <base64 SHA-256 of the 102,400-byte body>
Content-Type: image/jpeg
Content-Length: 102400

<102,400 bytes>

httpCopy
HTTP/1.1 200 OK
ETag: "9b2cf535f27731c974343645a3985328"
x-amz-checksum-sha256: <base64 SHA-256 of the 102,400-byte body>

The client computes a SHA-256 of the body and sends it. The service computes its own as the bytes stream in, and rejects the upload with 400 BadDigest if they differ. That catches corruption between the phone and our disks, which TCP's own 16-bit checksum can miss.

Read part of a large file

httpCopy
GET /acme-backups/orders-db/2026-09-26.dump HTTP/1.1
Range: bytes=0-8388607

httpCopy
HTTP/1.1 206 Partial Content
Content-Range: bytes 0-8388607/10737418240
Content-Length: 8388608

Ranged reads let a client download a big file in parallel pieces, and resume after a dropped connection.

List by prefix, one level at a time

httpCopy
GET /acme-uploads?list-type=2&prefix=photos/2026/&delimiter=/&max-keys=1000 HTTP/1.1

xmlCopy
<ListBucketResult>
  <Prefix>photos/2026/</Prefix>
  <KeyCount>3</KeyCount>
  <IsTruncated>true</IsTruncated>
  <CommonPrefixes><Prefix>photos/2026/08/</Prefix></CommonPrefixes>
  <CommonPrefixes><Prefix>photos/2026/09/</Prefix></CommonPrefixes>
  <Contents><Key>photos/2026/cover.jpg</Key><Size>184320</Size></Contents>
  <NextContinuationToken>cGhvdG9zLzIwMjYvY292ZXIuanBn</NextContinuationToken>
</ListBucketResult>

delimiter=/ folds every key that has another / after the prefix into one common prefix, which a UI shows as a folder. A page holds at most 1,000 entries; the client passes continuation-token to get the next page. The token is just the last key returned, encoded, so the next page starts right after it.

Multipart upload: three steps

httpCopy
POST /acme-backups/orders-db/2026-09-26.dump?uploads HTTP/1.1

xmlCopy
<InitiateMultipartUploadResult><UploadId>mpu_7a8b9c0d</UploadId></InitiateMultipartUploadResult>

httpCopy
PUT /acme-backups/orders-db/2026-09-26.dump?partNumber=1&uploadId=mpu_7a8b9c0d HTTP/1.1
Content-Length: 104857600

<100 MiB of bytes>

httpCopy
HTTP/1.1 200 OK
ETag: "b10a8db164e0754105b7a99be72e3fe5"

httpCopy
POST /acme-backups/orders-db/2026-09-26.dump?uploadId=mpu_7a8b9c0d HTTP/1.1

<CompleteMultipartUpload>
  <Part><PartNumber>1</PartNumber><ETag>"b10a8db164e0754105b7a99be72e3fe5"</ETag></Part>
  <Part><PartNumber>2</PartNumber><ETag>"1b2cf535f27731c974343645a3985328"</ETag></Part>
</CompleteMultipartUpload>

We adopt S3's limits: parts are 5 MiB to 5 GiB (the last part can be smaller), at most 10,000 parts, so the largest object is 10,000 × 5 GiB = 48.8 TiB, about 53.7 TB. S3 itself announced this 50 TB limit in December 2025, up from 5 TB. A single PUT carries at most 5 GB.

A pre-signed URL

Our API server signs a URL and hands it to the phone, which uploads straight to storage:

httpCopy
PUT /acme-uploads/avatars/u_1001.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIDAPP01%2F20260926%2Fus-east-1%2Fobjstore%2Faws4_request&X-Amz-Date=20260926T101500Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=a41f... HTTP/1.1
Host: storage.internal.acme.com
Content-Length: 102400

<102,400 bytes>

Status codes

Recap

• Seven operations: PUT, GET (with ranges), HEAD, DELETE, list by prefix, multipart upload, pre-signed URLs.
• About 50M objects and 100 TB, from KB to GB each; one region with 3 AZs.
• A successful upload is never lost, and is visible to the very next read.

Let's build it, starting with the simplest version that works.

R1.5 Design Evolution: From One Disk to Replicated Chunks

Every step follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

One server. Files on its disk. The path on disk is the bucket and key: /data/acme-uploads/avatars/u_1001.jpg. A small web server maps PUT to "write file", GET to "read file".

Synthesizing vector architecture diagram...

What's good about it: it takes an afternoon to build, and listing by prefix is a directory listing. For a prototype, this is right.

Everything that follows exists because this disk will die, and this one server will fill up.

Step 1.1: Where Does a File Live When We Have 100 Servers?

This split is the idea the whole loop is built on:

Synthesizing vector architecture diagram...

A read asks the metadata store first, then fetches the bytes from a storage node. The two are separate systems that scale separately.

Step 1.2: A Disk Died and Took Files With It

Why one copy per AZ? A failure that takes out a whole rack, or a whole AZ, then costs at most one copy of any chunk. It also means every AZ has a full copy of everything, so a front-end can always read from a node in its own AZ, which is faster and costs nothing in cross-AZ transfer.

Primitive: Distributed Consensus (Raft & Paxos) (why data copies don't need consensus here: chunks are immutable, and the metadata store decides which copies count)

Step 1.3: Uploading a 20 GB File Fails at 95%

Synthesizing vector architecture diagram...

An upload is invisible to readers until the single transaction in COMPLETING. From INITIATED or UPLOADING, an explicit abort or the 7-day deadline frees its parts.

The object's ETag for a multipart upload is not an MD5 of the whole file. S3-compatible stores compute it from the parts' MD5s and add -N for N parts, so clients should treat an ETag as an opaque version marker, and use the full-object checksum when they need to verify content.

Step 1.4: Uploads Pass Through Our API Servers and Saturate Them

Primitive: OAuth2, OIDC & Distributed Token Authentication (a pre-signed URL is a short-lived, narrowly scoped token)

Step 1.5: List Everything Under photos/2026/

Primitive: Database Sharding & Partition Keys (range vs hash partitioning)

Step 1.6: Metadata Says the File Exists, but the Bytes Are Missing

Round 1 Step Summary

R1.6 Architecture v1

Now the concepts get AWS names.

Synthesizing vector architecture diagram...

Clients talk to front-ends through an NLB, using URLs our apps signed. Front-ends look up and commit object records in Aurora, and move bytes to and from storage nodes, one copy of each chunk per AZ. The repair worker and the garbage collector run in the background and always check Aurora before acting.

Why these pieces:

• D3en instances carry large, cheap hard drives as instance store (disks physically attached to the host). A d3en.2xlarge has 8 vCPUs, 32 GiB of memory and 4 × 14 TB HDDs (56 TB). Hard drives are the right medium: our data is mostly large and read sequentially. Instance-store data is lost if the instance stops or its host fails, which is fine because every chunk has two other copies.
• Aurora PostgreSQL holds the metadata: a sorted B-tree index for listing, real transactions for multipart complete, and storage that Aurora itself keeps as six copies across three AZs. Front-ends read from the writer, not the reader, so reads always see the latest commit. The reader exists to take over in a failover.
• Fargate front-ends are stateless, so any front-end can serve any request and we can replace them freely.

The metadata schema

objects: one row per object, primary key (bucket_id, object_key)

object_chunks: primary key (object_id, seq)

chunk_replicas: primary key (chunk_id, node_id), index on node_id

The index on node_id answers the repair worker's question, "which chunks were on node b2?", with one range read.

multipart_uploads (upload_id, bucket_id, object_key, state, initiated_at) and multipart_parts (upload_id, part_number, size_bytes, etag, chunk_ids) hold uploads in progress. The garbage collector treats a part's chunks as referenced until the upload completes or is aborted.

Tracing one small PUT

1. The phone sends PUT /acme-uploads/avatars/u_1001.jpg with a pre-signed URL. The NLB passes the connection to a front-end.
2. The front-end checks the signature and expiry, and that the app's key may write acme-uploads.
3. It picks three nodes, one per AZ, and streams the 100 KB to all three as chunk c_91f2, computing SHA-256 as the bytes pass.
4. Each node writes the chunk, fsyncs, verifies the CRC32C, and replies. The front-end compares its SHA-256 with the client's header.
5. One Aurora transaction replaces the objects row and inserts the chunk rows. Commit.
6. 200 OK with the ETag. The old avatar's chunks are now orphans; the collector deletes them after 24 hours.

Tracing one GET

1. The front-end reads the object's record and chunk list from the Aurora writer (about 1–2 ms).
2. For each chunk it picks the copy in its own AZ, reads it (the node verifies CRC32C before sending), and streams it. It asks for chunk N+1 while sending chunk N, so a large file streams without gaps.
3. If the local copy is missing or fails its checksum, it reads another AZ's copy and reports the bad one to the repair worker.

Tracing a multipart upload

POST ?uploads inserts a multipart_uploads row. Each PUT ?partNumber writes chunks (three copies, as above) and upserts one multipart_parts row. POST ?uploadId checks the parts and, in one transaction, writes the objects row and the object_chunks rows (the parts' chunks in order) and marks the upload complete.

R1.7 Numbers

Targets

What we store (the interviewer's "most are under 5 MB", made concrete; this mix is our assumption)

The weighted average is 0.5 × 0.1 + 0.45 × 2 + 0.05 × 20 = 0.05 + 0.9 + 1.0 = 1.95 MB per stored object.

Traffic, reconciled with what we store. The store holds about two years of uploads, and files are rarely deleted. So writes must add up to the stored bytes over two years:

About 440 requests a second at peak, carrying a gigabit or so. This is a small system by request count. The bytes, not the requests, drive the design.

Storage fleet

Repairing a dead node. A node at 58% holds 56 TB × 0.58 ≈ 32.5 TB of chunks. Their other two copies are spread over the six nodes in the other two AZs. We launch a replacement node in the same AZ and copy the chunks into it. Its four drives can write about 1,000 MiB/s together (AWS's figure for d3en.2xlarge, sequential); we allow repair half of that, about 500 MB/s, so client traffic keeps its share: 32.5 × 10^12 ÷ 500 × 10^6 ≈ 65,000 s ≈ 18 hours. For those 18 hours, those chunks have two copies. Each source node sends only 500 ÷ 6 ≈ 83 MB/s.

Metadata

Front-ends: 6 Fargate tasks (2 vCPU, 4 GB, Graviton), two per AZ. At peak each carries about 75 requests a second and 200 Mbps.

Rough monthly cost (us-east-1 on-demand list prices; check the AWS Pricing Calculator before quoting)

Two lessons hide in this table. The biggest line after the disks is bytes to the internet, which any storage service pays, S3 included. And a 3-year commitment on the nodes (a d3en.2xlarge is about $0.454/hour on a 3-year reservation) takes the node line from $6,900 to about $3,000.

R1.8 Trade-Offs

Three copies vs RAID

RAID is cheaper, but it protects the disk, not the data. The failures we care about most take out whole servers.

Metadata in a relational store vs a key-value store

At 50M objects, one relational writer is the simplest correct answer. Round 2 has 36 times more objects, and we'll revisit this.

Streaming through front-ends vs pre-signed direct upload

Pre-signed URLs keep the bytes off our apps' servers. The bytes still pass through our storage front-ends, which we want: that's where signatures, checksums and placement happen. What we gave up: an app that needs to inspect or transform a file (virus scan, resize) must now do it after the upload, by reading the stored object.

Build vs S3

For honesty's sake: S3 would cost less here. Storage: 97,500 GB at $0.023 for the first 50 TB and $0.022 after ≈ $2,200; requests ≈ $170; the same $6,050 of internet egress: about $8,400 a month, and nobody on call. We build it because the interviewer asked, and because the design is what we'll grow. In a real company at this size, the first answer is "use S3."

R1.9 Failure Modes

R1.10 Pillar Check

Encryption at rest in detail. Each object gets its own random data key, and the storage nodes encrypt its chunks with it (AES-256). The data key is stored in the object record, encrypted ("wrapped") under a per-bucket key. The per-bucket key is itself generated and protected by AWS KMS, and front-ends cache the unwrapped bucket key for a few minutes. So KMS is called about once per bucket every few minutes, not once per object, and a stolen disk holds only ciphertext.

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Asks about sizes, mutability, listing and durability first, and says what each answer changes.
• Separates metadata from data, and explains why a formula like hash % N is the wrong way to place files.
• Replicates across AZs, and knows when a write counts as done.
• Uses multipart upload and pre-signed URLs, and says what each one costs.
• Writes bytes before the metadata commit, and cleans up orphans safely.
• Reconciles request rates with stored bytes, and sizes the fleet from the math.

Follow-up questions

1. "Why not acknowledge after two of the three copies, to make uploads faster?" Answer: we could, and Round 2 does something like it under a deadline. But then some acknowledged objects sit on two copies until the third is written, and a second failure in that window leaves one. At eight uploads a second, waiting for the third copy costs a few milliseconds on each upload; we'd rather pay that than explain a lost file.

2. "A user overwrites a photo and immediately reloads the page. Can they see the old photo?" Answer: not from us: the GET reads the committed record from the Aurora writer, which already points at the new chunks. They can see it from a cache in front of us (a browser cache or a CDN), which is why URLs for changing content carry a version or a content hash.

3. "How would you find every object that lost a copy when node b2 died?" Answer: the chunk_replicas table has an index on node_id, so it's one range read of node_id = 'b2'. Without that index, we'd scan every chunk. That's why placement is recorded, not computed.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "20 PB, Billions of Objects, and Cheap Durability"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 20 PB active, 1.8B objects · 25K writes/s and 250K reads/s peak · 99.99% · small GET P99 < 15 ms

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built file storage for one company's apps: about 50M objects and 100 TB, one region with three AZs, 99.9%, and a few hundred requests a second. We split metadata from data. Aurora PostgreSQL holds one record per object, sorted by bucket and key, so listing by prefix is a range read, and a multipart upload completes in one transaction. The bytes live as chunks of up to 64 MB on nine hard-drive storage nodes, three copies per chunk, one in each AZ, and we acknowledge only after all three are on disk. We write the bytes first and then commit the record, so the commit is the moment an object exists and every read after it sees it; a garbage collector deletes unreferenced chunks after 24 hours. Phones upload straight to our front-ends with pre-signed URLs. About $15K a month, where internet egress is almost as big as the disks. Open costs: three copies of everything, one database writer, and nothing special for tiny objects, slow nodes or silent corruption."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: records in Aurora, bytes as three copies of each chunk, and two background jobs that check Aurora before they touch any chunk.

Round 1 step summary

Open costs: 3× storage; one Aurora writer; tiny objects stored as their own chunk files; no protection against a slow node or silent corruption.

R2.1 The Scope Raise

Interviewer: "The service is now the company's storage platform: product media, database backups, analytics logs. We hold 20 petabytes in the active tier and 1.8 billion live objects, and we'll get to 100 billion objects. Peak traffic is 25,000 writes and 250,000 reads a second, about 400 gigabits a second out and 40 in. Finance says three copies of 20 PB is too expensive. Every read must see the latest write, including listings. Teams want old versions kept and old data made cheaper. Hundreds of millions of our objects are tiny, many under 4 KB. We've seen disks return wrong bytes without an error. It must survive losing an AZ at 99.99%. And small reads need to be fast: under 15 ms at P99."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

R2.2 What Breaks in the Round 1 Design

We fix them in this order: cost of copies (2.1), the repair traffic that follows (2.2), silent corruption (2.3), slow nodes (2.4), tiny objects (2.5), hot prefixes (2.6), versions and tiers (2.7).

R2.3 New Requirements and API Additions

Versioning. A bucket owner turns it on:

httpCopy
PUT /acme-media?versioning HTTP/1.1

<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>

Every PUT now creates a new version and returns its ID:

httpCopy
HTTP/1.1 200 OK
ETag: "c8b417e8ef28258b538053c9e99a8123"
x-amz-version-id: 3HL4kqtJlcpXroDTDmJ.rmSpXd3dIbrHY

A DELETE without a version ID removes nothing. It adds a delete marker: a special version that says "the latest thing here is a deletion".

httpCopy
DELETE /acme-media/covers/launch.png HTTP/1.1

httpCopy
HTTP/1.1 204 No Content
x-amz-delete-marker: true
x-amz-version-id: 7Zl1b0yJ4Uq2.dmKsp0mZt9FfZ3aQ9Xc

Now GET /acme-media/covers/launch.png returns 404 with x-amz-delete-marker: true, but the older versions are still there: GET ?versionId=3HL4kqtJ… reads one, and DELETE ?versionId=… permanently removes one version. GET /acme-media?versions&prefix=covers/ lists all versions and delete markers of the keys under a prefix, newest first per key.

Storage classes. A PUT can name one; the default is STANDARD:

httpCopy
PUT /acme-backups/orders-db/2026-09-26.dump HTTP/1.1
x-amz-storage-class: STANDARD_IA

Lifecycle rules. The bucket owner sets rules; a background engine applies them. (S3 takes this as XML; our API accepts the same fields as JSON.)

jsonCopy
{
  "Rules": [
    {
      "ID": "logs-expire",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Expiration": { "Days": 90 }
    },
    {
      "ID": "backups-to-ia",
      "Filter": { "Prefix": "orders-db/" },
      "Status": "Enabled",
      "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" } ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Full-object checksums. The client may send a CRC of the whole object (S3 supports ten algorithms, including CRC-64/NVME, its default, CRC-32C, CRC-32, SHA-1 and SHA-256). For streaming uploads, where the client doesn't know the checksum until it has sent the last byte, it sends the checksum as an HTTP trailer, after the body:

httpCopy
PUT /acme-logs/api/2026-09-26/10/part-0007.gz HTTP/1.1
Content-Encoding: aws-chunked
x-amz-decoded-content-length: 20971520
x-amz-trailer: x-amz-checksum-crc64nvme

<body in chunks> ... 0
x-amz-checksum-crc64nvme: dPQ4Zk3Hq0I=

We store it in the object record, and recompute it whenever we move the object's bytes (step 2.3). A CRC-64 of the full object has a useful property: it can be combined from the CRCs of the parts, so multipart uploads can get a true full-object checksum without re-reading the data.

R2.4 Design Evolution: Erasure Coding and Paying for It

Step 2.1: Three Copies Are Too Expensive at 20 PB

Synthesizing vector architecture diagram...

One 8+4 stripe after encoding: 12 chunks on 12 nodes, 4 per AZ, each on a different rack. A normal read takes D1 to D8 with no decoding. If AZ-b is down, the other 8 chunks are exactly enough to decode D4 to D6, with nothing to spare. The layout rotates per stripe so parity isn't always in the same AZ.

Go deeper: why no layout over 3 AZs does better than 1.5×. To survive losing an AZ, the chunks in the other two AZs must be at least k. With the same number of chunks per AZ (n ÷ 3), that means 2n ÷ 3 ≥ k, so n ÷ k ≥ 1.5. A wider code over 3 AZs (16+8, say) has the same 1.5× cost; what it buys is tolerance of more scattered drive failures (8 instead of 4), at the price of reading 16 chunks per repair. Going below 1.5× while surviving an AZ needs more AZs: 12+4 over 4 AZs (4 per AZ) is 1.33×. Round 3 uses that for cold data.

Primitive: Distributed Consensus (Raft & Paxos) (the metadata record, not the chunks, decides which bytes are the object)

Step 2.2: A Drive Died, and the Repair Floods the Network

Step 2.3: Bits Flipped on Disk, and Nobody Noticed

Step 2.4: One Slow Node Makes Reads Slow

If a chunk's recorded location points at a node that has since been replaced, the node answers "unknown chunk". The front-end then asks the placement service where the chunk lives now, and meanwhile decodes the range from other chunks. A stale location is a slow read, never a failed one.

Step 2.5: Billions of 2 KB Objects Waste Disks and Metadata

Why compaction can stall writes, and why we don't update in place. Compaction is the only thing that turns dead bytes back into free space. If deletes create dead bytes faster than compaction reclaims them, free space shrinks until new writes have nowhere to go, and the node must throttle or refuse them: a write stall caused by background work falling behind. LSM-tree databases hit the same wall in another form: when compaction lags, the number of unmerged files grows until the engine slows or stops writes to protect reads. We prevent it the same way: compaction has its own reserved I/O budget, dead space is an alarm long before disks fill, and placement sends new containers to nodes with free space. The alternative, an update-in-place layout like a B-tree, would reuse holes directly. But it would give up what makes containers cheap: every write would be a random write instead of an append, free space would fragment, and changing bytes inside an erasure-coded stripe means re-reading the stripe and recomputing its parity on every delete. Immutable, append-only containers with occasional compaction cost less overall.

Primitive: Write-Ahead Log & LSM-Trees · Drill: WAL/LSM compaction storm

Step 2.6: All Writes Hit One Prefix

Primitive: Database Sharding & Partition Keys · Loop: Design a Distributed Key-Value Store (partitioning and replication, the eventually consistent version)

Step 2.7: Old Data Should Cost Less, and People Need Old Versions

Recovering the 40,000 images is then a script: list versions under products/, and for each key whose newest version is from the bad deploy, copy the previous version on top. No bytes move.

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Clients find front-ends through DNS rather than a load balancer (R2.6 explains why). Every request asks the metadata partition that owns its key; small objects come from flash, recent data from the replicated window, older data from 8+4 stripes. Background services on the storage nodes repair, scrub, compact and collect garbage, and always check the metadata before deleting anything.

The object record in v2 (one per version; about 600 bytes with index overhead)

Tracing a PUT of a 50 MB object: when is it acknowledged?

1. The front-end checks the signature, then asks the partition owning the key to check the bucket exists and the caller may write.
2. It picks one hard-drive node per AZ and streams the body to all three as it arrives, in 8 MB pieces, never holding more than a few pieces in memory. It computes the CRC-64 as bytes pass.
3. Each node appends to an open extent, fsyncs, verifies block CRCs, and acknowledges.
4. When all three have acknowledged (or two, after the 50 ms deadline, with the third queued as urgent repair), the front-end compares its CRC-64 with the client's trailer.
5. It commits the version record at the partition leader (a Raft majority, 2 of 3 AZs). That commit is the acknowledgment point. 200 OK with the version ID.
6. On day 7 the encoder in each AZ reads its local copy, computes the 12 chunks, writes its 4 to four racks, verifies the result against the object checksum, and updates the placement map. Then each AZ deletes its full copy.

Tracing a degraded GET (AZ-b is down; the object is older than 7 days)

1. The front-end (AZ-a) reads the record from the partition leader. If the old leader was in AZ-b, a new leader was elected in seconds.
2. The stripe's layout puts D4–D6 and P2 in AZ-b. The front-end asks for D1–D3 and P1 (local), D7, D8, P3 and P4 (AZ-c): 8 chunks, no spare.
3. It decodes D4–D6 from those 8 and streams bytes. Reads that fall only in D1–D3 or D7–D8 need no decoding.

Tracing repair after a drive loss

1. Node a19 reports drive 7 failed. The placement map lists 11 TB of chunks on it.
2. Each affected stripe goes into the repair queue at its margin (normally 3 left: normal priority).
3. Rebuilders in AZ-a read 8 surviving chunks per stripe (3 local, 5 from other AZs), rebuild the lost chunk, write it to a node on another rack in AZ-a, verify the CRCs, and update the placement map. Object records don't change.

R2.6 Numbers and Cost

Reconciling traffic with storage. This is where the numbers we were handed must agree with each other.

So of 10,000 write requests a second on average, about 180 create the objects that make up the 20 PB. The rest are small, which is why request rate and bytes must be sized separately.

Reads (our assumed mix at the average of 100K/s)

Half of GETs are for small objects (our assumption): 81K a second at peak, served from flash.

Disk

Why 80% full? Repair needs free space to write rebuilt chunks, compaction needs room to write new containers before deleting old ones, and growth needs a few months of lead time.

Repair

Instance-store data doesn't survive a host failure, so a failed d3en host always means rebuilding its 265 TB. Such events are why the repair cap exists, and why critical stripes can jump the queue.

Metadata

Front-ends: 440 Gbps of client traffic at peak, in and out through the front-ends. A c7gn.8xlarge has 100 Gbps of network. 30 of them (10 per AZ) carry about 15 Gbps each at peak, and 22 Gbps each after losing an AZ.

Small-GET latency budget (P99, steps happen one after another, so they add)

The hedge isn't free in the worst case: if the local flash copy hasn't answered after 5 ms, we ask a second copy, so a hedged read's flash step is about 5 + 2 = 7 ms, and the total is 1 + 5 + 7 + 1 = 14 ms: still under 15, with 1 ms to spare. Most reads never hedge.

The line that surprised us: cross-AZ transfer. On AWS, bytes that cross between AZs cost $0.01 per GB in each direction, $0.02 per GB crossed. At our volume:

Three designs, compared (disk nodes at 3-year reserved prices, $2.73/hour per d3en.12xlarge; transfer at list price)

The finance premise, "erasure coding is cheaper", is true for disks and false for transfer. The hybrid wins only because reads concentrate on new data. Break-even with three copies forever: the hybrid's reads cost (1 − f) × \$701K, where f is the share of read bytes for data under 7 days old, so it wins while \$239K + \$210K + (1 − f) × \$701K < \$659K, that is while f > 70%. We measure f every week. If it drops, we lengthen the window.

Monthly cost of the whole platform (us-east-1 list prices)

That's about $0.031 per GB-month on 3-year pricing. Three copies forever, with the same flash, metadata and front-end lines, would be about $696K, or $0.035 per GB-month, so the hybrid saves about 10% of the total, not the 47% the disk math alone promised (31.8 PB instead of 60). Transfer is 56% of the bill.

Why no load balancer? A Network Load Balancer charges for processed bytes: one capacity unit is 1 GB per hour at $0.006, so $0.006 per GB. Our 57.8 PB a month through it would add about $347K a month. Front-ends instead publish their IP addresses in DNS: Route 53 multivalue answers return up to 8 healthy IPs per query, with health checks removing dead front-ends. S3's own endpoints also resolve to many IP addresses. Clients use the zonal name for their AZ, so traffic from a client to a front-end doesn't cross AZs either.

Compared with S3. The same 20 PB in S3 Standard: storage at $0.023, $0.022 and $0.021 per GB for the first 50 TB, the next 450 TB and the rest ≈ $421K; write-class and list requests (about 6K writes a second, since the other ~4K of the 10K are DELETEs, which S3 doesn't charge for, plus 5K lists: 11K × 2.628M s × $0.005 per 1,000) ≈ $145K; GET and HEAD (~95K a second) ≈ $100K; transfer between S3 and EC2 in the same region is free. About $665K a month. We're slightly cheaper only with a 3-year commitment, and S3 includes the people who run it.

Which numbers changed the design? The 11.21 MB average against 200 KB per write forced us to size bytes and requests separately. The 0.09% of bytes in small objects made three copies on flash affordable. And the cross-AZ price, not the disk price, decided how long data stays replicated.

R2.7 Trade-Offs

Erasure-code widths over 3 AZs

8+4 and 6+3 cost the same; 8+4 tolerates 4 scattered failures instead of 3, for reading 8 chunks per repair instead of 6. 10+4 looks cheaper and doesn't survive an AZ at all. Buying a spare chunk after an AZ loss costs about 11–14% more disk (R2.8 says when that's worth it).

Go deeper: local reconstruction codes. Azure Storage uses codes with extra local parities: for example 12 data chunks in two groups of 6, each group with its own parity, plus 2 global parities. A single lost chunk is rebuilt from its group of 6 instead of 12, which halves repair reads for the most common failure, at slightly more storage and a more complex decoder. For us, a variant with a local parity group per AZ would let most single-chunk repairs read only inside one AZ, cutting the cross-AZ repair bill.

EC vs three copies for small objects. Coding the flash containers 8+4 would need 12 flash nodes instead of 24, saving about $5.6K a month. But small GETs (about 32K a second on average, 50 KB each) would read from other AZs two thirds of the time: 32.5K × 50 KB × 2.628M s ≈ 4.3 PB a month, × 2/3 × $0.02 ≈ $57K. Three copies win by ten times, and every small read stays at one local flash read.

When to acknowledge a write

If we wrote large write-once data (backups nobody reads) straight to 8+4, we'd ack after all 12, or after 10 with the missing 2 queued as urgent repair and counted as reduced protection.

Choosing the metadata store

(Aurora's old multi-master option was a feature of Aurora MySQL 5.6-compatible version 1, which reached end of life in February 2023. It never existed for PostgreSQL.)

Our consistency stance. Every metadata read and write goes to the partition leader, confirmed by a majority, so reads are linearizable: CP for metadata. The data plane has no consistency question at all, because chunks are immutable and only become visible through a committed record. During a network split between AZs, the side with a partition's majority keeps serving it; the other side's front-ends get errors for that partition and retry through another AZ.

R2.8 Failure Modes

R2.9 Production Gotchas

1. Tiny objects stored straight into erasure coding

• Symptom: disks fill far faster than the stored bytes suggest; small reads are slow.
• Cause: a 2 KB object split 12 ways occupies 12 disk blocks (48 KB) and needs 8 reads.
• Fix: containers (step 2.5), and a check in the write path that routes anything under 128 KB to them.

2. Buffering whole uploads in front-end memory

• Symptom: front-ends run out of memory during a burst of large uploads.
• Cause: a front-end that holds each 5 GB part in memory before writing it needs 5 GB per upload; a few dozen at once exhaust a 64 GiB machine.
• Fix: stream in 8 MB pieces with backpressure (stop reading from the client while storage nodes are behind), so memory per upload is bounded to a few pieces.

3. No end-to-end checksum

• Symptom: a customer's file has one wrong byte in the middle, and every internal check passed.
• Cause: corruption happened in a front-end's memory or on a network card after the TCP checksum and before the chunk CRC was computed.
• Fix: a full-object checksum from the client (or computed at the edge), verified at commit and every time the bytes are rewritten (step 2.3).

4. Orphaned multipart uploads

• Symptom: "used space" is petabytes more than the sum of object sizes.
• Cause: clients start multipart uploads and never complete or abort them.
• Fix: a default 7-day abort rule on every bucket, a metric for bytes in incomplete uploads, and an alarm if the cleanup job stops.

5. Counting with counters instead of sets

• Symptom: containers are compacted while mostly live, or chunks are freed while a version still points at them.
• Cause: "dead bytes" or "reference count" was kept as a number that at-least-once processing incremented or decremented twice.
• Fix: keep sets (dead object IDs, referencing version IDs) and derive the numbers from them (steps 2.5 and 2.7).

R2.10 Pillar Check

Security in detail. Front-ends terminate TLS 1.3 and verify SigV4 signatures with signing keys cached per access key and day. Permissions are bucket policies stored in the metadata plane with the bucket; the partition that owns the bucket's range evaluates them, so a policy change and the next request see the same state. Front-ends cache bucket settings (policies, versioning state, lifecycle) with the setting's version number; a change pushes an invalidation, and a front-end refilling its cache only stores what it fetched if no newer version arrived meanwhile, so a late refill can't overwrite an invalidation. Inside the cluster, front-ends, metadata nodes and storage nodes authenticate each other with mutual TLS. Storage nodes sit in private subnets that accept traffic only from front-ends, rebuilders and each other.

Operations in detail.

Deploys go one AZ at a time, never touching two replicas of a partition or more than one AZ's chunks at once, starting with a canary node that bakes for an hour.

R2.11 Round 2 Rubric and Follow-Ups

What a strong senior (L6) answer adds over L5

• Reconciles request rates with stored bytes and notices when given numbers can't both be true.
• Explains erasure coding precisely: the 1.5× cost, the 4-per-AZ layout, and exactly which failures it survives, including "zero margin after an AZ".
• Does the repair math and prioritizes by remaining margin.
• Handles silent corruption end to end, and slow nodes with hedging.
• Treats small objects differently, with numbers.
• Keeps listings ordered while splitting hot partitions, and knows why prepending a hash breaks listing.
• Prices the design on AWS and finds the line (cross-AZ transfer) that changes it.

Follow-up questions

1. "How is a GET right after a PUT guaranteed to see it?" Answer: the PUT is acknowledged only after its version record commits at the partition leader, with a majority of replicas. The GET reads the record from that leader, which confirms it's still the leader before answering. So the read sees every committed write. Chunks are immutable and only reachable through a record, so there's no stale data to find. Nothing in front of the leader caches object records.

2. "Why not 10+4 over three AZs? It's cheaper." Answer: 14 chunks can't split evenly over 3 AZs; one AZ holds 5. Losing that AZ leaves 9 chunks, and we need 10. It's 1.4× and doesn't survive an AZ. Over 3 AZs, anything that survives an AZ costs at least 1.5×.

3. "Cut the bill by 30%." Answer: 30% of ≈ $628K is about $188K. Transfer is the lever: $350K. A local-parity code per AZ would keep most repairs inside an AZ; routing each GET of coded data to a front-end in the AZ holding most of its data chunks helps a little. At this spend we'd also negotiate private pricing with AWS. Storage has less room: nodes are already 79% full on 3-year prices. Honestly, the biggest saving is not building it: S3 at ≈ $665K list, with a volume discount, is about the same price and needs no team.

Interview gotchas from this round's wrong answers

---

Round 3 · Architect · "Exabytes, Public Cloud, and Compliance"

~45 min · Principal (L7) · 4 regions, 4 AZs each · ~1.12 EB, ~100B objects · ~1M requests/s average · 99.99% per region

R3.0 Where We Left Off

What the candidate says in the first 60 seconds of Round 3, and everything you need if you start here.

Round 2 in 60 seconds. "We run the company's storage platform in one region across three AZs: 20 PB active, 1.8 billion objects, 25K writes and 250K reads a second at peak, 99.99%. Metadata lives in a range-partitioned store of Raft groups, one replica per AZ; reads go to the leader, so every read and listing sees the latest write, and hot partitions split by load, with hash-spread tails whose listings merge sorted sub-ranges. New data is written as three copies, one per AZ, and after 7 days each AZ encodes its own copy into 8+4 Reed–Solomon chunks, 4 per AZ: 1.59× the data instead of 3×. That survives an AZ, with zero margin. Objects under 128 KB live packed in containers on flash with three copies; medium ones in hard-drive containers; tombstones and compaction at 30% dead. Repairs are prioritized by margin left, a scrubber checks every byte every 13 days, and reads hedge to a ninth chunk after the P95. It costs about $628K a month on AWS, and 56% of that is cross-AZ transfer. Open costs: one region, nothing that can't be deleted, cold data on the same disks as hot, and no billing."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: every request asks the metadata partition that owns its key; small objects come from flash, everything else from hard drives, where data spends its first week as copies and the rest of its life as 8+4 stripes.

Rounds 1–2 step summary

Open costs: a single region is a single blast radius; nothing stops deletion; cold data sits on disks priced for hot data; and nobody gets a bill.

R3.1 The Scope Raise

Interviewer: "We're turning the platform into a public cloud storage service. Millions of customer accounts, a bit over an exabyte, and we'll run it in several regions. Some customers need a copy of certain buckets in a second region. A bank tells us regulators require some records to be impossible to delete for seven years. Most of the data is cold and should cost almost nothing to keep, but it must come back when asked. Every byte stored and every request is billed. And a whole region can fail."

We ask back, and say what each answer changes.

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

Replication rules (per bucket; versioning must be on in both buckets, so every version keeps its identity):

jsonCopy
{
  "Role": "arn:ourcloud:iam::111122223333:role/replication",
  "Rules": [
    {
      "ID": "invoices-to-eu-west",
      "Status": "Enabled",
      "Filter": { "Prefix": "invoices/" },
      "Destination": {
        "Bucket": "arn:ourcloud:storage:::acme-docs-replica-euw",
        "StorageClass": "STANDARD_IA",
        "ReplicationTime": { "Status": "Enabled", "Minutes": 15 }
      },
      "DeleteMarkerReplication": { "Status": "Enabled" }
    }
  ]
}

HEAD on a source object then shows x-amz-replication-status: PENDING, COMPLETED or FAILED; on the copy it shows REPLICA.

Object lock. Turned on per bucket (it requires versioning), with an optional default retention:

jsonCopy
{
  "ObjectLockEnabled": "Enabled",
  "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Years": 7 } }
}

Or per object version, at upload:

httpCopy
PUT /bank-records/trades/2026/09/26/batch-0412.parquet HTTP/1.1
x-amz-object-lock-mode: COMPLIANCE
x-amz-object-lock-retain-until-date: 2033-09-27T00:00:00Z

A legal hold has no date; it's on until someone with permission turns it off:

httpCopy
PUT /bank-records/trades/2026/09/26/batch-0412.parquet?legal-hold&versionId=Qv1x9c... HTTP/1.1

<LegalHold><Status>ON</Status></LegalHold>

An archive class with asynchronous restore. Objects in ARCHIVE can't be read directly: a GET returns 403 InvalidObjectState. A restore request makes a temporary readable copy:

httpCopy
POST /acme-backups/orders-db/2025-01-31.dump?restore HTTP/1.1

<RestoreRequest>
  <Days>7</Days>
  <GlacierJobParameters><Tier>Bulk</Tier></GlacierJobParameters>
</RestoreRequest>

httpCopy
HTTP/1.1 202 Accepted

While it runs, HEAD shows x-amz-restore: ongoing-request="true"; when done, x-amz-restore: ongoing-request="false", expiry-date="Sun, 04 Oct 2026 00:00:00 GMT", and GET works until then. We offer Standard (within 12 hours) and Bulk (within 48 hours, cheaper), the same targets S3 Glacier Deep Archive publishes.

Usage reports

httpCopy
GET /v1/usage?account=111122223333&from=2026-09-01T00:00Z&to=2026-09-26T00:00Z&granularity=HOUR&group-by=bucket,storage_class HTTP/1.1

jsonCopy
{
  "rows": [
    { "hour": "2026-09-25T13:00Z", "bucket": "acme-media", "storage_class": "STANDARD",
      "byte_hours": 8.47e14, "put_requests": 1182233, "get_requests": 40114920, "bytes_out": 9.1e12 }
  ],
  "complete_through": "2026-09-25T21:00Z"
}

complete_through tells the customer which hours are final; later hours can still grow as late records arrive (step 3.5).

R3.4 Design Evolution: Regions, Law and Money

Step 3.1: Keep a Copy in Another Region

Synthesizing vector architecture diagram...

The client's PUT returns before any cross-region work starts. The replicator follows the partition's log in order; if it crashes anywhere, it resumes from its last recorded log position and repeats steps whose effects are idempotent.

Why not publish an event to a queue after each commit? That's a dual write: the commit can succeed and the publish fail (a timeout, a crash in between), and the replica silently never gets that version. Reading the committed log, or writing an "outbox" row in the same transaction as the record, makes the change and its notification one atomic fact. We read the log rather than poll an outbox table because polling adds queries to the busiest partition leaders, delivers only as fast as the poll interval, and must track what it has already seen; the log is already ordered, durable and positioned.

Why the Frankfurt booking goes wrong. The drill below asks about a seat booked in Frankfurt that still looks free in Virginia two seconds later. That's our two-way replication case: each region reads its own copy, the copy lags by seconds, and a conditional write checks only the local copy, so both regions can "win" the last seat. The fix is not faster replication but one owner per key: route writes that need a single answer to one region. And making every write synchronous to three regions to avoid it would put the slowest inter-region round trip on every write and stop all writes whenever any region is down.

Primitives: Cloud Disaster Recovery & Multi-Region Active-Active · Change Data Capture & Outbox Pattern · Drills: Replication multi-region consistency · CDC outbox dual-write drift

Step 3.2: Data Must Be Undeletable for 7 Years

Synthesizing vector architecture diagram...

A legal hold always wins. Compliance mode waits for the date plus a safety margin; governance mode lets a specially permitted caller through.

Step 3.3: Cold Data Costs Too Much on Active Disks

The same logic gave us the IA class: it stays on hard drives (it must be readable in milliseconds), but on a wider, cheaper code spread over 4 AZs, at a higher fill, with no flash or replicated window, and a retrieval fee that keeps hot data out.

Step 3.4: Durability Must Be a Number We Can Defend

Step 3.5: Metering Trillions of Requests for Billing

Primitive: Message Queues vs Event Streams · Loop: Design a Distributed Message Queue

Step 3.6: One Tenant's Traffic Hurts Others

The biggest tenant is about to grow tenfold. Their buckets' partitions split by load automatically, but the cells holding them would run out of headroom. So we plan it like a migration: raise their limits deliberately, move their largest buckets to a dedicated (or nearly empty) cell before the growth lands, and, if one bucket outgrows a cell, split it by key range across cells. A tenant that big is carved out on purpose, not discovered by an outage.

A platform-wide report across all tenants ("top 100 accounts by growth this month") never queries the live metadata partitions, which are sharded by bucket and key across thousands of partitions in about two dozen cells. It runs on the metering data and on daily inventory snapshots copied into an analytics store, where scanning everything is cheap and hurts nobody.

Primitive: Circuit Breaker, Bulkhead & Fault Tolerance · Drill: Sharding tenant hotspot · Loop: Design a Distributed Rate Limiter

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Each region is a set of independent cells behind a cell directory, plus an archive tier, replicators and a metering pipeline. Replication crosses regions only through the destination's own front-ends, so the destination applies the same checks as for any write. Only billing is global, and it's off the request path.

Tracing a replicated PUT

1. us-east front-end: limits check, signature, cell directory → cell 1.
2. Bytes to three flash or hard-drive nodes in three AZs; version record committed at the partition leader with replication_status = PENDING. 200 OK.
3. The replicator for that partition reads the new log entry, streams the bytes to eu-west, and eu-west commits the same version ID as REPLICA.
4. The replicator marks the source COMPLETED. The lag metric for that rule drops back.

Tracing a delete of a locked version

1. DELETE ?versionId=Qv1x9c… reaches the partition leader for the key.
2. The record says COMPLIANCE, retain until 2033-09-27. Leader time is earlier than retain-until plus the one-day margin.
3. 403 AccessDenied. An audit entry records who tried. Nothing else happens. The same check stops lifecycle expiration and garbage collection from touching it.

Tracing an archive restore

1. POST ?restore (Bulk, 7 days) → the partition records a restore request on the version and returns 202.
2. The restore scheduler collects requests for the same cartridge and mounts it when the batch is big enough or the 48-hour target approaches.
3. It reads the object's range from its data chunk, checks the object checksum, writes a temporary Standard copy (three copies), and updates the record: restored until a date. HEAD shows ongoing-request="false".
4. After 7 days, the temporary copy's reference is dropped and the garbage collector frees it. The archive copy never moved.

R3.6 Numbers and Cost

Data by class (1.121 EB = 100B objects × 11.21 MB)

Traffic. We scale Round 2's rates per PB: Standard ≈ 5,000 reads/s and 500 writes/s per PB (100K and 10K for 20 PB), IA about a twentieth of that for reads.

Drives and repair (our own nodes, with 24 TB drives, an assumption for current high-capacity drives)

Bigger drives mean more data per failure and longer repairs: a 24 TB drive holds about 1.7× (24 ÷ 14) the data of a 14 TB drive at the same fill, and T enters the durability model to the fourth power. The repair read per failure grows more than that, from about 88 TB in Round 2 to about 198 TB here, because IA's 12+4 code reads 12 chunks per lost chunk instead of 8. That's a reason to keep repair bandwidth growing with drive size.

Cross-region replication

Metering volume

Cost per GB-month per class (our cost with the d3en stand-in of $5.93 per raw TB-month; serving adds Round 2's flash, metadata and front-end cost per TB for Standard, about $1.8, and an assumed $0.3 for IA; archive is an assumed target, not a derived number)

Storage revenue at the reference prices would be about $8.4M a month (Standard $3.53M, IA $3.50M, archive $0.67M, replicated copies $0.70M), before requests and transfer. These costs leave out people, buildings and the backbone, so the margin is thinner than it looks; the point is that each class is priced above what it costs, which Round 2's AWS-hosted design could not do for Standard.

R3.7 Trade-Offs

Replication between regions vs erasure coding across regions

Geo-distributed coding halves the cost of surviving a region, and pays for it in latency and inter-region traffic on every operation. We offer replication per bucket, and would consider cross-region coding only for the archive.

Archive restore time vs cost. Restores are slow because batching makes them cheap: one mount serves many requests. A 1–5 minute restore would need cartridges mounted and waiting, or data on powered disks, which is exactly the cost we left behind. So we sell speed as a price: Standard within 12 hours, Bulk within 48 hours for less.

Cell size

Closing the loop. The question was: how do we store any amount of bytes so they are never lost, always findable, and cheap?

• Never lost: a stripe survives any 4 losses; repairs are prioritized by margin; a scrubber reads every byte twice a month; locked data can't be deleted; replicated buckets survive a region; and the durability model says which input to watch.
• Always findable: one ordered, strongly consistent metadata plane decides what exists; bytes become visible only through a committed record.
• Cheap: three copies only while data is hot, 8+4 or 12+4 afterwards, offline media for the archive, and per-class prices set above per-class costs.

R3.8 Failure Modes

Drills: Replication multi-region consistency · CDC outbox dual-write drift

R3.9 Runbook and Incident Response

Golden signals, per cell and per region OPS 8 · REL 6

Procedure: a drive fails

1. The node reports the drive; its chunks are marked lost; stripes enter the repair queue at their margin.
2. Check that the queue drains at the expected rate (about 198 TB of reads for a 24 TB drive; around 4 hours at full priority).
3. The drive is replaced in the next scheduled visit. Nothing is urgent unless its stripes were already below normal margin.

Procedure: a rack fails (power or top-of-rack switch)

1. All nodes in it go silent. Placement never puts two chunks of a stripe in one rack, so each stripe lost at most one chunk.
2. Wait 15 minutes: most rack events are power or network and come back. Hedged reads cover reads meanwhile.
3. If it stays down, its nodes are declared lost and their chunks repaired at normal priority. Watch the zero-margin count: it should stay at zero.

Incident flow OPS 10

Synthesizing vector architecture diagram...

The first question is always the blast radius: a tenant, a cell, an AZ or a region. Each has a prepared response.

Go deeper: operator commands. Our own regions are run through an internal admin API, not a cloud provider's CLI. Two examples:

Raise the repair budget in a cell while a failing batch is drained:

httpCopy
PUT /admin/cells/use1-cell-03/repair-budget HTTP/1.1
Authorization: Bearer <operator token, two-person approved>
Content-Type: application/json

{ "normal_gbps_per_node": 15, "critical_share": 0.5, "expires_at": "2026-09-27T06:00:00Z" }

List stripes at zero margin, oldest first:

httpCopy
GET /admin/cells/use1-cell-03/stripes?margin=0&order=oldest&limit=100 HTTP/1.1
Authorization: Bearer <operator token>

Every change carries an expiry, so a temporary override can't become permanent by accident.

Game days. Monthly: power off a rack, then a whole AZ in one cell, during business hours, and watch repair and zero-margin counts. Quarterly: restore a cell's metadata from backups into a scratch cell and compare it with the live one; restore a random sample of archive objects end to end. REL 12 · OPS 11

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Turns "survive a region" into a per-bucket choice with an RPO, a price and stated conflict behavior.
• Makes "cannot be deleted" a mechanism with no bypass, including for the provider, and names the business question it creates.
• Prices each class backward from what it can be sold for, and changes the medium when the math says so.
• Defends a durability number with a model, knows which input is the lever, and knows the model's blind spots (correlation, bugs).
• Builds metering that is idempotent end to end.
• Thinks in blast radius: tenants, partitions, cells, regions.

Follow-up questions

1. "A customer deleted a bucket's objects by mistake an hour ago. Versioning was on. What can they do? And if it was off?" Answer: with versioning, the deletes only added delete markers; removing the markers (or copying the previous versions on top) brings everything back, with no bytes moved. Without versioning, the records are gone from the API's point of view. The chunks still exist for the 7-day grace period, and we have metadata backups, but restoring a single customer's deletes is an operator action we'd offer only as support, not a promise. That's why versioning, and object lock for regulated data, are the recommendation.

2. "Why not make the archive a 1–5 minute restore, like an expedited tier?" Answer: a minutes-level restore means the data must be on media that's already mounted or powered, which is the cost the archive exists to avoid. We could offer a small expedited capacity (a few mounted drives reserved for urgent requests) at a much higher per-GB price, as long as its volume is capped so it can't turn the archive back into a disk tier.

3. "Your 11 nines: what would make you lower it?" Answer: a sustained rise in real repair time (the queue ages instead of draining), a drive batch whose AFR we can't drain fast enough, or a design change that removes margin, such as a code with zero spare after an AZ loss for a class that promises the same durability. The model tells us how much each moves the number; the monitoring tells us when it's happening.

Interview gotchas from this round's wrong answers

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design, let me ask a few scoping questions."
• When the scope is raised: "Here's what breaks in the current design, and here's the order I'll fix it in."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels