Design an Offline-First Mobile News Feed

This page is one interview loop in three rounds. All three rounds design the same system. Each round opens with the interviewer raising the scope, and the design from the round before has to evolve to meet it.

In this loop the phone is part of the system. It has its own database, a flaky network, a battery, and an operating system that can kill the app at any moment. Every round designs both halves: the app on the device and the backend it talks to. How the server builds each user's feed (fanout, timelines, ranking) is the subject of the news feed loop; here we link to it instead of teaching it again.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary that catches you up.

Loop Opener: What Is an Offline-First App?

You Already Use One: a Notebook You Can Read on the Plane

Think of a notebook where a friend copies the latest news for you every morning. On the plane you can't call anyone, but you can still read the notebook, and you can scribble notes in the margin. When you land, you hand the notes over and get the pages you missed.

An offline-first app works the same way. It keeps its own copy of your feed on the phone, so it opens instantly and still works in a tunnel. The network is only used to catch up: download what changed, upload what you did.

What Makes It Hard

• The network drops constantly, often in the middle of a request, and we don't know whether the server got it.
• The OS kills apps at any moment to free memory, and it decides when (and whether) the app may run in the background.
• Batteries and data plans are precious. An app that wakes the radio every minute gets noticed, uninstalled, and restricted by the OS.
• Offline actions must not be lost or applied twice. A like sent three times because of three retries must count once.
• You can't update every phone at once. Last year's app version is still out there, talking to today's server.

The Question the Whole Loop Answers

How do we make the app instant and fully usable offline, and still keep it in agreement with the server?

The answer gets sharper every round:

• Round 1: the screen reads from a database on the phone, never from the network. The network only fills that database.
• Round 2: actions made offline go into a durable queue on the phone and reach the server exactly once in effect; refreshes download only what changed, deletions included.
• Round 3: the hard part is time. Dozens of app versions, database upgrades on millions of phones, and releases we can't recall, so the server has to stay compatible and able to switch features off remotely.

---

Round 1 · Mid-level · "A Feed That Opens Instantly, Even Offline"

~35 min · SDE II (L5) · 1 region, 3 AZs · 1M DAU · ~347 refreshes/s peak · 99.9% API · ~100 MB on the device

R1.1 Establish Design Scope

The interviewer says: "Our news app shows a spinner every time it opens, and a blank screen on the subway. Design the feed so it opens instantly and works offline." We ask before we draw.

Out of scope for this round: offline actions, syncing only the changes, deletions reaching the phone, and background refresh.

R1.2 Functional Requirements, Derived Step by Step

Not yet: actions while offline, and a refresh that sends only what changed. The interviewer will bring them back.

R1.3 Non-Functional Requirements: the Questions

Numbers come in R1.7. For now, the questions and why each matters:

• Time to content. What do we measure, from when? A cold start (the app's process isn't running) includes the OS launching the process, which we don't control and which takes hundreds of milliseconds or more on a mid-range phone. So we measure two things separately: time to first frame (the platform's launch metric), and time to cached content: from the first frame to the saved feed on screen. That second number is ours to make small.
• Smooth scrolling. A display refreshing at 60 Hz draws a frame every $1000 \div 60 \approx 16.7$ ms; at 120 Hz, every $1000 \div 120 \approx 8.33$ ms. Any work on the main thread (the one thread that draws the UI and handles touches) that takes longer than that drops a frame, and the user sees a stutter.
• Storage. A phone is shared with every other app and with the user's photos. How much may we take, and what happens when the phone is nearly full?
• Data usage. Many users pay per gigabyte. How many bytes does one day of use cost them?

R1.4 The API

One endpoint for the feed, and images from a CDN. Every call carries the app's normal login token.

1. The newest page (a refresh).

httpCopy
GET /v1/feed?limit=20 HTTP/1.1
Host: api.example-news.com
Authorization: Bearer <token>
Accept-Encoding: gzip
If-None-Match: "h-8f2c91"

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: gzip
ETag: "h-8f2d07"

{
  "items": [
    {
      "post_id": "97301141913751557",
      "sort_key": 23198400000,
      "title": "City opens a new bike bridge",
      "summary": "The 400 m bridge links the two riverbanks...",
      "author": { "id": "401", "name": "Metro Desk" },
      "image": {
        "id": "7Qz2",
        "width": 1600, "height": 900,
        "url_template": "https://img.example-news.com/i/7Qz2/w{width}.{format}",
        "widths": [360, 720, 1080],
        "formats": ["avif", "webp", "jpg"]
      }
    }
  ],
  "next_cursor": "eyJ2IjoxLCJzIjoyMzE5ODM5OTg3NiwicCI6Ijk3MzAxMTQxMzk2OTY0MzIwIn0.k3Jd9w",
  "server_time": "2026-09-28T08:00:00.120Z"
}

• ETag and If-None-Match. An ETag is a version label for a response. The app sends back the last one it saw; if the newest page hasn't changed, the server answers 304 Not Modified with no body, a few hundred bytes instead of a full page.
• sort_key is the server's order for the feed (here, the post's time in milliseconds from the ID). The app sorts by it and never by its own clock.
• url_template lets the app pick the image width that fits its screen (step 1.4).

2. Older items.

httpCopy
GET /v1/feed?limit=20&cursor=eyJ2IjoxLCJzIjoyMzE5ODM5OTg3NiwicCI6Ijk3MzAxMTQxMzk2OTY0MzIwIn0.k3Jd9w HTTP/1.1
Host: api.example-news.com
Authorization: Bearer <token>

The cursor is opaque: a signed token saying "continue after this item". The app stores it and sends it back, and never builds or edits one. The server side of this cursor (why offsets repeat items, and how the signature works) is step 1.3 of the news feed loop.

Recap

• Two calls: the newest page (with an ETag) and older pages (with a cursor).
• Images come from a CDN, in sizes the app chooses.
• The saved feed must show before any of these calls finishes.

Let's build it, starting with the simplest thing that works.

R1.5 Design Evolution: From "Call the API and Render" to a Local Database

Every step below follows the same pattern: a problem, your turn to think, the answer, and what the answer costs us. The cost is always the next problem.

Step 1.0: The Baseline

The screen calls the API when it opens, parses the JSON, keeps the list in memory and draws it. Images are downloaded as each row scrolls into view.

Synthesizing vector architecture diagram...

The screen owns the data, and the data lives only as long as the screen does.

What's good about it: it is simple, and what you see is always what the server just said.

What it costs us: on a slow network the user stares at a spinner; with no network they get an error screen; and every launch downloads everything again.

Step 1.1: The App Shows a Spinner on Every Launch

Step 1.2: Scrolling Stutters

Step 1.3: Reads Block Writes During a Refresh

Primitive: Write-Ahead Log & LSM-Trees

Step 1.4: Images Reload Every Time and Eat the User's Data

Primitive: Distributed Cache Patterns & Eviction · Drill: CDN edge image delivery

Step 1.5: The Cache Fills the Phone

Round 1 Step Summary

Two costs stay open for Round 2: the phone can't act offline, and a refresh downloads the whole newest page even when one item changed.

R1.6 Architecture v1

Now the concepts get AWS names and platform names.

Synthesizing vector architecture diagram...

Follow the arrows into the screen: there is only one, from the database. The refresh worker is the only thing that talks to the feed API, and it only writes to the database. Images are the one thing the screen asks for by URL, and they come from the phone's cache first, then CloudFront.

The pieces:

• Feed API on ECS (Fargate) behind an Application Load Balancer in three AZs. It reads each user's feed from the feed service the news feed loop designs (a precomputed timeline plus a batch read of the posts), and computes the ETag.
• S3 and CloudFront for images. An upload triggers a small job that writes the three widths in each format.
• On the phone: SQLite in WAL mode, an observed query, a refresh worker on a background thread, and an image cache.

The local schema (SQL; the same on both platforms):

sqlCopy
CREATE TABLE feed_item (
    post_id      TEXT PRIMARY KEY,
    sort_key     INTEGER NOT NULL,   -- server's order; never the phone's clock
    title        TEXT NOT NULL,
    summary      TEXT,
    author_id    TEXT NOT NULL,
    author_name  TEXT NOT NULL,
    image_id     TEXT,               -- builds URLs from the template
    image_w      INTEGER,
    image_h      INTEGER,
    saved_at_ms  INTEGER NOT NULL    -- phone's clock; only for "last updated" on screen
);
CREATE INDEX feed_item_order ON feed_item (sort_key DESC, post_id);

CREATE TABLE feed_meta (             -- one row per key
    key   TEXT PRIMARY KEY,          -- 'head_etag', 'older_cursor', 'last_server_time'
    value TEXT NOT NULL
);

The screen's query is SELECT ... FROM feed_item ORDER BY sort_key DESC, post_id DESC LIMIT 200, served by the index.

Trace 1: a cold launch in a tunnel.

Synthesizing vector architecture diagram...

Time to cached content is measured from the first frame to the rows on screen, and it doesn't depend on the network at all.

Trace 2: a refresh with new items.

Synthesizing vector architecture diagram...

If nothing changed, the API answers 304 and no row is written at all.

R1.7 Numbers

Traffic (assumptions: 10 refreshes per user per day; older-page requests are few and folded into that number; peak is 3× the average)

Payloads (assumptions: a full page of 20 items is about 60 KB on the wire after gzip, including author objects and image metadata; 20% of refreshes find nothing new and get a 304; about 5 new images per refresh on average, counting the 304s, at a ~60 KB phone-sized variant)

Per user, per day: $8 \times 60 \text{ KB} + 50 \times 60 \text{ KB} \approx 0.48 + 3 = 3.5$ MB, about 106 MB a month. Without the disk cache, every refresh would reload all 20 images: $10 \times 20 \times 60 \text{ KB} = 12$ MB a day for images alone.

On the phone

Monthly cost (us-east-1 on-demand list prices; CloudFront at United States and Europe rates; rounded)

Why 3 API tasks: one task handles about 250 refreshes a second (an assumption to confirm by load test), so the 347/s peak needs 2; losing an AZ must leave 2, so we run 1 per AZ, 3 in all.

The feed service's own cost (timelines and fanout) belongs to the server loop; this table is what the offline-first app adds or resizes.

Say the headline: delivery is almost the whole bill, and the phone's cache is what keeps it small. Without the disk cache, images would be about 365 TB a month (≈ $18.6K) plus 6.1B requests (≈ $6.1K), about $24.7K instead of $8K. The cache also saves the user about 9 MB a day.

R1.8 Trade-Offs

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Makes the local database the only source for the screen, and says why an in-memory list isn't enough (the OS kills apps).
• Defines what "instant" measures: time to cached content after the first frame, not the OS's process launch.
• Keeps every disk and network call off the main thread, and knows the frame budgets (16.7 ms at 60 Hz, 8.33 ms at 120 Hz).
• Knows what WAL mode changes (readers and the writer don't block each other; still one writer) and batches writes in a transaction.
• Serves images as CDN variants with immutable URLs and caches them on disk with a size cap.
• Derives the traffic, bytes per user and the monthly cost, and sees that delivery dominates.

Follow-up questions

1. "Why not use the HTTP cache instead of a database?" Answer: it works for images, and we use it for them. For the feed, an HTTP cache stores whole responses keyed by URL. It can't answer "the newest 200 items across the last five refreshes", can't change one item in place, and its eviction isn't ours to control. The feed needs queries and transactions, which is what a database is for.

2. "The phone's clock is wrong by a day. Does anything break?" Answer: ordering doesn't: we sort by the server's sort_key. "Updated 5 minutes ago" would be wrong if computed from the phone's clock, so we compute it from server_time: the app stores the difference between the server's time and its own clock at each refresh, and uses it for display only.

3. "How would you know if time to content got worse in the field?" Answer: the app records first frame to cached content, and first frame itself, and reports percentiles per app version (Round 3 builds this telemetry). Both platforms also collect launch times (Android vitals, iOS MetricKit), which catch regressions we didn't instrument.

Interview gotchas from this round's wrong answers

---

Round 2 · Senior · "Likes and Posts Offline, Synced Safely at 50M DAU"

~40 min · Senior SDE (L6) · 1 region, 3 AZs · 50M DAU · ~35K syncs/s and ~8.7K offline actions/s at peak · 99.99% sync API · ≤ ~4 KB per sync · < 1.5% battery a day

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "We built a news app for 1M daily users that opens instantly and reads offline. The rule is: the screen reads from a SQLite database on the phone, never from the network. The screen observes a query; a refresh worker on a background thread fetches the newest page and writes it in one transaction, and the screen updates because the table changed. SQLite runs in WAL mode, so the screen's reads and the refresh's write don't wait on each other, though writes still take turns. No disk or network work on the main thread: frames are 8.33 ms at 120 Hz. Images come from CloudFront as fixed width and format variants with immutable URLs, into a 90 MB disk cache. We keep the newest 200 items. About 347 refreshes a second at peak and about $10.3K a month, almost all of it delivery. Two costs are open: the phone can't act offline, and every refresh downloads a whole page even if one item changed."

Architecture v1, compact

Synthesizing vector architecture diagram...

Round 1 in one picture: one path into the screen, from the local database.

Round 1 step summary

Open costs: no offline actions; full-page refreshes.

R2.1 The Scope Raise

Interviewer: "We're a social app now: 50 million daily users. People like, comment and post on the subway, and they expect to see it happen at once. Refreshes must send only what changed, because our users are on expensive data plans. The OS kills our app in the middle of uploads, phone clocks are all over the place, and sometimes the server has to refuse an action. And the battery team says sync may use at most 1.5% of the battery a day."

A scope raise is not the end of scoping. We ask back, and say what each answer changes.

Scope change

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

1. Delta sync. The phone sends its sync cursor (an opaque token for "how far I've read my change log") and the IDs of the posts it's showing, so their counters can be refreshed in the same call.

httpCopy
POST /v1/sync HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>
Content-Type: application/json
Accept-Encoding: gzip

{
  "cursor": "c1.ep_Hx7.450912.Qm8aZ1",
  "limit": 100,
  "visible_post_ids": ["97301141913751557", "97301141861222400"]
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: gzip

{
  "entries": [
    { "seq": 450913, "op": "ADD", "version": 3,
      "post": { "post_id": "97301143011622913", "sort_key": 23198431000, "author": { "id": "802", "name": "Priya" },
                "text": "Sunset from the ferry", "image": { "id": "m_91xQ", "width": 1080, "height": 1350 },
                "like_count": 12, "comment_count": 2, "viewer_liked": false } },
    { "seq": 450914, "op": "UPDATE", "version": 5,
      "post": { "post_id": "97301141861222400", "text": "Edited: the meetup moved to 7pm" } },
    { "seq": 450915, "op": "DELETE", "version": 8, "post_id": "97301141396964320" }
  ],
  "counters": { "97301141913751557": { "like_count": 131, "comment_count": 9, "status": "ACTIVE" } },
  "cursor": "c1.ep_Hx7.450915.pT3xLw",
  "has_more": false,
  "server_time": "2026-09-28T08:00:00.120Z"
}

A DELETE entry is a tombstone: a record that says "this was deleted", so a phone can remove its copy. If the phone's cursor is older than the log still holds, the answer is 410 Gone with "error": "CURSOR_EXPIRED", and the phone rebuilds its feed from the head:

httpCopy
GET /v1/sync/head?limit=20 HTTP/1.1
Authorization: Bearer <token>

It returns the newest 20 posts (about 60 KB) and a fresh cursor; older posts are fetched as the user scrolls. The shape is the same as above.

2. Batch mutations. Queued actions go up in batches, each with an idempotency key created on the phone (client_mutation_id, a UUIDv7):

httpCopy
POST /v1/mutations:batch HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>
Content-Type: application/json

{
  "mutations": [
    { "client_mutation_id": "01927e3a-7f12-7000-8000-123456789abc", "type": "LIKE",
      "post_id": "97301141913751557" },
    { "client_mutation_id": "01927e3a-7f13-7000-8000-123456789abd", "type": "COMMENT",
      "post_id": "97301141913751557", "payload": { "text": "Great shot!" } }
  ]
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{
  "results": [
    { "client_mutation_id": "01927e3a-7f12-7000-8000-123456789abc", "status": "APPLIED" },
    { "client_mutation_id": "01927e3a-7f13-7000-8000-123456789abd", "status": "APPLIED",
      "comment_id": "97301143552201984" }
  ],
  "server_time": "2026-09-28T08:00:01.015Z"
}

The user is always the one in the token; the payload never says who is acting. The batch is processed in order, and each mutation gets its own result:

R2.4 Design Evolution: Offline Writes, Delta Sync and Battery

Step 2.1: Liking a Post With No Signal

Primitive: Change Data Capture & the Outbox Pattern

Step 2.2: The App Was Killed While Sending, and the Retry Liked Twice

Primitive: Distributed Unique ID Generators

Step 2.3: Refreshing Downloads the Whole Feed

Primitive: Change Data Capture & the Outbox Pattern · Primitive: Message Queues vs Event Streams · Primitive: Distributed Locks & Leases · Drill: CDC outbox dual-write drift

Step 2.4: A Deleted Post Still Shows on Phones

Step 2.5: The Server Rejected an Action

Primitive: Two-Phase Commit & Saga Orchestration

Step 2.6: Background Sync Drains Batteries

Loop: Design a Notification System

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

Two loops meet here. Writes go phone → outbox → mutation API → social table, and every consequence (fanout, the followers' logs, counters, notifications) flows from that table's change stream. Reads go log → sync API → phone, where they land in the database the screen observes. Nothing on the server writes the log except the log writers, and nothing on the phone writes the screen except the database.

Local schema (SQLite)

sqlCopy
CREATE TABLE post (                      -- server-confirmed state only
    post_id       TEXT PRIMARY KEY,
    sort_key      INTEGER NOT NULL,
    version       INTEGER NOT NULL,       -- apply an entry only if its version is higher
    author_id     TEXT NOT NULL,
    author_name   TEXT NOT NULL,
    text          TEXT,
    image_id      TEXT,
    like_count    INTEGER NOT NULL DEFAULT 0,
    comment_count INTEGER NOT NULL DEFAULT 0,
    viewer_liked  INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX post_order ON post (sort_key DESC, post_id);

CREATE TABLE pending_mutation (          -- the outbox
    local_seq          INTEGER PRIMARY KEY AUTOINCREMENT,   -- send order
    client_mutation_id TEXT NOT NULL UNIQUE,                -- UUIDv7, the idempotency key
    type               TEXT NOT NULL,       -- LIKE, UNLIKE, COMMENT, CREATE_POST, EDIT_POST
    target_id          TEXT NOT NULL,       -- post ID, or a local ID for an offline post
    depends_on         TEXT,                -- client_mutation_id this one needs first
    payload_json       TEXT NOT NULL,
    payload_version    INTEGER NOT NULL,    -- lets a newer app read an older row (Round 3)
    state              TEXT NOT NULL CHECK (state IN ('QUEUED', 'IN_FLIGHT')),
    claim              TEXT,                -- run_id of the drainer that claimed it
    attempts           INTEGER NOT NULL DEFAULT 0,
    next_attempt_at_ms INTEGER NOT NULL DEFAULT 0     -- local scheduling only
);

CREATE TABLE quarantined_mutation (
    client_mutation_id TEXT PRIMARY KEY,
    type               TEXT NOT NULL,
    payload_json       TEXT NOT NULL,
    error_code         TEXT NOT NULL,
    user_notified      INTEGER NOT NULL DEFAULT 0
);

CREATE TABLE tombstone (post_id TEXT PRIMARY KEY, version INTEGER NOT NULL, received_at_ms INTEGER NOT NULL);

CREATE TABLE sync_state (key TEXT PRIMARY KEY, value TEXT NOT NULL);   -- 'cursor', 'server_clock_offset_ms'

Server tables (DynamoDB)

seq is a Number, so it sorts numerically (as a string, "10" would sort before "9"). Counters (like_count, comment_count) are summed from the social table's change stream into ElastiCache and written back to the post item every few seconds, instead of being incremented on the post item by every like, which would turn a viral post into one hot item. DynamoDB Streams orders records per item only: each item's records arrive in order on one shard, but the likes of one post are different items, and when DynamoDB splits a hot partition, a viral post's likes can spread across shards. So the aggregator stores, on the post item, the sequence number of the last record it applied per (post, stream shard), and applies a batch from a shard only if the batch's first record is newer than that shard's stored number, so a replayed batch after a crash isn't counted twice.

Trace 1: a like made offline, synced later.

Synthesizing vector architecture diagram...

If the response is lost, the row stays; the retry's conditional put fails and the answer is DUPLICATE, which removes the row the same way.

Trace 2: a rejected comment, rolled back.

Synthesizing vector architecture diagram...

Trace 3: a delta sync with a tombstone.

Synthesizing vector architecture diagram...

One local transaction applies the entries and moves the cursor, so a crash leaves either the old state and old cursor, or the new state and new cursor.

R2.6 Numbers and Cost

Traffic (the interviewer's figures, derived; peak is 3× the average)

Bytes on the wire

The 4 KB is our planning figure for an average sync, the NFR's ceiling: about 2.3 new log entries (below), each hydrated to about 1 KB, plus counters for the posts on screen and headers, after gzip. Many syncs return nothing and are far smaller.

Synthesizing vector architecture diagram...

Syncing only what changed cuts peak egress by 15 times, and it's the user's data plan that saves the most.

The change log (assumptions: 0.2 posts per user per day; 200 followers per post on average; 2% of posts later edited or deleted; every action also appends one SELF entry to its own author's log, for their other devices)

On the phone

Cellular data per user per day: $20 \times 4 \text{ KB} = 80$ KB of sync, plus about 1.5 MB of images (assumption: ~20 new images viewed, mostly thumbnails on cellular and feed-width on Wi-Fi, plus ~20% prefetched and never viewed, plus avatars). About 1.6 MB a day, 49 MB a month.

Fleet sizing (per-task rates are assumptions to confirm by load test; every fleet must carry the peak with one AZ lost, and we round per AZ)

Quotas. The log's peak is about 79.5K entries/s × 2 writes (entry + META) ≈ 159K write units a second, far above DynamoDB's default of 40,000 per table (and 80,000 per account for provisioned tables), so we request the increases before launch. The Kinesis stream needs 80 shards at peak (1,000 records/s each); we run 100 to leave headroom for skew between shards. Background pushes, at a few per device per day (say 150M a day, about 5.2K/s at peak), sit under FCM's default 600,000 messages a minute (10K/s) per project, but close enough to alarm on.

Monthly cost (us-east-1 list prices; CloudFront at United States and Europe rates; rounded)

Three notes on that table:

• Why provisioned capacity for the log. At on-demand prices its writes would be 69.6\text{B} \times 2 \times \0.625 \text{ per million} \approx $87\text{K}$. The load is steady and predictable (it follows posting), which is what provisioned capacity with auto scaling is for, and it cuts the line by about 60%.
• Why the API goes straight to the ALB, not through CloudFront. CloudFront's data price is a little lower, but every API call would also pay its request fee: 1B syncs + 100M batches a day ≈ 33.4B requests a month ≈ $33K, more than all the API's bytes cost.
• What the deltas saved. Full pages would be 1.82 PB a month of API egress, about $95K instead of $9.6K, and about 1.1 MB a day more on every user's data plan (20 × 60 KB instead of 20 × 4 KB).

The feed service itself (timelines, fanout, ranking) and media storage are priced in the server loop; this is what the offline-first sync adds. About $0.004 per daily user per month, and images are more than half of it.

R2.7 Trade-Offs

R2.8 Failure Modes

Drill: WebSocket reconnect thundering herd (the fleet-level storm above, and the SSE vs WebSocket row in R2.7)

R2.9 Production Gotchas

R2.10 Pillar Check

Alarms and first actions

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Writes offline actions to a local outbox in the same transaction as the optimistic change, and keeps server state separate from pending changes.
• Makes retries safe with keys born on the phone and guards on the server that don't expire, and checks who the caller is and what they may touch.
• Rejects timestamp cursors for the right reason (late commits, not just skewed clocks), and designs how the sequence is assigned: one fenced writer, dense numbers, a high-water mark that survives expiry.
• Carries deletes as tombstones with versions, bounds the window, and detects a phone that fell out of it without a clock.
• Separates transient from permanent errors, and designs the rollback the user sees.
• Lives inside the OS's background rules, and checks the battery budget with rough numbers.
• Derives traffic, bytes and cost, and finds that the log and image delivery dominate.

Follow-up questions

1. "A user writes 10 comments offline, then reconnects on a slow network. What do they see?" Answer: all 10 show at once with a "Sending" badge, in the order written. The engine sends them in batches in local_seq order; each result removes its row and the server's version of the comment arrives through the log, replacing the local one by client_mutation_id in one transaction, so nothing jumps. A slow network only means the badges clear slowly.

2. "The same user has a phone and a tablet. They like a post on the phone and unlike it on the tablet while both are offline." Answer: each device's queue is sent when it reconnects, and the server applies them in the order they arrive: last write wins for that user's like, which is acceptable for a like (only the user writes it). Each applied action appends a SELF entry to the user's own log, so both devices converge on the final state at their next sync. For something that matters more, like an edit, EDIT_POST carries the version it was based on, and a stale one gets VERSION_CONFLICT and goes to quarantine with the user's text.

3. "Why not use CRDTs so phones never conflict?" Answer: a CRDT (a data type whose copies merge the same way whatever order updates arrive in) earns its complexity when many people edit the same object at once, like a shared document. Here, likes are per-user state, comments are append-only, and posts have one author. A server order with version checks is simpler and enough. The file sync loop discusses true multi-writer conflicts.

Interview gotchas from this round

---

Round 3 · Architect · "Many App Versions, Platforms and a Risky Rollout"

~45 min · Principal (L7) · 1 region, 3 AZs (global placement is the server loop's Round 3) · 200M installs, 80M DAU · ~55.6K syncs/s peak · 30 app versions · crash-free users ≥ 99.5% per version · a bad feature off in minutes

R3.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 3. If you're starting here, it's everything you need from Rounds 1 and 2.

Round 2 in 60 seconds. "We run an offline-first social feed for 50 million daily users: a billion syncs a day, about 35,000 a second at peak, and 250 million actions, many made offline. On the phone, SQLite is the only source for the screen. A tap writes an outbox row in one transaction, with a UUIDv7 key born with the action; the screen shows server state plus pending actions, so a sync can't erase a pending like. One claimed drainer sends batches; rows leave the outbox only when the server says applied or duplicate. The server's guards never expire: the like item itself, or a dedup item written in the same transaction as a comment. Permanent errors go to a quarantine and the screen rolls back with a notice. Reads are delta syncs: each user has a change log in DynamoDB with dense sequence numbers, written by one fenced writer per user from the fanout stream, with a high-water mark that never expires. The phone advances its cursor in the same transaction that applies entries. Deletes are versioned tombstones kept 14 days; a phone away longer gets 410 and rebuilds from the head, keeping its outbox. Background work is scheduled by WorkManager and BGTaskScheduler; prefetch depends on network and battery. About 4 KB a sync, 1.12 Gbps of delta egress at peak, about $207K a month, over half of it images. Open costs: one protocol version, a schema we can't change safely, releases we can't pull back, plaintext data on lost phones, and no idea what happens on real devices."

Architecture v2, compact

Synthesizing vector architecture diagram...

Round 2 in one picture: actions go up through the outbox, changes come down through a per-user log, and the phone's database is where they meet.

Steps so far

Open costs: one protocol version; destructive local upgrades; no way to stop a bad release; plaintext on lost phones; no telemetry.

R3.1 The Scope Raise

Interviewer: "We're on iOS, Android and the web now, with 200 million installs. We ship every week, but a third of our users haven't updated in months, and we count 30 app versions still in use. Next quarter's release changes the local database a lot, and people will have offline actions queued when they upgrade. Last month a bad release crashed on some Android phones for four days while we waited for the store. Families share tablets, phones get lost, and we learn about problems from one-star reviews."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

1. Every request says who is talking.

httpCopy
POST /v1/sync HTTP/1.1
Host: api.example-social.com
Authorization: Bearer <token>
X-App-Version: 5.14.2
X-Platform: android
X-Sync-Protocol: 2
Content-Type: application/json

{ "cursor": "c1.ep_Hx7.450912.Qm8aZ1", "capabilities": ["op.edit", "media.avif", "post.poll"] }

The response carries a client policy and the current flags version:

jsonCopy
{
  "entries": [],
  "cursor": "c1.ep_Hx7.450912.Qm8aZ1",
  "client_policy": { "min_supported": "5.2.0", "recommended": "5.14.0", "flush_only_until": null },
  "flags_version": 43
}

A version below min_supported gets 400 with "error": "CLIENT_VERSION_UNSUPPORTED" on reads, and the app shows its upgrade screen (step 3.1).

2. Flags and kill switches, served as static files from CloudFront. A small latest.json names the current version; each version's document has its own path and never changes:

httpCopy
GET /flags/v43/android.json HTTP/1.1
Host: cfg.example-social.com

jsonCopy
{
  "version": 43,
  "kill": { "poll_posts": true },
  "flags": {
    "poll_posts":   { "default": false, "rollout_percent": 20, "min_app": "5.14.0" },
    "avif_images":  { "default": true }
  },
  "client_policy": { "min_supported": "5.2.0", "recommended": "5.14.0" },
  "telemetry": { "session_sample_percent": 10 }
}

3. Telemetry, in batches:

httpCopy
POST /v1/telemetry:batch HTTP/1.1
Authorization: Bearer <token>
Content-Type: application/json

{
  "install_id": "i_4f9c2e",
  "app_version": "5.14.2", "platform": "android", "device_class": "mid-2023",
  "day": "2026-09-28",
  "health": { "sessions": 6, "crashes": 0, "syncs_ok": 19, "syncs_failed": 2,
              "queue_depth_max": 3, "queue_age_max_s": 540, "quarantined": 0,
              "ttc_ms_p50": 38, "ttc_ms_p95": 120, "wal_bytes_max": 4194304 }
}

install_id is a random ID made at install time, not an advertising ID and not the user ID.

R3.4 Design Evolution: Versions, Migrations, Rollouts, Devices and Platforms

Step 3.1: 30 App Versions Talk to Our API

Step 3.2: A New Release Changes the Local Database

Step 3.3: A Release Is Crashing on Some Phones

Step 3.4: A Lost Phone Exposes the Feed and Drafts

Primitive: OAuth2, OIDC & Distributed Token Authentication · Drill: JWT revocation token blacklist

Step 3.5: We Can't See What's Happening on Devices

Step 3.6: Web and Mobile Behave Differently

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

Three clients share one sync core and one specification. The sync gateway turns one internal model into each protocol version. Two new paths serve every version: flags flow from AppConfig to static documents on CloudFront, and telemetry flows from phones to Firehose, S3 and Athena. The feed backend below the log (fanout, timelines, ranking) is unchanged from Round 2 and the server loop.

Trace 1: an old app version syncs a poll.

Synthesizing vector architecture diagram...

When this user updates, the new app refetches posts it stored as fallbacks and shows the poll.

Trace 2: a kill switch.

Synthesizing vector architecture diagram...

Trace 3: a local migration with queued actions.

Synthesizing vector architecture diagram...

Every path keeps the outbox if it can be read at all, and every path ends with caches rebuilt from the server.

R3.6 Numbers and Cost

Traffic (80M DAU; 20 syncs and 5 actions per user per day as in Round 2; peak 3× average)

Sync traffic across versions (planning split by protocol: v3 72%, v2 23%, v1 5%). Every version reads the same log; only the encoding differs. We assume encoding for v1 and v2 costs about 10% more CPU per sync (translation and fallbacks), which adds $0.28 \times 10\% \approx 2.8\%$ to the gateway fleet.

Write rates we must size (heartbeat-like traffic that grows with installs, not with engagement)

The telemetry peak of about 33 MB/s is over Firehose's default Direct PUT quota of 5 MiB/s per stream (with 2,000 requests/s and 500,000 records/s) in us-east-1. We request an increase before launch (or spread across several streams), and the ingest service batches up to 500 records per PutRecordBatch call.

Fleet sizing (same per-task rates as Round 2, AZ-loss sizing, rounded per AZ)

Why not let phones read AppConfig directly? At list price AppConfig charges about $0.0008 each time a client receives configuration. 240\text{M} \times \0.0008 = $192\text{K} a day, about \5.8M a month. The same documents from CloudFront cost about $7.6K.

Monthly cost (us-east-1 list prices; CloudFront at United States and Europe rates; rounded)

About $0.004 per daily user per month, the same as Round 2: the new work (telemetry, flags, extra protocol versions) is small next to delivery. Images are still more than half the bill. The regional price matters too: these are United States and Europe CloudFront rates, and viewers in Asia or South America cost more per GB. CloudFront's cheaper price classes limit which edge locations serve our content; they don't limit where users are, so a user in Asia is then served from a farther edge, slower, not refused.

Firehose bills each record in 5 KB increments, so the 2 KB health summaries are billed as 5 KB. At this size it doesn't matter ($0.35K a month for all of them); it would if we sent one tiny record per event.

R3.7 Trade-Offs

Closing the loop. The opening question was: how do we make the app instant and fully usable offline, and still keep it in agreement with the server? The answer is now:

• Instant: the screen reads only the phone's database; the network never sits between a tap and the screen (Round 1).
• Fully usable offline: actions go into an outbox in the same transaction as the change the user sees, and the server's guards make every retry safe (Round 2).
• In agreement with the server: changes come down through a per-user log with numbers the server assigns, deletes included, and a phone that falls behind rebuilds instead of guessing (Round 2).
• And it stays that way over time: every version speaks a compatible protocol, local upgrades protect what only the phone has, and features can be turned off without an app-store release (Round 3).

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals, per app version and platform OPS 8 · OPS 4

Staged-rollout halt procedure OPS 6 · REL 8

1. Stop the spread. Pause the phased release in App Store Connect; halt the staged rollout in the Play Console. (Neither removes the version from phones that have it.)
2. Find the feature. Crash reports by version and device model; the top crash's stack names the screen or flag.
3. Kill it. Set the flag in the kill layer in AppConfig and deploy (CLI 2). Watch flags_version reach the sync fleet (CLI 3), then crash-free users recover over the next minutes as phones sync.
4. If there's no flag for it, raise min_supported only as a last resort, and only with flush_only_until set, so queued actions still drain.
5. Fix forward. Ship a new build with a higher version number; resume the rollout from 1%.
6. Record it: who, what, when; add a flag for anything that shipped without one.

Go deeper: CLI playbook

Plain commands an on-call engineer runs, one at a time. Replace the IDs with real ones.

textCopy
# 1. Crash-free users by app version, yesterday (health summaries in Athena)
aws athena start-query-execution --work-group telemetry --query-string "SELECT app_version, 1 - CAST(approx_distinct(IF(crashes > 0, install_id)) AS double) / approx_distinct(install_id) AS crash_free_users FROM health_daily WHERE day = '2026-09-27' GROUP BY app_version ORDER BY crash_free_users"

# 2. Kill switch: deploy the new kill-layer version at once
aws appconfig start-deployment --application-id a1b2c3d --environment-id e4f5g6h --configuration-profile-id k7l8m9n --configuration-version 44 --deployment-strategy-id AppConfig.AllAtOnce

# 3. Which flags version is published?
aws s3 cp s3://example-flags/flags/latest.json -

# 4. Log writer lag
aws cloudwatch get-metric-statistics --namespace AWS/Kinesis --metric-name GetRecords.IteratorAgeMilliseconds --dimensions Name=StreamName,Value=sync-log-events --statistics Maximum --period 60 --start-time 2026-09-28T09:00:00Z --end-time 2026-09-28T10:00:00Z

# 5. Log table write throttling
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions Name=TableName,Value=FeedSyncLog --statistics Sum --period 60 --start-time 2026-09-28T09:00:00Z --end-time 2026-09-28T10:00:00Z

# 6. Scale the sync gateway
aws ecs update-service --cluster sync --service sync-gateway --desired-count 220

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Treats time as a dimension of the system: many versions at once, and designs the protocol, fallbacks and retirement policy for it.
• Splits local data into "copy of the server" (rebuild) and "only on this phone" (migrate, rescue first), and tests every upgrade path.
• Knows what app-store rollouts can't do, and puts a kill layer, safe defaults and a crash-loop guard in place before they're needed.
• Chooses encryption classes by what background work must do, and designs revocation and wipe for lost and shared devices, including backups and reinstalls.
• Builds telemetry that counts people, not events, and respects privacy by design.
• Keeps three platforms honest with one spec and a conformance suite.
• Prices the new paths and catches the traps (AppConfig per-receive pricing, Firehose's default quota and record rounding).

Follow-up questions

1. "The backend becomes multi-region (see the server loop's Round 3). What happens to the per-user log and cursors in a failover?" Answer: the log stays single-writer per user: only the user's home region writes it. If the home region is lost and the partner takes over that user's writes, it must not continue the old numbers, because the last entries may not have replicated. So the partner starts a new log epoch for the moved users; phones presenting a cursor from the old epoch get 410 and rebuild from the head, keeping their outboxes. The epoch is part of the log's key (USER#<id>#E#<epoch>, with one META per epoch), so late replication of old-epoch entries from the returning region lands in the old log and can't touch the new one. Actions whose guards replicated before the failure stay safe. For the last seconds of lag, a resent comment can be applied twice, so a reconciliation job flags comments sharing a client_mutation_id once the region returns. Failover is detected by health checks evaluated outside the failed region (Route 53's health checkers, not calls into the failed region's APIs), and the DNS move is gated: users are routed to the partner only after it has been promoted to write their logs, and the partner needs capacity for the rebuild storm (millions of 60 KB head calls). Where users' data may live is a residency decision made before any of this: a user may only fail over to a region their data is allowed to be in.

2. "Why not force everyone onto the latest version every month?" Answer: the store can't make people update; a forced-upgrade screen turns into uninstalls for users short of storage or on old OS versions that the new build doesn't support. We force only below min_supported, with data (share under 0.5% for 30 days) or a security reason, and even then let the outbox flush.

3. "A user says their likes keep disappearing. How do you debug it?" Answer: from their install_id (they can share it from a debug screen), check the health summary: quarantine count and rollback rate. If rejections are the cause, the server logs by client_mutation_id show the error code. If there are no rejections, check for a sync overwriting server state in one version (the overlay bug from R2.9) or a drainer resending acknowledged rows (a DUPLICATE spike). The version split usually points at the build.

---

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint.

The Two Sentences That Matter Most

• Opening a round: "Before I design: must it work offline, for reading only or for actions too, and how much may we keep on the phone?"
• When the scope is raised: "Here's what breaks, and I'll fix it in this order: anything that can lose or duplicate a user's action, then anything that can show them something wrong, then battery, data and cost."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels