Shopify: Pods, Flash Sales and Payment Resilience

This page is one interview loop in three rounds, built on a real company's public engineering history. All three rounds design the same system: the part of Shopify that serves storefronts, takes buyers through checkout and charges their cards, for millions of shops that share one platform. Each round is an era. In each era, a new kind of load forced Shopify to change how that shared platform keeps one shop's big moment from hurting everyone else.

You can start at any round. Rounds 2 and 3 open with a "Where we left off" summary of the earlier rounds.

Loop Opener: Why Shopify?

One Shop's Sale Can Slow Down Every Other Shop

Picture a shopping mall with a million small shops and a few famous ones, all sharing the same doors, corridors and cash registers. Most days, traffic spreads out. Then a celebrity announces a new lipstick on social media, and in ten seconds a hundred thousand people run for the same shop. If the mall is built naively, the crowd jams every entrance, and the bakery at the other end loses its customers too.

That is Shopify's problem. In Shopify's own words (2017), its merchants put it to the test daily: "on any given day, they'll hurl Super Bowl-sized traffic, often without notice." Its first big flash sale came in 2007, when fans rushed to buy T-shirts after the Super Bowl. In February 2016, a Kylie Cosmetics sale "took down not just her store, but all others on the database shard where her store was allocated". Shopify's stated policy (2017): "Shopify decided to never fire a customer for the amount of traffic they bring."

Synthesizing vector architecture diagram...

The timeline lists, in order, the published changes to Shopify's platform from 2006 to 2025. Shopify launched on Ruby on Rails in 2006, and its first big flash sale came in 2007, after the Super Bowl. By 2014 it hosted 100,000 stores and built Semian and Toxiproxy to prepare for BFCM. In 2015 the one database was sharded; in 2016 the Kylie sale took down a shard, and pods and Sorting Hat followed. In 2016 and 2017 checkout got a throttle and then a fair queue. In 2018 Shopify announced it was building its cloud with Google and built the Shop Mover; from 2019 to 2022 it described its modular monolith and its idempotent payment service; and in 2025 BFCM peaked at 489M requests a minute at the edge. Read it as three eras, one per round. Round 1 is one application and one database; Round 2 splits the data into isolated pods and defends checkout during flash sales; Round 3 prepares the whole platform, including payments, for the biggest weekend of the year.

A few words we'll use all page:

What Makes It Hard

• Traffic is spiky per tenant, not just per season. One shop can bring orders of magnitude more traffic than its baseline for a few minutes (Shopify, 2017). You can't plan capacity per shop.
• Stock is scarce and must not oversell. Thousands of buyers race for a few thousand units. Every "yes" must be backed by a real unit.
• Checkout depends on outside systems. A payment provider that slows down keeps our workers waiting for its answers, and a timeout doesn't tell us whether the card was charged.
• Everyone shares the platform. A shop's sale, a slow query or a failing provider must stay inside its own blast radius (the set of shops and buyers that one failure affects).

The Question the Whole Loop Answers

How do we let one merchant's biggest moment happen without hurting anyone else, and without overselling or charging anyone twice?

The answer grows every round:

• Round 1: cache the reads, make every stock decrement and every payment safe under concurrency and retries, and split background work by priority, all on one database.
• Round 2: isolate shops into pods with a router in front, move shops between pods without downtime, and put a fair queue and a stock gate in front of checkout during flash sales.
• Round 3: fail fast around slow dependencies with circuit breakers and bulkheads, prove it with failure injection and full-scale load tests, resolve unknown payment outcomes, and run pods across cloud regions that can take over for each other.

Round 1 · Mid-level · "Era 1: One Rails App, One Database"

~35 min · SDE II (L5) · about 2006–2015 · 100,000 shops (published, 2014) · 3,500 storefront requests/s and 10 orders/s at the daily peak (planning assumptions) · stay up as merchants grow, never oversell, never charge twice

R1.1 Establish Design Scope

The interviewer sets the scene: "It's 2014. We host 100,000 online stores. There's one Ruby on Rails application and one big MySQL database for all of them. Merchants run their own sales, and some run flash sales that bring a crowd in a minute. Last month one of them made the whole platform slow. Tell us how you'd keep it running, and what you'd fix first."

Out of scope for this round: sharding and pods (Round 2), waiting rooms (Round 2), multiple regions (Round 3).

R1.2 Functional Requirements, Derived Step by Step

Not yet: pods, a checkout queue, a stock gate, resilience to a whole region failing.

R1.3 Non-Functional Requirements: the Questions

We name each quality first; the numbers come in R1.7.

• Inventory correctness. Never sell a unit we don't have, unless the merchant chose to (Shopify lets a merchant keep selling when out of stock, per product variant: inventoryPolicy DENY or CONTINUE).
• Payment correctness. A buyer is charged at most once per order, even when requests are retried.
• Availability. One shop's traffic must not take down the others. (This round can only soften that; Round 2 enforces it.)
• Storefront latency. Product pages must load fast: most traffic is buyers browsing without buying yet, and slow pages cost sales.
• Checkout latency. The buyer is waiting on a loading screen. Shopify's 2022 payments post suggests a one-second connect timeout and a five-second read timeout as "a decent starting point", from the view of a person who won't wait longer.

R1.4 The API

This API is illustrative: Shopify's internal checkout endpoints aren't public in this form, so the paths and fields below are ours.

Start a checkout

httpCopy
POST /v1/shops/tshirthero/checkouts HTTP/1.1
Content-Type: application/json
Idempotency-Key: 01J9ZQ3C6X8R2M4T7V1B5N0K9D

{
  "line_items": [ { "variant_id": 4471, "quantity": 1 } ],
  "email": "sam@example.com",
  "shipping_address": { "country": "CA", "zip": "H2X 1Y4" }
}

httpCopy
HTTP/1.1 201 Created
Content-Type: application/json

{
  "checkout_id": "chk_5f1c2a9e",
  "status": "OPEN",
  "total": { "amount": "32.00", "currency": "CAD" }
}

Complete it (pay)

httpCopy
POST /v1/shops/tshirthero/checkouts/chk_5f1c2a9e/complete HTTP/1.1
Content-Type: application/json
Idempotency-Key: 01J9ZQ4F2H7W3K8P1Q6S0D5M2A

{
  "payment": { "card_token": "ct_91ab", "amount": "32.00", "currency": "CAD" }
}

httpCopy
HTTP/1.1 200 OK
Content-Type: application/json

{ "checkout_id": "chk_5f1c2a9e", "status": "COMPLETED", "order_id": "ord_8842" }

• Idempotency-Key names one attempt. A retry with the same key returns the same result instead of charging again (step 1.3). We use a ULID (a sortable ID: 48 bits of time, then 80 random bits), the choice Shopify published in 2022, because keys that increase over time insert at the end of a B-tree index.
• card_token, never a card number. Shopify (2022 talk) keeps its main app out of card data entirely: the card fields are iframes on a separate domain that post to a service called CardSink, which returns a token. We come back to this in Round 3.

Status codes

A checkout's states

Synthesizing vector architecture diagram...

The diagram shows every state a checkout can be in and the event that moves it. A checkout starts OPEN when the buyer enters details. When the buyer asks to complete it, it becomes PAYING. From PAYING there are three exits: the payment succeeds and the order is created (COMPLETED); the card is declined and the checkout returns to OPEN so the buyer may retry; or the provider times out and the checkout becomes UNKNOWN. An UNKNOWN checkout leaves that state only when the provider confirms the charge (COMPLETED) or confirms that there was no charge (OPEN). An OPEN checkout that the buyer leaves becomes ABANDONED. A checkout only moves along these arrows. UNKNOWN is a real state, not an error: a timeout means we don't know yet whether the card was charged, and the buyer must not be allowed to start a second charge while we find out.

Recap

• Storefronts are read-heavy and cacheable; checkout is write-heavy and talks to outside providers.
• Every write that matters is idempotent, and every checkout has a small set of explicit states.

R1.5 Design Evolution: Making One Database Survive

Each step is a problem, your turn to think, the answer, and what it costs us.

Step 1.0: The Baseline

Synthesizing vector architecture diagram...

The diagram shows the starting point: one application and one database. Buyers reach the Rails monolith through load balancers; the monolith serves the storefront, checkout and the merchants' admin pages. The monolith reads and writes every shop's data in one MySQL database that holds all 100,000 shops, and it calls the outside payment providers directly. Background jobs are rows in a jobs table in the same MySQL database: the monolith inserts them, and separate job workers read and update them. So storefront reads, checkout writes and the background job queue all land on the same MySQL server, and anything that loads that server slows all three for every shop.

Shopify published this shape: "Our main tool of choice for building backend system is Ruby on Rails with MySQL, Redis, and memcached as our datastores" (2022 talk), and its main monolith is "under continuous development since at least 2006" (2020). Until 2015, Shopify kept "buying a larger database server" (2018).

Step 1.1: Storefront Reads Overload the Database

Primitive: Distributed Cache Patterns and Eviction · Drill: The Product Page That Melted Redis (answered here: the cold-cache stampede and why "cache forever" is the wrong trade)

Step 1.2: Two Buyers Got the Last Unit

Synthesizing vector architecture diagram...

Primitive: Database Isolation Levels, ACID and Concurrency Anomalies · Loop primitive: Write-Ahead Log, fsync & Group Commit (why a commit's flush is part of every lock hold) · Loop: Design a Hotel Reservation System (Round 1, step 1.2: the same race for the last room)

Step 1.3: A Buyer Was Charged Twice

The key table (our schema; same database and shard as the checkout, so the claim and the order can share a transaction):

sqlCopy
CREATE TABLE payment_requests (
    shop_id          BIGINT UNSIGNED NOT NULL,
    idempotency_key  CHAR(26) NOT NULL,               -- ULID
    checkout_id      BIGINT UNSIGNED NOT NULL,
    request_hash     BINARY(32) NOT NULL,             -- SHA-256 of the body; a different body under the same key gets 422
    status           ENUM('IN_FLIGHT','COMPLETED','UNKNOWN') NOT NULL,
    locked_until     TIMESTAMP(3) NOT NULL,           -- lease on IN_FLIGHT, from the database clock: NOW(3) + 30 s
    provider_charge  VARCHAR(64) NULL,
    response_code    SMALLINT NULL,
    response_body    JSON NULL,
    created_at       TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
    PRIMARY KEY (shop_id, idempotency_key)
) ENGINE=InnoDB;

Loop primitive: Idempotency & Effectively-Once Processing (the section The key record and the durable guard: claim, record, run, answer, on the durable guard, claimed before the side effect) · Loop: Design a Payment Processing System (Round 1, steps 1.1 and 1.2, in depth)

Step 1.4: The Background Job Queue Grows During a Sale

Loop primitive: Change Streams & the Transactional Outbox (the sections Why writing twice fails, on why writing to two systems fails, and The relay and at-least-once delivery) · Primitive: Message Queues vs Event Streams

Round 1 Step Summary

R1.6 Architecture v1

Synthesizing vector architecture diagram...

The diagram shows Round 1's end state. Buyers reach the CDN, which serves static assets, then the load balancers and the Rails monolith. The monolith reads rendered pages from the Memcached page cache first. On a miss, browsing queries go to the read replicas, which MySQL feeds by asynchronous replication, while anything that decides stock or money goes to the MySQL primary. The monolith writes outbox rows in the same transaction as the order, and the relay moves them into Redis job queues, one per priority (critical, default, low). Job workers, with one pool per tier, run the jobs and call the payment providers for follow-ups; the monolith also calls the providers directly during checkout. So jobs no longer add writes to the database, and a flood of low-priority jobs can use only its own pool of workers.

Trace: one checkout completes (our design around the published shape)

Synthesizing vector architecture diagram...

Our AWS translation (not Shopify's setup). The monolith on EKS behind a Network Load Balancer; CloudFront for assets; one Aurora MySQL cluster with readers; ElastiCache for the page cache and for the job queues.

Sources for this round

• Shopify, 100,000 Online Stores Now Use Shopify, Shopify blog, 2014: the 100,000-store milestone.
• Eskildsen, Building and Testing Resilient Ruby on Rails Applications, Shopify Engineering, January 2015: BFCM 2014 preparation; the session store example.
• Stolarsky, Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part I), Shopify Engineering, February 2017: the 2007 flash sale; one checkout record updated at every step; the dynamic caching layer that "helped but not enough".
• Denis, A Pods Architecture To Allow Shopify To Scale, Shopify Engineering, March 2018: buying ever-larger database servers until 2015.
• Müller, Under Deconstruction: The State of Shopify's Monolith, Shopify Engineering, September 2020: the monolith in development since at least 2006.
• de Water, Shopify's Architecture to Handle the World's Biggest Flash Sales, QCon Plus talk, published by InfoQ in October 2022: Rails, MySQL, Redis and Memcached; the client-side cart; CardSink; Delayed Job and Resque.
• de Water, 10 Tips for Building Resilient Payment Systems, Shopify Engineering, July 2022: timeouts; idempotency keys and ULIDs; "once in a million".

Everything else in this round (the cache design, the API shapes, the tables, the job tiers, the outbox and every number in R1.7 not marked published) is our design or our assumption.

R1.7 Numbers

Figures marked published come from the sources above; the rest are assumptions (2014 planning figures) or derived from them.

Storefront reads at the daily peak

Checkout writes at the daily peak

One flash sale on the shared database

After about nine seconds, every shop on the database starts getting "too many connections" errors, not just the one running the sale. With the hot-row update moved last (about 1,000/s ceiling), this backlog doesn't form, but the 2,000 extra writes a second remain. That is the limit this era reaches: Shopify's February 2016 outage took down every shop on the sale's database shard.

Synthesizing vector architecture diagram...

The chart compares two bars from the flash-sale table: 416 writes a second at the platform's normal daily peak, and 2,416 writes a second when one shop's sale adds 2,000 checkout-row writes a second on top. The sale alone is 4.8 times the whole platform's normal write peak, and the total is 5.8 times. Caching can't reduce these numbers, because every one of them is a write that the one MySQL primary must perform. On a shared database, every other shop's writes wait behind this load.

R1.8 Trade-Offs

A monolith vs services (Shopify's published stance)

Shopify chose a modular monolith: "we would keep all of the code in one codebase, but ensure that boundaries were defined and respected between different components" (2019). By 2020 the core monolith had "over 2.8 million lines of Ruby code", and in the 2022 talk it was deployed "around 40 times a day". Boundaries are checked automatically with Packwerk, an open-source static analysis tool that flags one component reaching into another's internals (2022 talk).

The point to make: a monolith doesn't stop you from scaling the data. Round 2 shards the data under the same app.

Where the stock check lives

R1.9 Failure Modes

R1.10 Pillar Check

R1.11 Round 1 Rubric and Follow-Ups

What a strong mid-level (L5) answer shows

• Separates cacheable storefront reads from checkout writes, and knows caches don't fix writes.
• Makes the stock decrement one atomic conditional statement, and knows where the lock is held.
• Claims an idempotency key durably before calling the provider, and passes a key to the provider too.
• Splits background work by priority, and spots the dual write when jobs leave the database.
• Computes why one shop's sale can saturate a shared database.

Follow-up questions

1. "Why not SELECT ... FOR UPDATE on the inventory row, then decide in Ruby?" Answer: it's correct, but it holds the row lock across a round trip to the app and back, which lowers the hot row's ceiling. The conditional UPDATE decides inside the database in one step.

2. "The idempotency key is in the same database as the order. What if that database is down?" Answer: then checkout is down anyway: we can't create orders. Putting the key next to the order is what lets the claim, the order and the stock change commit together. A key store in a separate system would add a dual write.

3. "A buyer pays, the stock update finds 0 rows. Now what?" Answer: roll back the order, keep the payment request's record, and void the authorization with the provider (a void releases the hold on the card; nothing was captured). Tell the buyer the item sold out. Round 2 makes this rare by holding stock before payment.

Interview gotchas from this round's wrong answers

Round 2 · Senior · "Era 2: Pods and Flash-Sale Defenses"

~40 min · Senior SDE (L6) · about 2015–2018 · 600,000+ merchants (published, March 2018) · 100 pods and one sale of 100,000 buyers for 5,000 units (planning figures) · one merchant's sale never degrades the others

R2.0 Where We Left Off

This is what the candidate says aloud in the first 60 seconds of Round 2. If you're starting here, it's everything you need from Round 1.

Round 1 in 60 seconds. "Shopify is one Rails monolith, and until 2015 it ran on one big MySQL database for every shop. Storefronts are reads, so we cached rendered pages in Memcached with a per-shop version for instant invalidation, coalesced rebuilds to avoid stampedes, and kept stock and money decisions on the primary. Stock is a conditional UPDATE ... WHERE available >= qty, placed last in the transaction, because the row lock lasts until commit and one hot row can only take about 1 ÷ (lock hold time) orders a second. Payments claim an idempotency key with an insert-if-absent before calling the provider, and pass a key to the provider too; a timeout is UNKNOWN, not failed. Jobs moved from the database to Redis queues split by priority, fed through an outbox. Open costs: one database for everyone, so one flash sale adds several times the platform's normal write load, and every shop on the database goes down with it."

Architecture v1, compact

Synthesizing vector architecture diagram...

The diagram is Round 1 in one picture. Buyers reach the Rails monolith, which reads pages from the page cache, writes to one MySQL database that holds every shop, puts jobs on Redis queues and calls the payment providers. Reads are cached and writes are safe under concurrency and retries, but every shop's writes still land on the one database. Round 2 starts from exactly this: one shop's sale can still overload the database that all other shops use.

Round 1 step summary

Open costs: a single shared database, no way to hold back a crowd, and no way to keep one shop's sale away from its neighbors.

R2.1 The Scope Raise

Interviewer: "It's 2016. We've sharded the database, and it bought us capacity, but in February a celebrity's lipstick sale took down every shop on her shard. She sells again next week, and the week after. We're heading for hundreds of thousands of merchants. We need one merchant's sale to never hurt the others, a crowd at checkout that doesn't crash anything and doesn't feel like a lottery, no overselling, and bots not buying everything."

Scope change

The "Not yet" list from R1.2 comes back: pods, a checkout queue and a stock gate are in scope now.

R2.2 What Breaks in the Round 1 Design

R2.3 New Requirements and API Additions

1. Every request belongs to one pod. Shopify (2021) keeps a routing table that maps each shop's domain to a pod ID, "in a separate database that isn't sharded". The shape below is ours:

sqlCopy
CREATE TABLE shop_routes (
    domain        VARCHAR(255) NOT NULL,         -- 'tshirthero.com' or 'tshirthero.myshopify.com'
    shop_id       BIGINT UNSIGNED NOT NULL,
    pod_id        SMALLINT UNSIGNED NOT NULL,
    route_version BIGINT UNSIGNED NOT NULL,      -- bumped on every move
    PRIMARY KEY (domain),
    KEY idx_shop (shop_id)
) ENGINE=InnoDB;

A request then carries the answer to the app as a header (Shopify's load balancers add one; the name here is ours):

httpCopy
GET /products/hoodie HTTP/1.1
Host: tshirthero.com
X-Pod-Id: 17
X-Route-Version: 42

2. A queue pass for checkout during a sale. When the checkout throttle is engaged for a shop, a buyer who gets through receives a signed cookie. Shopify published that the cookie is "securely signed", carries the buyer's first-attempt timestamp while they wait, and lets an admitted buyer "skip the throttle for the rest of their session". This payload is our design:

jsonCopy
{
  "v": 1,
  "kid": "q-2016-11",
  "shop_id": 812,
  "first_seen_ms": 1478530800123,
  "admitted_ms": 1478530811456,
  "exp_ms": 1478531711456,
  "session_hash": "b3f1c0e2",
  "pass_id": "01BX5ZZKBKACTAV9WEVGEMMVRZ"
}

The cookie is base64url(payload) + "." + base64url(HMAC-SHA256(key[kid], payload)). HMAC-SHA256 is a keyed hash: only a holder of the key can compute it, so a payload that was edited no longer matches its signature. kid is the key ID, which says which signing key was used. Every load balancer verifies the cookie without a data store:

1. Reject a cookie over 1 KB or one that doesn't parse.
2. Look up kid in the load balancer's local set of signing keys (we rotate keys; the old key stays valid until every pass it signed has expired).
3. Recompute the HMAC and compare in constant time. A mismatch means forged or edited.
4. Check shop_id against the shop Sorting Hat resolved for this request. A pass for one shop is useless at another.
5. Check exp_ms against the load balancer's clock, allowing 2 seconds of clock skew. The edge wrote both timestamps, so the buyer can't move them.
6. Check session_hash against a hash of the buyer's session cookie, so a pass copied to another browser fails.
7. Let the request through. The pass is not single-use at the edge (that would need a shared store); instead, the pod's database allows one stock hold per pass_id (next item), which is what stops a shared pass from buying twice.

3. Stock is held during checkout. Shopify hasn't published how its checkout holds stock during a sale, so this is a design that fits: when a buyer with a pass starts checkout, we hold the units for 10 minutes; payment turns the hold into a sale; an expired hold gives the units back.

sqlCopy
CREATE TABLE inventory_holds (
    shop_id      BIGINT UNSIGNED NOT NULL,
    hold_id      CHAR(26) NOT NULL,              -- ULID
    variant_id   BIGINT UNSIGNED NOT NULL,
    bucket       TINYINT UNSIGNED NOT NULL,      -- which stock bucket it came from (step 2.4)
    quantity     INT UNSIGNED NOT NULL,
    checkout_id  BIGINT UNSIGNED NOT NULL,
    pass_id      CHAR(26) NULL,                  -- one hold per queue pass
    status       ENUM('ACTIVE','PAYING','CONVERTED','RELEASED') NOT NULL,
    expires_at   TIMESTAMP(3) NOT NULL,          -- set from the database clock
    PRIMARY KEY (shop_id, hold_id),
    UNIQUE KEY uq_pass (shop_id, pass_id),       -- NULLs don't collide in MySQL, so shops without a queue are unaffected
    KEY idx_expiry (status, expires_at)
) ENGINE=InnoDB;

Synthesizing vector architecture diagram...

The diagram shows the states of one stock hold. A hold starts ACTIVE when its stock is held, with a 10-minute timer. From ACTIVE, it becomes RELEASED if the timer expires or the buyer leaves, or PAYING when the buyer presses Pay, which stops the timer. From PAYING, it becomes CONVERTED when the payment succeeds and the order is committed, goes back to ACTIVE with a new timer when the payment attempt is finally DECLINED (the buyer may retry), or becomes RELEASED when the attempt is finally VOIDED. Each transition is a conditional update that names the state it leaves, so a hold is released exactly once, by the transition that wins. Only an ACTIVE hold has a timer. A hold in PAYING is never released by a timer, so a slow card challenge or an UNKNOWN payment can't lose the buyer's unit.

Recap

• Every request and job names one pod; the routing table is a lookup, so any shop can live on any pod.
• A queue pass is verified with a key and a clock, no shared store.
• Stock is held with an expiry set by the database's clock, and released exactly once.

R2.4 Design Evolution: Isolate the Tenants, Then Defend Checkout

Step 2.1: One Database Unit for Hundreds of Thousands of Shops

Synthesizing vector architecture diagram...

The diagram follows one request. A request for tshirthero.com reaches Sorting Hat in the load balancer, which looks the domain up in the unsharded routing table and finds pod 17. Sorting Hat adds the header X-Pod-Id 17 and passes the request to the shared app workers, which connect only to the datastores in the subgraph Pod 17: its own MySQL, Redis and Memcached. The subgraph Pod 18 has its own separate copies of the same three datastores, and nothing in this request path reads them. Each pod's MySQL also feeds the data warehouse through a replica, so questions that span many shops are answered there. So the load balancer decides the pod before the app sees the request, workers are shared, and datastores are not.

Primitive: Database Sharding and Partition Keys · Drill: One Customer, One Shard, One Outage (answered here and in step 2.2: the biggest tenant gets its own pod through a pinned route before its load arrives, and platform-wide reports read the warehouse, not the pods)

Step 2.2: One Merchant's Sale Slows the Other Shops in Its Pod

Loop primitive: Leases, Fencing Tokens & Distributed Locks (the section The pause, the zombie and the fencing token) · Loop primitive: Change Streams & the Transactional Outbox (the section Reading the database's own log)

Step 2.3: A Hundred Thousand Buyers Hit Checkout at Once

Synthesizing vector architecture diagram...

Primitive: Distributed Rate Limiting · Drill: The Partner Whose Retry Loop Took Down Everyone Else (answered here: per-node limits must be divided across nodes, and a reset-every-period bucket allows a double burst at the boundary where a token bucket doesn't)

Step 2.4: Every Checkout Updates the Same Stock Row

Synthesizing vector architecture diagram...

The flowchart shows the same steps as the hold pseudocode above. Admitted buyers arrive at 250 a second and ask the Redis gate whether a counter can cover qty. If the counters are missing, MySQL decides alone. If a counter can, the gate picks bucket b, decrements its counter, and MySQL tries bucket b first. If none can, the database is read at most once a second per variant: if it still shows stock, the gate is re-seeded and MySQL tries the buckets; every other buyer gets "sold out" in about 1 ms without reaching MySQL. In MySQL, one transaction inserts the hold and runs the conditional bucket decrement. When one row is updated, checkout continues with the hold ACTIVE for 10 minutes. When a bucket updates 0 rows, that bucket's counter is lowered to the database's value and the next bucket is tried; when every bucket updates 0 rows, the buyer gets "sold out". The dotted lines are the return path: the reaper, running on the database's clock, releases expired holds and returns their units to the gate exactly once, which keeps the gate from underselling. So the gate turns away the crowd after the sellout, and the database decides every real claim.

Primitive: Distributed Locks and Leases · Loop primitive: Leases, Fencing Tokens & Distributed Locks (the sections Time: whose clock decides? and Conditional writes are the check) · Loop: Design a Hotel Reservation System (Round 2, steps 2.2 and 2.3)

Step 2.5: Bots Buy the Stock Before People Can

Primitive: Bot Defense, Sybil Resistance and Registration Abuse · Primitive: Cloudflare Turnstile Managed Challenges

Round 2 Step Summary

R2.5 Architecture v2

Synthesizing vector architecture diagram...

The diagram shows Round 2's end state. Buyers and bots reach the CDN for assets, then the subgraph Edge: OpenResty load balancers, where the bot filter, Sorting Hat and the checkout throttle with its queue page run. Sorting Hat reads a local copy of the routing table. Requests that pass go to the shared Rails workers, which connect to exactly one pod per unit of work (pod 1, pod 17 with its MySQL, Redis gate, Memcached and job queues, up to pod N) and to the payment providers. The Shop Mover, built on Ghostferry, copies a shop's rows and binlog changes between pods and then updates the shop's pod in the routing table. So a request is filtered, routed and, during a sale, throttled before any Rails code runs, and moving a shop is a copy plus a routing-table update.

Trace 1: a flash-sale checkout (our design around the published shape)

Synthesizing vector architecture diagram...

Trace 2: moving a shop from pod 1 to pod 2 (published phases, our timings)

Synthesizing vector architecture diagram...

Our AWS translation (not Shopify's setup). The edge as OpenResty on EKS behind a Network Load Balancer, with CloudFront and AWS WAF in front; each pod as its own Aurora MySQL cluster, ElastiCache (Valkey) for the gate and job queues, and ElastiCache for Memcached; the routing table in a small unsharded Aurora MySQL cluster (a DynamoDB table would also fit); shared Rails workers on EKS.

Sources for this round

• Stolarsky, Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers, Part I and Part II, Shopify Engineering, February 2017: Kylie's weekly sales and the February 2016 shard outage; Nginx and Lua; the leaky bucket with 5-second periods; the cached queue page and signed cookie; the 40-minute lottery; timestamps, the threshold and the P controller; why not a data store.
• Denis, A Pods Architecture To Allow Shopify To Scale, Shopify Engineering, March 2018: sharding in 2015; with_each_shard; pods; shared workers; one pod per unit of work; Sorting Hat; pairs of data centers and the Pod Mover.
• Neufeld, Shopify's Infrastructure Collaboration with Google, Shopify Engineering, March 2018: 600,000+ merchants; Kylie selling out in 20 seconds; the Shop Mover's 2.5-second average downtime.
• Madan, Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale, Shopify Engineering, September 2021: pods with MySQL, Redis and Memcached; the routing table; four-times and two-times utilization spreads; Ghostferry's phases; the multi-reader, single-writer lock; verification and pruning; TLA+.
• de Water, Shopify's Architecture to Handle the World's Biggest Flash Sales, QCon Plus talk, published by InfoQ in October 2022: shared stateless workers; OpenResty, bot blocking and Sorting Hat; round-robin assignment; extra-large merchants on their own pods; move downtime; the warehouse fed from replicas.
• Shopify open source: Ghostferry (checked September 2026).

Everything else in this round (the pod count, the pass payload and its checks, the hold table, the gate, the buckets, the bot layers, the translation and every number in R2.6 not marked published) is our design or our assumption.

R2.6 Numbers and Cost

All figures are assumptions unless marked published or derived.

Pods

The sale and the admission rate

The stock rows

Holds

Keeping the line open after "sold out" is what makes the second wave fair: returned units go to the next buyers in line, not to whoever refreshes fastest.

The gate after a Redis failover

A shop move

Cost: spreading app workers across three AZs (AWS translation, us-east-1; a 30-day month of 720 hours on every cost on this page). Each pod's writer lives in one Availability Zone, so about two-thirds of app-to-database traffic crosses zones, and AWS charges $0.01/GB in each direction: $0.02 for every GB that crosses.

It's the price of surviving a zone loss with workers in every zone. Keeping workers in the writer's zone would save it, but a zone failure would then take the workers and the writer at once.

R2.7 Trade-Offs

Pod size

Shopify's answer, in public: limits found by scale tests ("let's just keep throwing more checkouts at it until we find the limit"), extra-large merchants on their own pods, and continuous rebalancing (2022 talk).

Queue fairness

Gate vs database checks

A watch-list note for translations. If you move holds to DynamoDB, don't rely on TTL to release them: TTL deletes items only eventually (typically within a few days), so an expired hold must be ignored by a condition on expires_at. DynamoDB conditions have no server time, so the writer supplies now, and with global tables a condition checks only the local Region's copy.

R2.8 Failure Modes

R2.9 Production Gotchas

R2.10 Pillar Check

R2.11 Round 2 Rubric and Follow-Ups

What a senior (L6) answer adds over L5

• Explains why sharding without isolation lowers availability, and designs pods with one pod per unit of work.
• Chooses a routing table over a hash so tenants can be pinned and moved.
• Walks through an online move: copy, binlog tail, cutover under a lock, routing update, fence, prune, with a timing chain.
• Designs a stateless, fair queue at the edge and sizes the admission rate from the database's write budget.
• Separates a fast gate from the database's final decision, and fixes a hot row with buckets.
• Knows asynchronous replication loses acknowledged writes, and where that's harmless.

Follow-up questions

1. "Why did Shopify put the throttle in the load balancers instead of Rails?" Answer: turning a buyer away in Rails already costs a worker and often a database connection, which is the capacity we're protecting. The edge is event-driven Nginx that handles huge connection counts cheaply, and Lua let a small team change it without touching the monolith (Shopify, 2017).

2. "Your threshold controller overshoots. What happens?" Answer: the bucket still caps what reaches the pod, so overshoot means some admitted buyers are turned back to the queue page for a moment, not an overload. The controller only decides who is next; the bucket decides how many.

3. "Why not hold stock at add-to-cart?" Answer: carts are client-side and most never convert; holding at cart would lock stock for window shoppers and bots. Holding at checkout, with a short expiry, reserves stock only for buyers who've shown intent and passed the queue.

Interview gotchas from this round's wrong answers

Round 3 · Architect · "Era 3: Black Friday at Planet Scale, Resilient Payments"

~45 min · Principal (L7) · about 2018–today · BFCM 2025: $14.6B of sales, 489M requests/min at the edge and 117M at the app servers (published) · checkout keeps working while payment providers, pods or a region fail

R3.0 Where We Left Off

Round 2 in 60 seconds. "Sharding the database added capacity but not isolation, so Shopify built pods: each pod is a set of shops on a fully isolated set of datastores (MySQL, Redis, Memcached), while app servers, job workers and load balancers are shared. Every request or job touches exactly one pod. Sorting Hat, a Lua module in the OpenResty load balancers, looks up the shop in a routing table and adds the pod as a header. Because routing is a table, shops are pinned and moved: Ghostferry copies a shop's rows, tails the binlog, and cuts over under a per-shop writer lock in a few seconds. During flash sales, a leaky bucket at the edge admits checkouts at the rate the pod can take and parks everyone else on a cached queue page; a first-attempt timestamp in a signed cookie and a proportional controller make the line fair without any shared store. Stock sits behind a Redis gate that says 'no' cheaply, while the database's conditional updates and expiring holds decide every real unit. Open costs: checkout still calls outside payment providers that can be slow or down, nobody has proven which failures we survive, capacity for the biggest weekend is a guess, and a whole site can fail."

Architecture v2, compact

Synthesizing vector architecture diagram...

The diagram is Round 2 in one picture. Buyers pass through the edge, where the bot filter, Sorting Hat and the checkout throttle run, to the shared Rails workers. The workers use one pod at a time, each pod with its own MySQL, Redis gate and Memcached, and the Shop Mover moves shops between pods. So shops are isolated by pod, the crowd is admitted at the edge, and stock is decided by each pod's database. The payment providers are still a dependency that every pod shares and that can slow down, and that is where Round 3 starts.

Round 2 step summary

Open costs: slow providers, unknown failure modes, capacity by guesswork, and a single site that can still fail.

R3.1 The Scope Raise

Interviewer: "It's today. We run on Google Cloud. Black Friday Cyber Monday breaks our records every year, and our merchants plan their year around it. Checkout talks to our own payments stack in many countries and to many third-party providers. When one of them slows down, checkout must not slow down with it. We want proof, before the weekend, of which failures we survive. And a region can fail on Black Friday."

Scope change

R3.2 What Breaks in the Round 2 Design

R3.3 New Requirements and API Additions

A payment attempt as a resource. Shopify's payment service (2019) takes the idempotency key as an input of every mutation, "a 'first class citizen' of the API", and stores each incoming request "uniquely identified by the client and idempotency key", with a progress column of named steps. This request body (a data transfer object, DTO) is ours:

jsonCopy
{
  "client": "checkout",
  "idempotency_key": "01JBQ7D2M9X4K6T8V3P1R5S0NA",
  "shop_id": 812,
  "checkout_id": 55120931,
  "amount": { "value": "96.00", "currency": "USD" },
  "card_token": "cs_tok_7Hq2",
  "merchant_country": "US"
}

Synthesizing vector architecture diagram...

The diagram shows the states of one payment attempt. The attempt is CREATED when its key is claimed and the attempt row is committed. If the circuit is open, the request is never sent and the attempt is DECLINED. Otherwise the provider is called with the attempt's key (AUTH_REQUESTED), and the provider approves (AUTHORIZED: the amount is held on the card), declines (DECLINED), or the call times out or its reply is lost (UNKNOWN). An UNKNOWN attempt becomes AUTHORIZED when the resolver, a webhook or a report says it was approved, or DECLINED when the resolver finds no charge. An AUTHORIZED attempt ends as CAPTURED when the capture (collecting the held amount) is confirmed, or VOIDED when a void is confirmed because no order exists. Every arrow is a conditional update that names the state it leaves, so two paths that learn the same outcome (the job's reply and the provider's webhook) can't both apply it.

A circuit and bulkhead per provider and country. Shopify (2022) adds "the merchant's country code to the endpoint host and port to create a more fine-grained Semian identifier", so a local outage in one country doesn't block payments in others. This YAML is ours, modeled on Semian's parameters: the timeouts belong to the HTTP client, and the _s suffixes and the timeouts block are our notation, not Semian's names. The values are ours (p99 below means the latency that 99% of calls stay under):

yamlCopy
resource: "gateway.example-psp.com:443:CA"   # host:port:country
timeouts:
  open_s: 1          # Shopify's suggested starting point (2022)
  read_s: 5
circuit_breaker:
  error_threshold: 3                 # errors within error_threshold_timeout (defaults to error_timeout) that open the circuit
  error_timeout_s: 30                # how long it stays open; then the next real request is the half-open call
  half_open_resource_timeout_s: 3    # the half-open call's own timeout, above healthy p99
  success_threshold: 2               # consecutive successes to close again
bulkhead:
  quota: 0.33        # at most ceil(0.33 x workers on the host) in flight to this resource

A resiliency matrix per team. Shopify asks teams to write "a user-centric resiliency matrix, documenting the expected user experience under various scenarios" (2020). An example in that spirit (our rows):

A capacity plan per component (our shape; Shopify partitions "each component into a separate GCP project, which makes it a lot easier to think of quotas per every project", 2020):

yamlCopy
component: storefront-renderer
forecast_peak_rpm: 150000000
safety_margin: 0.5              # plan for 1.5x the forecast
failover: survive_one_region    # capacity in the survivors, not just in total
regions: [us-central, us-east, europe-west4]
quota_checked: 2026-09-15
scale_test_dates: [2026-05-12, 2026-07-14, 2026-09-15]

Recap

• A payment attempt is a resource with explicit states, changed only by conditional updates.
• Circuits and bulkheads are keyed by provider, endpoint and country.
• Failure behavior is written down per team, then tested.

R3.4 Design Evolution: Fail Fast, Prove It, and Survive the Weekend

Step 3.1: A Slow Payment Provider Keeps All the Workers Busy

Synthesizing vector architecture diagram...

The diagram shows one Semian circuit, which exists per process and per resource. It starts Closed, and calls go through. When error_threshold errors happen within the error window, it opens; while Open, calls raise an exception instantly and never reach the provider. After error_timeout passes, it becomes HalfOpen, and the next real request goes through as the half-open call. If that call fails or takes longer than half_open_resource_timeout, the circuit opens again; after success_threshold successes in a row, it closes. The circuit only reacts after errors have happened, so the bulkhead is what limits how many workers can be stuck on a slow provider while the circuit is still Closed.

Primitive: Circuit Breaker, Bulkhead and Fault-Tolerance Patterns · Drill: The Slow Recommendation Service That Took Down Checkout (answered here: why a slow dependency exhausts the shared workers, by Little's law, and why a tiny timeout on every call fails healthy calls where a breaker reacts to a sustained pattern) · Loop: Design a Payment Processing System (Round 2, step 2.4: a breaker per payment service provider, PSP)

Step 3.2: Nobody Has Tested Which Failures Checkout Survives

Synthesizing vector architecture diagram...

The diagram shows a loop with three kinds of work. The resiliency matrix writes down the expected behavior for each dependency that is slow or down. From the matrix come Toxiproxy tests that run in CI and game days that inject faults at scale. When a test finds a regression, or a game day shows that the real behavior differs from the model, the result is a fix to the system or an update to the matrix, and the loop starts again from the corrected matrix. So the matrix is the model, tests and game days check it against reality, and every disagreement changes either the system or the matrix.

Step 3.3: Capacity for the Biggest Weekend

Primitive: Cloud Disaster Recovery and Multi-Region Active-Active

Step 3.4: Payments Must Never Double-Charge, Whatever Fails

Synthesizing vector architecture diagram...

Loop primitive: Idempotency & Effectively-Once Processing (the sections The key record and the durable guard: claim, record, run, answer and Time: how long to remember, and the late original) · Primitive: Two-Phase Commit and Saga Orchestration · Drill: The Flight Booking That Charged Without a Seat (answered here: a failed compensation is retried idempotently from an outbox, then becomes an anomaly for people; and a provider can't join a two-phase commit, so compensation replaces it) · Loop: Design a Payment Processing System (Round 1, steps 1.2 and 1.4: unknowns and webhooks)

Step 3.5: Where Does It Run, and What If a Region Fails?

Synthesizing vector architecture diagram...

Loop primitive: Replication, Quorums & Read-Your-Writes (the sections When to say OK: sync, async, semi-sync, The leader crashes: elections and A network partition and two leaders) · Primitive: Cloud Disaster Recovery and Multi-Region Active-Active

Round 3 Step Summary

R3.5 Global Architecture

Synthesizing vector architecture diagram...

The diagram shows Round 3's global design with two regions. Buyers worldwide reach the CDN and edge, which sends them to the subgraph Region A or Region B. Each region runs the same parts: OpenResty with the bot filter, Sorting Hat and the throttle; the shared workers; the pods active in that region; and payment jobs with breakers and bulkheads. The pods active in A replicate asynchronously to replicas that sit in region B, and the pods active in B replicate to replicas in region A. Payment jobs in both regions call CardServer and its provider adapters, which reach the payment providers per country. So each region runs the pods active there and holds replicas of the other region's pods, and payment calls leave through circuits and bulkheads keyed by provider and country.

Synthesizing vector architecture diagram...

The diagram shows the readiness cycle as Shopify describes it, in order. A forecast with a margin leads to a capacity plan per component, which leads to quotas and reservations with the cloud provider. Scale tests with Genghis, including regional failovers, then check the load, and game days with Toxiproxy faults check the failures. The results go into the resiliency matrix and its runbooks. Close to the weekend, the rate of change slows down: nothing is frozen, but changes are fewer and smaller. The BFCM weekend itself runs with live monitoring, and its lessons (the dotted line) feed next year's forecast.

Trace: a payment when provider X's Canadian path is down (our design around the published pieces)

Synthesizing vector architecture diagram...

Our AWS translation (not Shopify's setup). Two or three Regions, each with CloudFront and AWS WAF at the edge, OpenResty on EKS behind a Network Load Balancer, and per-pod Aurora MySQL clusters; each pod as an Aurora Global Database whose secondary sits in the pod's paired Region (Aurora's managed failover to a secondary is itself an asynchronous-replication failover, so the RPO is its lag); ElastiCache in each Region; the routing table as a DynamoDB global table, written only from a single control-plane home Region because global tables resolve concurrent writes by last writer wins (of two conflicting writes, the one with the later timestamp is kept) and replicate in a few seconds; Route 53 Application Recovery Controller routing controls as the gated switch; CloudWatch alarms fed by probes running in other Regions. Before a failover, check the survivor's On-Demand vCPU quota.

Sources for this round

• Eskildsen, Building and Testing Resilient Ruby on Rails Applications, Shopify Engineering, January 2015: the dependency matrix; Toxiproxy; circuit breakers; Semian's bulkheads and the 0.001% reasoning.
• Neufeld, Shopify's Infrastructure Collaboration with Google, Shopify Engineering, March 2018: building its cloud with Google; GKE; over 50% of workloads migrated.
• Denis, A Pods Architecture To Allow Shopify To Scale, Shopify Engineering, March 2018: pod pairs of data centers; the Pod Mover; evacuating a data center pod by pod.
• Jefferson, Building Resilient GraphQL APIs Using Idempotency, Shopify Engineering, August 2019: the payment service; keys as first-class inputs; the lock and 409; incoming requests and recovery points.
• Polan, Your Circuit Breaker is Misconfigured, Shopify Engineering, February 2020: Semian's parameters; per-process state; 263% to 4%.
• Tang and Shatrov, Capacity Planning at Scale, Shopify Engineering, December 2020: forecasts, safety margins and failover buffers; a GCP project per component; scale-up tests since 2018; GKE capacity for test windows.
• McIlmoyl, Resiliency Planning for High-Traffic Events, Shopify Engineering, December 2020: the user-centric resiliency matrix; game days; slowing the rate of change; over $5.1 billion in BFCM 2020 sales.
• Vaillancourt, How Shopify Reduced Storefront Response Times with a Rewrite, Shopify Engineering, August 2020: the storefront renderer, separate from the monolith, reading from dedicated read replicas.
• Inch, Pummelling the Platform–Performance Testing Shopify, Shopify Engineering, December 2020: load and stress tests; Go and Lua flows; HAR-based tests.
• Leinwand, Shopify's cloud, load and modular code in 2022, Shopify Engineering, January 2022: BFCM 2021 peaks of 32M app-server and 34M load-balancer requests a minute; a 50.7M load test.
• de Water, 10 Tips for Building Resilient Payment Systems, Shopify Engineering, July 2022: timeouts; Semian and country identifiers; Little's law; idempotency; reconciliation and anomalies; the benchmark gateway.
• de Water, Shopify's Architecture to Handle the World's Biggest Flash Sales, QCon Plus talk, published by InfoQ in October 2022: CardSink and CardServer; 17 countries; pods in two regions; Genghis; Resumption.
• Petroski and Frail, How we prepare Shopify for BFCM, Shopify Engineering, November 2025: 2024 records; capacity planning with cloud providers; game days; the Resiliency Matrix; five scale tests; regional failovers.
• Shopify BFCM announcements: 2021 (November 2021), 2022 (November 2022), 2023 (November 2023), 2024 (December 2024) and 2025 (December 2025); Shopify, Performance up, complexity down: killer updates from Shopify engineering, January 2024 (the 2023 core application server peak); @ShopifyEng on X, BFCM 2022 stats, November 2022 (75.98M requests a minute).
• Shopify open source: Semian and Toxiproxy (checked September 2026).

Everything else in this round (the attempt states, the resolver and void design, the worker and bulkhead numbers, the region sizing, the failover chain, the translation, and every number in R3.6 not marked published) is our design or our assumption.

R3.6 Numbers and Cost

BFCM, as published

The request figures don't all measure the same layer: the 2022 figure is "our commerce platform"; the 2023 figure is the "core application server", which we treat as the app-server layer, so a year-to-year line across them would mislead. The app-server series (2021, 2023, 2024, 2025) is consistent:

Synthesizing vector architecture diagram...

The chart shows the published BFCM peak at the app-server layer, in millions of requests per minute: 32 in 2021, 58 in 2023, 80 in 2024 and 117 in 2025. 2022 is left out because its published figure measures a different layer, "our commerce platform". From 2021 to 2025 the peak app-server rate grew about 3.7 times (117 ÷ 32), and from 2024 to 2025 by 1.46 times, faster than the 27% growth in sales that year. The capacity sizing in this section starts from the 2025 bar.

What 2025's peak means per second

Payments at the peak (assumptions unless marked)

Capacity for next year (assumptions)

The load test itself is cheap; the fleet it proves is not. Shopify's 2020 note on why the cloud helps: running on GKE "allowed us to grab extra compute capacity just for the window of the exercise, and only pay for those hours when we needed it".

A test at 146M requests a minute sits well above 2025's 117M app-server peak, but Shopify doesn't say which layer its test figures measure, so compare them with care. In 2021 the comparison is clean: a load-balancer test at 50.7M against a real load-balancer peak of 34M, about 1.5×.

Region failover (assumptions)

R3.7 Trade-Offs

A modular monolith with pods vs microservices (the choice from R1.8, revisited)

Shopify's stated view (2022 talk): the monolith won't be broken into microservices "anytime soon", because things "that pertain to a single shop, like an order or a chargeback or a refund, it's just easier to keep it in a monolith". It does split where the traffic profile differs: storefront rendering moved to a separate application, reading from "dedicated read replicas" (2020).

Failing fast vs retrying

Headroom

R3.8 Failure Modes

R3.9 Runbook and Incident Response

Golden signals OPS 8 · REL 6

Shopify's 2022 payments post lists the four golden signals and one nuance for payments: "we distinguish between payment failures and errors". A decline isn't an incident; a provider's 500 is.

Procedure: pod overload during a sale REL 5

1. Confirm it's one pod. Saturation on one pod's MySQL, normal elsewhere.
2. Find the shop. Top shops by writes and lock waits on that pod.
3. Throttle, don't block. Engage or tighten that shop's checkout throttle; the queue page absorbs the crowd.
4. Protect neighbors. Check the pod's bulkhead tickets are holding; shed low-priority jobs for the pod.
5. Afterwards, move. Schedule the shop onto a quieter or dedicated pod with the Shop Mover, outside the sale.

Procedure: tuning the waiting room COST 9

1. Read the budget. Pod write budget from the last load test, minus other shops' current load.
2. Set the rate. Admission = budget ÷ writes per purchase; divide by the number of load balancers.
3. Watch the controller. Admitted rate should track the bucket limit; big oscillations mean the controller is overcorrecting.
4. Watch fairness. Median and p95 queue times should stay close; a widening gap means buyers are skipping the line.
5. Keep the line after sellout so returned stock goes to the next buyers.

Procedure: provider degradation REL 11

1. Scope it. Which provider, which countries, errors or latency?
2. Check our side. Worker saturation, bulkhead tickets, open circuits: are we protecting ourselves?
3. Degrade visibly. Other payment methods shown; no re-routing of unknown attempts.
4. After recovery, drain UNKNOWN attempts through the resolver and check reconciliation anomalies.

Procedure: region evacuation REL 13 · OPS 10

1. Confirm from outside that the region, not a service, is down.
2. Check the survivors have capacity and quota for the extra pods, and hold those pods' data where it's allowed to live.
3. Flip the routing control for the region's pods and confirm each pod's epoch moved.
4. Watch checkout completion for the moved shops and the payment reconciliation queue.
5. Fence and rejoin the old region's primaries as replicas; reconcile their late writes before moving pods back one at a time.

Shopify's incident process (2022) names three roles: the Incident Manager on Call (IMOC), a Support Response Manager for public communication, and the service owners, with a retrospective "within a week after the incident occurred".

Go deeper: CLI and SQL checks (AWS translation; replace names with real ones)

bashCopy
# 1. Alarms currently firing for checkout and payments
aws cloudwatch describe-alarms --state-value ALARM --alarm-name-prefix checkout-

# 2. The state of a pod's stock-gate cache
aws elasticache describe-replication-groups --replication-group-id pod-017-gate

# 3. A pod's database cluster and its members
aws rds describe-db-clusters --db-cluster-identifier pod-017

# 4. Unplanned failover of a pod's global database to its paired Region (accepts the replication lag as data loss)
aws rds failover-global-cluster --global-cluster-identifier pod-017-global --target-db-cluster-identifier arn:aws:rds:us-east-2:111122223333:cluster:pod-017-use2 --allow-data-loss

# 5. Read, then flip, a Region's routing control (run against a cluster endpoint)
aws route53-recovery-cluster get-routing-control-state --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/abc/routingcontrol/def --region us-west-2 --endpoint-url https://host-aaa.us-west-2.example.com/v1
aws route53-recovery-cluster update-routing-control-state --routing-control-arn arn:aws:route53-recovery-control::111122223333:controlpanel/abc/routingcontrol/def --routing-control-state Off --region us-west-2 --endpoint-url https://host-aaa.us-west-2.example.com/v1

# 6. The vCPU quota for Running On-Demand Standard instances in the survivor Region
aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A --region us-east-2

# 7. Reserve capacity for the weekend in one AZ
aws ec2 create-capacity-reservation --instance-type c7g.4xlarge --instance-platform Linux/UNIX --availability-zone us-east-1a --instance-count 50 --end-date-type limited --end-date 2026-12-03T00:00:00Z

# 8. Which pod serves a shop (strongly consistent read in the table's home Region)
aws dynamodb get-item --table-name shop-routes --key '{"domain":{"S":"tshirthero.com"}}' --consistent-read

sqlCopy
-- 9. On a pod's MySQL: who is waiting for which lock
SELECT *
FROM performance_schema.data_lock_waits;

-- 10. On a MySQL replica: how far behind is it?
SHOW REPLICA STATUS\G

R3.10 Pillar Check

R3.11 Round 3 Rubric and Follow-Ups

What an architect (L7) answer adds over L6

• Computes, with Little's law, how one slow dependency saturates shared workers, and sizes bulkheads and breaker parameters from the numbers.
• Treats failure behavior as something to write down and test: matrices, failure injection, game days, a fake provider for load tests.
• Plans capacity for the failover case, not just for growth, and compares region layouts by headroom.
• Designs payments so every path that learns an outcome uses the same conditional transition, with one live attempt per checkout, voids driven from an outbox, and reconciliation against the provider.
• Designs a region failover end to end: outside detection, a gate, epochs, capacity and residency, a timing chain and a stated RPO with a repair source.
• Separates what Shopify published from what it designs, and reads published numbers for which layer they measure.

Follow-up questions

1. "Why not make the breaker's error_timeout 1 second so recovery is instant?" Answer: every open circuit then lets a real request through as the half-open call every second, and each can wait its full half-open timeout while the provider is still down: with error_timeout 1 s and h = 3 s, 3 ÷ 4 = 75% of a worker's time goes to half-open calls, each of them a real buyer's payment that fails. Shopify's own tuning went the other way, to 30 seconds, trading slower recovery for a 4% cost instead of 263%.

2. "The resolver and the webhook disagree: one says approved, one says declined." Answer: the first to apply its conditional transition wins, and the attempt is final; the disagreement becomes a reconciliation anomaly. A card outcome doesn't change once it is final, so a disagreement means a bug or a lookup of the wrong attempt, which is exactly what anomalies are for.

3. "Could every pod be active in two regions to avoid the 4-minute failover?" Answer: not for stock and orders: two writable copies can each sell the last unit, and last-writer-wins would silently drop an order. Active-active (every region accepting writes for the same data) fits data that's read-mostly or mergeable (storefront rendering from replicas, the routing table written from one home region). For checkout, one writer per pod with a fast, gated failover is the correct design, with a short lease so a region cut off from the control plane stops writing before the other one is promoted.

4. "Is Semian still how Shopify does this?" Answer: Semian has been in production at Shopify since October 2014 (README), and Shopify's 2022 payments post describes it protecting Net::HTTP, MySQL, Redis and gRPC. The repository is active, with adapters for MySQL, Redis, Net::HTTP and Active Record.

Interview gotchas from this round's wrong answers

Loop Closer: Interview Strategy for All Three Rounds

How to Run Each 60-Minute Round

For how to spend a single 45-minute round, see the 45-minute interview blueprint. For payments in depth, see the payment processing loop; for holds, flash sales and gate counters, the hotel reservation loop; for another company that outgrew its first database, the Uber case study.

The Two Sentences That Matter Most

• Opening any round: "This is a multi-tenant platform where one merchant's flash sale looks like an attack on everyone else, so I'll isolate tenants, admit the crowd at the rate the database can take, and let the database, never a cache, decide every unit of stock and every charge."
• When scale arrives: "I'll put every shop in exactly one pod, every outside call behind a breaker and a bulkhead, every payment outcome behind one conditional transition, and then prove each of those with failure injection and full-scale tests before the weekend that matters."

Well-Architected Review Sheet

Interviewers rarely ask "which pillar is this?". They ask the pillar's question in plain words. Rehearse one sentence per row.

Rubric Across Levels

Sources

All sources used on this page, oldest first.

• Shopify, 100,000 Online Stores Now Use Shopify, Shopify blog, 2014.
• Eskildsen, Building and Testing Resilient Ruby on Rails Applications, Shopify Engineering, January 2015.
• Stolarsky, Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers, Part I and Part II, Shopify Engineering, February 2017.
• Denis, A Pods Architecture To Allow Shopify To Scale, Shopify Engineering, March 2018.
• Neufeld, Shopify's Infrastructure Collaboration with Google, Shopify Engineering, March 2018.
• Westeinde, Deconstructing the Monolith, Shopify Engineering, February 2019.
• Jefferson, Building Resilient GraphQL APIs Using Idempotency, Shopify Engineering, August 2019.
• Polan, Your Circuit Breaker is Misconfigured, Shopify Engineering, February 2020.
• Müller, Under Deconstruction: The State of Shopify's Monolith, Shopify Engineering, September 2020.
• Vaillancourt, How Shopify Reduced Storefront Response Times with a Rewrite, August 2020.
• Inch, Pummelling the Platform–Performance Testing Shopify, Shopify Engineering, December 2020.
• Tang and Shatrov, Capacity Planning at Scale, Shopify Engineering, December 2020.
• McIlmoyl, Resiliency Planning for High-Traffic Events, Shopify Engineering, December 2020.
• Madan, Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale, Shopify Engineering, September 2021.
• Shopify, BFCM 2021 results, November 2021.
• Leinwand, Shopify's cloud, load and modular code in 2022, Shopify Engineering, January 2022.
• de Water, 10 Tips for Building Resilient Payment Systems, Shopify Engineering, July 2022.
• de Water, Shopify's Architecture to Handle the World's Biggest Flash Sales, QCon Plus talk, published by InfoQ in October 2022.
• Shopify, BFCM 2022 results, November 2022, and @ShopifyEng on X, BFCM 2022 stats, November 2022.
• Shopify, BFCM 2023 results, November 2023.
• Shopify, Performance up, complexity down: killer updates from Shopify engineering, January 2024.
• Shopify, BFCM 2024 results, December 2024.
• Petroski and Frail, How we prepare Shopify for BFCM, Shopify Engineering, November 2025.
• Shopify, BFCM 2025 press release, December 2025.
• Shopify open source: Semian, Toxiproxy and Ghostferry (checked September 2026).
• Shopify API reference, ProductVariantInventoryPolicy (DENY and CONTINUE).
• MySQL 8.4 Reference Manual, CHECK Constraints and Semisynchronous Replication.
• AWS, EC2 data transfer pricing (us-east-1), Aurora Global Database documentation and the AWS CLI reference (checked September 2026), for the translation's figures and commands.