The Site Creation Whose Orchestrator Died During the Domain Registration
The Site Creation Whose Orchestrator Died During the Domain Registration
This situation follows the lesson Distributed Transactions: Sagas vs Two-Phase Commit. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Hostwell is a web-hosting company. A customer creates a site in one request and chooses a domain name for it, for example sunny-bakery.com. The request carries a client idempotency key, and the sites service creates the site record site-9012 from it once. A site needs five things, and you decide the order:
- An IPv4 address from Hostwell's pool. The
networkservice owns a PostgreSQL database with the rowip_pool(region, available), one row per Region, and one row per address. Every site creation in a Region updates that Region'sip_poolrow, so it is the hot row of this scenario.allocate_iptakes an address (available− 1) in 10 ms;release_ipgives it back (available+ 1) in 10 ms. An allocated address is a hold: it expires 10 minutes after it is taken, by thenetworkdatabase's own clock, and a sweeper runs every 10 s and releases expired holds that are stillHELD. - A compute slot for the site's container. The
computeservice owns the rowcapacity(region, free_slots)and one row per slot.hold_slottakes a slot (free_slots− 1) in 10 ms;release_slotgives it back in 10 ms; the same 10-minute expiry and sweeper apply.provision_sitestarts the container on a held slot with the site's address in 1,500 ms;delete_siteremoves a provisioned site in 500 ms. - The domain name, registered at a registrar, a partner company reached over HTTP. These are assumptions about the registrar:
registertakes 2,000 ms and charges Hostwell $12; a registration is never refunded; the registrar has no idempotency keys; aregisterrequest for a name that is already registered, to anyone, Hostwell included, is refused after 200 ms with the one answerunavailable; a read-onlycheck("is this name free?") takes 300 ms; alookup("is this name registered to Hostwell's account?") takes 200 ms and answers either "registered to Hostwell" or "not yours". - A TLS certificate for the name, from a certificate authority (CA), a second partner. Issuing takes 4,000 ms and is free. Assumption: the CA accepts an idempotency token, and a repeated request with the same token within one hour returns the same certificate. A certificate can be requested only after the domain is registered.
- DNS records that point the name at the address. The
sitesservice writes them (publish_dns, 20 ms).sitesalso owns the site record, the saga log and an outbox for the welcome email.
The sites service runs the orchestrator, as in the lesson Distributed Transactions: Sagas vs Two-Phase Commit: it writes one saga-log record (4 ms) after every reply and before the next call, it owns each saga with a lease of 8 s, renewed on each log write and every 2 s, and it puts a fencing epoch on every log write. Timeouts: 10 s for the registrar, 10 s for the CA, 2 s for network and compute. The network adds nothing, so every time is exact arithmetic from these numbers. Times are seconds after the request arrives.
Failure rates (assumptions): the chosen name turns out to be already registered in 20% of requests, the Region's pool has no free address in 1%, and no slot is free in 0.1%.
Synthesizing vector architecture diagram...
sites, talks to every other box, and it keeps the only record of the saga: the saga log, the lease of 8 s and the epoch. Two of the boxes are Hostwell's own databases, network and compute, and each has one row per Region that every site creation updates, so each is a hot row. Two boxes are partners outside Hostwell: the registrar, whose register call costs $12, has no idempotency keys, and offers check and lookup as its other calls, and the certificate authority, which honours a token for one hour. The last box is the sites service's own database, where the DNS records, the site record and the outbox live. No arrow runs between the partners and the databases: only the orchestrator connects them.The failure. The orchestrator instance that owns site-9012 (epoch 1) sends the register request to the registrar at .350 s, right after the log write that renewed its lease, and dies at 1.000 s. The registrar registers the name at 2.350 s and charges $12. Its reply goes nowhere.
The proposal. A colleague proposes two-phase commit (XA) across network, compute and sites, with the registrar call inside the lock window, "so that all or nothing covers the domain too". Use the two-phase commit settings of the section Three databases, one decision of the lesson Distributed Transactions: Sagas vs Two-Phase Commit: 1 ms per round trip, 5 ms per forced write (each PREPARE TRANSACTION, the decision record and each COMMIT PREPARED), the coordinator's own record of its participants, 4 ms, before the first write, and the sites row write inside the window, 4 ms (our assumption). At a launch peak one Region receives 2 site creations a second.
Your task
Design Hostwell's site-creation saga and answer the colleague. A complete answer holds three things: the order of the steps with the pivot named, and the time at which the customer is told "your site is being created"; what the replacement orchestrator (epoch 2) does after the failure, and when that customer is told; and, for the proposal, how long the ip_pool row stays locked per site creation and how many site creations a second one Region's row can commit.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Site Creation Whose Orchestrator Died During the Domain Registration
This situation follows the lesson Distributed Transactions: Sagas vs Two-Phase Commit. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Hostwell is a web-hosting company. A customer creates a site in one request and chooses a domain name for it, for example sunny-bakery.com. The request carries a client idempotency key, and the sites service creates the site record site-9012 from it once. A site needs five things, and you decide the order:
- An IPv4 address from Hostwell's pool. The
networkservice owns a PostgreSQL database with the rowip_pool(region, available), one row per Region, and one row per address. Every site creation in a Region updates that Region'sip_poolrow, so it is the hot row of this scenario.allocate_iptakes an address (available− 1) in 10 ms;release_ipgives it back (available+ 1) in 10 ms. An allocated address is a hold: it expires 10 minutes after it is taken, by thenetworkdatabase's own clock, and a sweeper runs every 10 s and releases expired holds that are stillHELD. - A compute slot for the site's container. The
computeservice owns the rowcapacity(region, free_slots)and one row per slot.hold_slottakes a slot (free_slots− 1) in 10 ms;release_slotgives it back in 10 ms; the same 10-minute expiry and sweeper apply.provision_sitestarts the container on a held slot with the site's address in 1,500 ms;delete_siteremoves a provisioned site in 500 ms. - The domain name, registered at a registrar, a partner company reached over HTTP. These are assumptions about the registrar:
registertakes 2,000 ms and charges Hostwell $12; a registration is never refunded; the registrar has no idempotency keys; aregisterrequest for a name that is already registered, to anyone, Hostwell included, is refused after 200 ms with the one answerunavailable; a read-onlycheck("is this name free?") takes 300 ms; alookup("is this name registered to Hostwell's account?") takes 200 ms and answers either "registered to Hostwell" or "not yours". - A TLS certificate for the name, from a certificate authority (CA), a second partner. Issuing takes 4,000 ms and is free. Assumption: the CA accepts an idempotency token, and a repeated request with the same token within one hour returns the same certificate. A certificate can be requested only after the domain is registered.
- DNS records that point the name at the address. The
sitesservice writes them (publish_dns, 20 ms).sitesalso owns the site record, the saga log and an outbox for the welcome email.
The sites service runs the orchestrator, as in the lesson Distributed Transactions: Sagas vs Two-Phase Commit: it writes one saga-log record (4 ms) after every reply and before the next call, it owns each saga with a lease of 8 s, renewed on each log write and every 2 s, and it puts a fencing epoch on every log write. Timeouts: 10 s for the registrar, 10 s for the CA, 2 s for network and compute. The network adds nothing, so every time is exact arithmetic from these numbers. Times are seconds after the request arrives.
Failure rates (assumptions): the chosen name turns out to be already registered in 20% of requests, the Region's pool has no free address in 1%, and no slot is free in 0.1%.
Synthesizing vector architecture diagram...
sites, talks to every other box, and it keeps the only record of the saga: the saga log, the lease of 8 s and the epoch. Two of the boxes are Hostwell's own databases, network and compute, and each has one row per Region that every site creation updates, so each is a hot row. Two boxes are partners outside Hostwell: the registrar, whose register call costs $12, has no idempotency keys, and offers check and lookup as its other calls, and the certificate authority, which honours a token for one hour. The last box is the sites service's own database, where the DNS records, the site record and the outbox live. No arrow runs between the partners and the databases: only the orchestrator connects them.The failure. The orchestrator instance that owns site-9012 (epoch 1) sends the register request to the registrar at .350 s, right after the log write that renewed its lease, and dies at 1.000 s. The registrar registers the name at 2.350 s and charges $12. Its reply goes nowhere.
The proposal. A colleague proposes two-phase commit (XA) across network, compute and sites, with the registrar call inside the lock window, "so that all or nothing covers the domain too". Use the two-phase commit settings of the section Three databases, one decision of the lesson Distributed Transactions: Sagas vs Two-Phase Commit: 1 ms per round trip, 5 ms per forced write (each PREPARE TRANSACTION, the decision record and each COMMIT PREPARED), the coordinator's own record of its participants, 4 ms, before the first write, and the sites row write inside the window, 4 ms (our assumption). At a launch peak one Region receives 2 site creations a second.
Your task
Design Hostwell's site-creation saga and answer the colleague. A complete answer holds three things: the order of the steps with the pivot named, and the time at which the customer is told "your site is being created"; what the replacement orchestrator (epoch 2) does after the failure, and when that customer is told; and, for the proposal, how long the ip_pool row stays locked per site creation and how many site creations a second one Region's row can commit.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).