The Slow Recommendation Service That Took Down Checkout
The Slow Recommendation Service That Took Down Checkout
Your retail website's API Gateway serves 30,000 requests/second. On every product details and cart page request, the gateway coordinates calls to five backend microservices: Product Catalog, Pricing, Inventory, User Auth, and Personalized Recommendations ("Customers also bought"). The gateway's HTTP server pool maintains 1,000 worker threads. During a machine learning model update, the non-critical Recommendation Service suffers database contention: its response latency degrades from 25ms to 5,500ms (just under the gateway's 6-second default client read timeout). Even though recommendations are completely optional for rendering the page, gateway threads wait on the slow HTTP responses. Within 8 seconds, all 1,000 gateway worker threads become blocked waiting for recommendation replies. Ingress connection queues fill up, the gateway drops incoming TCP connections, and paying customers can no longer load cart or checkout pages — an outage across 100% of revenue traffic caused by a non-critical optional widget. You are asked to implement Circuit Breaker and Bulkhead isolation patterns to insulate core business services from downstream latency degradation.
The Slow Recommendation Service That Took Down Checkout
Your retail website's API Gateway serves 30,000 requests/second. On every product details and cart page request, the gateway coordinates calls to five backend microservices: Product Catalog, Pricing, Inventory, User Auth, and Personalized Recommendations ("Customers also bought"). The gateway's HTTP server pool maintains 1,000 worker threads. During a machine learning model update, the non-critical Recommendation Service suffers database contention: its response latency degrades from 25ms to 5,500ms (just under the gateway's 6-second default client read timeout). Even though recommendations are completely optional for rendering the page, gateway threads wait on the slow HTTP responses. Within 8 seconds, all 1,000 gateway worker threads become blocked waiting for recommendation replies. Ingress connection queues fill up, the gateway drops incoming TCP connections, and paying customers can no longer load cart or checkout pages — an outage across 100% of revenue traffic caused by a non-critical optional widget. You are asked to implement Circuit Breaker and Bulkhead isolation patterns to insulate core business services from downstream latency degradation.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).