The Gateway Restart That DDOSed the Chat Fleet
The Gateway Restart That DDOSed the Chat Fleet
Your enterprise collaboration platform powers real-time messaging for 5 million concurrent users. Edge gateway servers terminate long-lived WebSocket connections, mapping each active user to their open TCP socket and subscribing to their personal user ID channel via a Redis Pub/Sub backplane. During a scheduled zero-downtime rolling deployment of the ingress load balancer fleet, 1.5 million active WebSocket connections are terminated over a 20-second window as nodes cycle. Client frontend apps immediately fire reconnection loops with 0ms delay. The resulting wave of 1.5 million simultaneous TLS handshakes, JWT authentication database queries, and Redis SUBSCRIBE commands saturates the gateway CPU to 100%, exhausts Linux ephemeral socket ports, and crashes the session authorization database. When gateway pods restart, the clients immediately retry, creating a self-inflicted Distributed Denial of Service (DDoS) thundering herd that keeps the platform down for 45 minutes. You are tasked with designing a connection lifecycle architecture that survives mass disconnects, smooths reconnection waves, and minimizes backend auth storms.
The Gateway Restart That DDOSed the Chat Fleet
Your enterprise collaboration platform powers real-time messaging for 5 million concurrent users. Edge gateway servers terminate long-lived WebSocket connections, mapping each active user to their open TCP socket and subscribing to their personal user ID channel via a Redis Pub/Sub backplane. During a scheduled zero-downtime rolling deployment of the ingress load balancer fleet, 1.5 million active WebSocket connections are terminated over a 20-second window as nodes cycle. Client frontend apps immediately fire reconnection loops with 0ms delay. The resulting wave of 1.5 million simultaneous TLS handshakes, JWT authentication database queries, and Redis SUBSCRIBE commands saturates the gateway CPU to 100%, exhausts Linux ephemeral socket ports, and crashes the session authorization database. When gateway pods restart, the clients immediately retry, creating a self-inflicted Distributed Denial of Service (DDoS) thundering herd that keeps the platform down for 45 minutes. You are tasked with designing a connection lifecycle architecture that survives mass disconnects, smooths reconnection waves, and minimizes backend auth storms.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).