The Log Ingestion That Admitted Ten Times the Plan
The Log Ingestion That Admitted Ten Times the Plan
This situation follows the lesson Rate-Limiting Algorithms. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Logly is a log-ingestion service. Customers run an agent on each of their servers, and the agent sends batches of log lines to POST /v1/ingest. The API runs on 16 ingest nodes, N1 to N16, behind a load balancer that spreads requests round-robin, so each node receives one sixteenth of the requests. The nodes pass admitted batches to one indexer, which can take 50 MB a second from all customers together. In this situation, 1 MB is 1,000,000 bytes and 1 KB is 1,000 bytes.
A customer's plan is a limit in bytes a second, and the cost of one request is the size of its body in bytes. The tenant t-42, a game studio, pays for 2 MB a second with a burst of 8 MB: the rate r is 2,000,000 bytes a second and the burst b is 8,000,000 bytes. t-42 runs 200 game servers. Each server sends one batch a second, one every 5 ms across the 200, and a batch is normally 5 KB, so t-42 normally sends 200 × 5 KB = 1 MB a second. The other customers send 45 MB a second in all, each within its own plan.
As shipped, every node keeps its own token bucket for t-42, configured with the full plan, 2 MB a second and 8 MB, in its own memory. No node can read another node's bucket. This situation calls that design separate local buckets, in the same way that the lesson Rate-Limiting Algorithms calls its design with one count per gateway separate local counters.
Two more things exist in t-42's traffic:
- A crash report is one batch of 5 MB, sent by a game server when it crashes. It is sent to the same route and charged to the same plan.
- The limiter's store is Amazon ElastiCache, reached in 1 ms from any node, with a check timeout of 3 ms, a circuit breaker that opens after 5 timeouts in a row and probes once a second, and a key TTL of 60 s.
Synthesizing vector architecture diagram...
t-42's 200 game servers send one batch a second each, and the load balancer spreads the batches round-robin over the 16 nodes. Every node has its own bucket for t-42, configured with the full plan of 2 MB a second and 8 MB, and every node passes what it admits to the one indexer, which also receives 45 MB a second from the other customers. The dashed store is where one shared bucket for t-42 would live. As shipped, that store does not exist, and the dashed arrows are the calls the nodes would make to it and do not make.The incident. At 20:00:00 a bug in the game's new build makes every game server log 20 times more. Each batch grows from 5 KB to 100 KB, still one a second per server, for 10 minutes. t-42 now offers 200 × 100 KB = 20 MB a second, ten times its plan's rate. At 20:03:00 one game server crashes and sends a crash report of 5 MB.
The proposal. A colleague proposes exact shares: each node enforces one sixteenth of the plan in its own memory, 2 MB ÷ 16 = 125 KB a second with a burst of 8 MB ÷ 16 = 500 KB, "so that the 16 nodes together can never admit more than the plan, and we need no store".
The outage. At 20:02:00, during the incident, the limiter store's primary fails. The promotion of its replica takes 20 s. Every node's breaker opens within about 0.1 s of the failure, so for 20 s no node can check any bucket.
Your task
Design t-42's limit for the 16 nodes, and its failure policy. A complete answer holds four things: how many megabytes the fleet admits from t-42 in the first second and in the 10 minutes of the incident, as shipped and under your design; why the colleague's exact shares break on a crash report; the Retry-After the design sends for a crash report that finds 1.2 MB in the bucket; and the megabytes t-42 gets in the 20 s outage under open, closed and a local share, with the policy you choose and why.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Log Ingestion That Admitted Ten Times the Plan
This situation follows the lesson Rate-Limiting Algorithms. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Logly is a log-ingestion service. Customers run an agent on each of their servers, and the agent sends batches of log lines to POST /v1/ingest. The API runs on 16 ingest nodes, N1 to N16, behind a load balancer that spreads requests round-robin, so each node receives one sixteenth of the requests. The nodes pass admitted batches to one indexer, which can take 50 MB a second from all customers together. In this situation, 1 MB is 1,000,000 bytes and 1 KB is 1,000 bytes.
A customer's plan is a limit in bytes a second, and the cost of one request is the size of its body in bytes. The tenant t-42, a game studio, pays for 2 MB a second with a burst of 8 MB: the rate r is 2,000,000 bytes a second and the burst b is 8,000,000 bytes. t-42 runs 200 game servers. Each server sends one batch a second, one every 5 ms across the 200, and a batch is normally 5 KB, so t-42 normally sends 200 × 5 KB = 1 MB a second. The other customers send 45 MB a second in all, each within its own plan.
As shipped, every node keeps its own token bucket for t-42, configured with the full plan, 2 MB a second and 8 MB, in its own memory. No node can read another node's bucket. This situation calls that design separate local buckets, in the same way that the lesson Rate-Limiting Algorithms calls its design with one count per gateway separate local counters.
Two more things exist in t-42's traffic:
- A crash report is one batch of 5 MB, sent by a game server when it crashes. It is sent to the same route and charged to the same plan.
- The limiter's store is Amazon ElastiCache, reached in 1 ms from any node, with a check timeout of 3 ms, a circuit breaker that opens after 5 timeouts in a row and probes once a second, and a key TTL of 60 s.
Synthesizing vector architecture diagram...
t-42's 200 game servers send one batch a second each, and the load balancer spreads the batches round-robin over the 16 nodes. Every node has its own bucket for t-42, configured with the full plan of 2 MB a second and 8 MB, and every node passes what it admits to the one indexer, which also receives 45 MB a second from the other customers. The dashed store is where one shared bucket for t-42 would live. As shipped, that store does not exist, and the dashed arrows are the calls the nodes would make to it and do not make.The incident. At 20:00:00 a bug in the game's new build makes every game server log 20 times more. Each batch grows from 5 KB to 100 KB, still one a second per server, for 10 minutes. t-42 now offers 200 × 100 KB = 20 MB a second, ten times its plan's rate. At 20:03:00 one game server crashes and sends a crash report of 5 MB.
The proposal. A colleague proposes exact shares: each node enforces one sixteenth of the plan in its own memory, 2 MB ÷ 16 = 125 KB a second with a burst of 8 MB ÷ 16 = 500 KB, "so that the 16 nodes together can never admit more than the plan, and we need no store".
The outage. At 20:02:00, during the incident, the limiter store's primary fails. The promotion of its replica takes 20 s. Every node's breaker opens within about 0.1 s of the failure, so for 20 s no node can check any bucket.
Your task
Design t-42's limit for the 16 nodes, and its failure policy. A complete answer holds four things: how many megabytes the fleet admits from t-42 in the first second and in the 10 minutes of the incident, as shipped and under your design; why the colleague's exact shares break on a crash report; the Retry-After the design sends for a crash report that finds 1.2 MB in the bucket; and the megabytes t-42 gets in the 20 s outage under open, closed and a local share, with the policy you choose and why.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).