The Removed User Who Kept Access for an Hour
The Removed User Who Kept Access for an Hour
This situation follows the lesson Caching & Invalidation. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Docsly is a document-sharing service for companies. Every request to open, list or download a document starts with a permission check: the API server reads the user's membership record (the groups the user belongs to, the user's status, and a version number that the database raises by 1 on every change to the record) and compares it with the document's list of allowed groups. The document's list travels with the document and is not part of this problem. Only the membership record is cached.
The record is cached in two places. Each API server keeps an in-process copy (a copy inside the server's own memory) of every record it has read. Behind the servers, a Valkey cluster (Valkey: an in-memory key-value store, the open-source fork of Redis) holds member:{user} for every user who was read recently. On a miss in both, the server reads the membership database and stores the answer in Valkey and in its own memory.
| Setting | Value |
|---|---|
| Users | 400,000 accounts. 60,000 users are active at any moment (assumption). An active user's apps make on average 0.5 permission checks a second, so the service answers 30,000 checks a second |
| Dana | A contractor who is downloading reports: 2 checks a second (assumption) |
| API servers | 40. The load balancer sends each check to any server. Each server keeps its in-process copy of a record for 60 s (as shipped) |
| Valkey | member:{user} = {groups, status, version}, TTL 3,600 s (as shipped). A miss reads the read replica and stores the answer with a plain SET |
| Database | PostgreSQL. A primary for writes. A read replica 200 ms behind the primary (assumption) answers fills, at most 500 membership queries a second (assumption) |
| A change | The admin tool commits on the primary and receives the record's new version. 5 ms later it runs DEL member:{user} on Valkey (as shipped). Nothing is retried if the DEL fails. Nothing tells the API servers. About 200 changes an hour (assumption) |
| A change-stream invalidator | Available but not used as shipped: a process that reads the primary's change stream can write to Valkey about 50 ms after a commit (assumption) |
Synthesizing vector architecture diagram...
DEL to Valkey, drawn dashed because it can be lost. The crossed dashed line from the primary to the API servers is the message nobody sends: no component tells the 40 servers that a record changed, so each server's copy becomes fresh only when its 60 s run out.What happened. On Thursday at 09:00:00.000 an administrator removes Dana from the group "finance". The commit returns version 12 of her record, and the DEL reaches Valkey at 09:00:00.005. At 09:00:02 the administrator opens the "verify access" screen for Dana and sees her still in "finance". Dana's app keeps opening finance reports: the last check that was still allowed came 56 s after the removal (the reference run; 52 s with a second seed). On another day the DEL timed out, and a removed user kept access for most of an hour.
The changed requirement. The security team now requires three things: a removal takes effect on every API server within 5 s of the commit; the administrator's "verify access" screen shows the new record at once; and the read replica stays below its 500 queries a second.
Your task
Design the caching of the membership record so that a removal takes effect within 5 s everywhere, the administrator sees it at once, and the replica stays below 500 queries a second. A complete answer has the worst-case staleness as shipped with its arithmetic, the TTL you choose for each copy and why, the checks a second that reach Valkey and the queries a second that reach the replica, what a change writes to Valkey and what a fill does, how the administrator's screen reads, and one failure the design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Removed User Who Kept Access for an Hour
This situation follows the lesson Caching & Invalidation. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Docsly is a document-sharing service for companies. Every request to open, list or download a document starts with a permission check: the API server reads the user's membership record (the groups the user belongs to, the user's status, and a version number that the database raises by 1 on every change to the record) and compares it with the document's list of allowed groups. The document's list travels with the document and is not part of this problem. Only the membership record is cached.
The record is cached in two places. Each API server keeps an in-process copy (a copy inside the server's own memory) of every record it has read. Behind the servers, a Valkey cluster (Valkey: an in-memory key-value store, the open-source fork of Redis) holds member:{user} for every user who was read recently. On a miss in both, the server reads the membership database and stores the answer in Valkey and in its own memory.
| Setting | Value |
|---|---|
| Users | 400,000 accounts. 60,000 users are active at any moment (assumption). An active user's apps make on average 0.5 permission checks a second, so the service answers 30,000 checks a second |
| Dana | A contractor who is downloading reports: 2 checks a second (assumption) |
| API servers | 40. The load balancer sends each check to any server. Each server keeps its in-process copy of a record for 60 s (as shipped) |
| Valkey | member:{user} = {groups, status, version}, TTL 3,600 s (as shipped). A miss reads the read replica and stores the answer with a plain SET |
| Database | PostgreSQL. A primary for writes. A read replica 200 ms behind the primary (assumption) answers fills, at most 500 membership queries a second (assumption) |
| A change | The admin tool commits on the primary and receives the record's new version. 5 ms later it runs DEL member:{user} on Valkey (as shipped). Nothing is retried if the DEL fails. Nothing tells the API servers. About 200 changes an hour (assumption) |
| A change-stream invalidator | Available but not used as shipped: a process that reads the primary's change stream can write to Valkey about 50 ms after a commit (assumption) |
Synthesizing vector architecture diagram...
DEL to Valkey, drawn dashed because it can be lost. The crossed dashed line from the primary to the API servers is the message nobody sends: no component tells the 40 servers that a record changed, so each server's copy becomes fresh only when its 60 s run out.What happened. On Thursday at 09:00:00.000 an administrator removes Dana from the group "finance". The commit returns version 12 of her record, and the DEL reaches Valkey at 09:00:00.005. At 09:00:02 the administrator opens the "verify access" screen for Dana and sees her still in "finance". Dana's app keeps opening finance reports: the last check that was still allowed came 56 s after the removal (the reference run; 52 s with a second seed). On another day the DEL timed out, and a removed user kept access for most of an hour.
The changed requirement. The security team now requires three things: a removal takes effect on every API server within 5 s of the commit; the administrator's "verify access" screen shows the new record at once; and the read replica stays below its 500 queries a second.
Your task
Design the caching of the membership record so that a removal takes effect within 5 s everywhere, the administrator sees it at once, and the replica stays below 500 queries a second. A complete answer has the worst-case staleness as shipped with its arithmetic, the TTL you choose for each copy and why, the checks a second that reach Valkey and the queries a second that reach the replica, what a change writes to Valkey and what a fill does, how the administrator's screen reads, and one failure the design still has.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).