The Exam Journal That Lost 2,850 Saved Answers
The Exam Journal That Lost 2,850 Saved Answers
This situation follows the lesson Write-Ahead Log, fsync & Group Commit. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Examly runs online exams for universities. During an exam, each student's browser sends every answer to Examly's servers as a submission: a record of about 200 bytes that holds the student id, the question id, the answer and the time. The browser shows "Saved" when the server replies. A student who sees "Saved" moves to the next question and never sends that answer again.
All submissions go through one journal server: an EC2 instance (a virtual machine on AWS) that appends every submission to one append-only file, the journal. The journal sits on one EBS gp3 volume (AWS's general-purpose SSD block volume, attached to the instance over the network). Grading workers read finished answers back from the same volume.
How the journal server works today:
- It writes each submission's record into the journal file with
write(). That call puts the bytes in the kernel's page cache and returns. - It replies "Saved" at once.
- A background timer calls
fdatasyncon the journal file every 250 ms.
The settings, with every value the problem needs:
| Setting | Value | Where it comes from |
|---|---|---|
| Submissions at the peak | 15,000 a second | Measured in the last exam session |
| Record size | 200 bytes of payload plus a 7-byte header (checksum, length, type) = 207 bytes | The header the lesson uses |
| Sync latency of the volume | 2.0 ms per fdatasync | Measured on this instance and volume with fio --fdatasync=1. An assumption for this exercise |
| Grading workers' reads | 1,200 reads a second, 16 KiB each, from the same volume | An assumption |
| EBS gp3 volume | 3,000 IOPS (I/O operations a second) and 125 MiB/s baseline. An I/O of up to 256 KiB counts as one | AWS documentation |
| Kernel page size | 4 KiB. write() dirties whole pages | As in the lesson |
| Timer sync | Every 250 ms | The current design |
| Writers | Thousands of browsers send at the same time | Many submissions in flight |
Synthesizing vector architecture diagram...
The failure. During an exam session, the rack lost power. When the journal server restarted, the recovery scan found that the journal on the disk ended 190 ms before the power cut. The last timer sync had run 190 ms before the cut, and nothing written after it had reached the disk. Every submission in those 190 ms had been answered "Saved". 15,000 a second × 0.190 s = 2,850 submissions were lost, and their students did not resend them, because their screens said "Saved".
The new requirement. The exam board now requires two things. First, "Saved" must mean that the submission survives a power loss. Second, at the peak, the time from a submission reaching the journal server to its "Saved" must stay at most 5 ms; above that the browser shows a spinner (a turning icon that means waiting). The team wants to keep the gp3 volume at its baseline (3,000 IOPS, 125 MiB/s). One team member proposes: "Open the journal with O_DIRECT, so each write() goes to the disk and not to the page cache, and keep replying at once."
Your task
Redesign the journal server's write path so that "Saved" survives a power loss inside the 5 ms rule. A complete answer gives: what happens before "Saved", in order; the batch size, the syncs a second, and the best, average and worst wait at 15,000 submissions a second; whether the gp3 baseline holds together with the grading reads; what a browser must do when it gets no "Saved"; and the one failure "Saved" still does not survive.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
The Exam Journal That Lost 2,850 Saved Answers
This situation follows the lesson Write-Ahead Log, fsync & Group Commit. Solve it on paper before you open a hint. The Teacher Gold Answer holds the worked solution, the other valid approaches and what a strong answer at each level demonstrates.
The scenario
Examly runs online exams for universities. During an exam, each student's browser sends every answer to Examly's servers as a submission: a record of about 200 bytes that holds the student id, the question id, the answer and the time. The browser shows "Saved" when the server replies. A student who sees "Saved" moves to the next question and never sends that answer again.
All submissions go through one journal server: an EC2 instance (a virtual machine on AWS) that appends every submission to one append-only file, the journal. The journal sits on one EBS gp3 volume (AWS's general-purpose SSD block volume, attached to the instance over the network). Grading workers read finished answers back from the same volume.
How the journal server works today:
- It writes each submission's record into the journal file with
write(). That call puts the bytes in the kernel's page cache and returns. - It replies "Saved" at once.
- A background timer calls
fdatasyncon the journal file every 250 ms.
The settings, with every value the problem needs:
| Setting | Value | Where it comes from |
|---|---|---|
| Submissions at the peak | 15,000 a second | Measured in the last exam session |
| Record size | 200 bytes of payload plus a 7-byte header (checksum, length, type) = 207 bytes | The header the lesson uses |
| Sync latency of the volume | 2.0 ms per fdatasync | Measured on this instance and volume with fio --fdatasync=1. An assumption for this exercise |
| Grading workers' reads | 1,200 reads a second, 16 KiB each, from the same volume | An assumption |
| EBS gp3 volume | 3,000 IOPS (I/O operations a second) and 125 MiB/s baseline. An I/O of up to 256 KiB counts as one | AWS documentation |
| Kernel page size | 4 KiB. write() dirties whole pages | As in the lesson |
| Timer sync | Every 250 ms | The current design |
| Writers | Thousands of browsers send at the same time | Many submissions in flight |
Synthesizing vector architecture diagram...
The failure. During an exam session, the rack lost power. When the journal server restarted, the recovery scan found that the journal on the disk ended 190 ms before the power cut. The last timer sync had run 190 ms before the cut, and nothing written after it had reached the disk. Every submission in those 190 ms had been answered "Saved". 15,000 a second × 0.190 s = 2,850 submissions were lost, and their students did not resend them, because their screens said "Saved".
The new requirement. The exam board now requires two things. First, "Saved" must mean that the submission survives a power loss. Second, at the peak, the time from a submission reaching the journal server to its "Saved" must stay at most 5 ms; above that the browser shows a spinner (a turning icon that means waiting). The team wants to keep the gp3 volume at its baseline (3,000 IOPS, 125 MiB/s). One team member proposes: "Open the journal with O_DIRECT, so each write() goes to the disk and not to the page cache, and keep replying at once."
Your task
Redesign the journal server's write path so that "Saved" survives a power loss inside the 5 ms rule. A complete answer gives: what happens before "Saved", in order; the batch size, the syncs a second, and the best, average and worst wait at 15,000 submissions a second; whether the gp3 baseline holds together with the grading reads; what a browser must do when it gets no "Saved"; and the one failure "Saved" still does not survive.
Hints
Open a hint only when you are stuck. Each hint gives a little more.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).