The Banking Ledger That Took 3 Days to Replay
The Banking Ledger That Took 3 Days to Replay
Your neobank platform implements Event Sourcing with Command Query Responsibility Segregation (CQRS) for core account balances. Every deposit, withdrawal, fee, and transfer is appended as an immutable domain event (MoneyDeposited, TransferInitiated, FeeDeducted) to an event store table. The read side maintains a materialized projection table in PostgreSQL (account_balances) updated asynchronously by an event projector. After 4 years in production, the system contains 800 million events. High-volume business accounts have accumulated over 2.5 million individual transaction events each. A bug in a new tax withholding service corrupts the materialized account_balances projection table, requiring the ops team to drop and rebuild the read model by replaying the event log from offset 0. The replay process takes 78 hours, during which customers cannot view their current account balances on mobile apps. Furthermore, when high-activity accounts initiate new transfers, the Command Handler takes 12 seconds just to replay that account's 2.5 million historical events to compute whether the current balance can cover the withdrawal. You are tasked with designing a scalable snapshotting and projection rebuild architecture for event-sourced financial ledgers.
The Banking Ledger That Took 3 Days to Replay
Your neobank platform implements Event Sourcing with Command Query Responsibility Segregation (CQRS) for core account balances. Every deposit, withdrawal, fee, and transfer is appended as an immutable domain event (MoneyDeposited, TransferInitiated, FeeDeducted) to an event store table. The read side maintains a materialized projection table in PostgreSQL (account_balances) updated asynchronously by an event projector. After 4 years in production, the system contains 800 million events. High-volume business accounts have accumulated over 2.5 million individual transaction events each. A bug in a new tax withholding service corrupts the materialized account_balances projection table, requiring the ops team to drop and rebuild the read model by replaying the event log from offset 0. The replay process takes 78 hours, during which customers cannot view their current account balances on mobile apps. Furthermore, when high-activity accounts initiate new transfers, the Command Handler takes 12 seconds just to replay that account's 2.5 million historical events to compute whether the current balance can cover the withdrawal. You are tasked with designing a scalable snapshotting and projection rebuild architecture for event-sourced financial ledgers.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).