CHEAT SHEET #05Quick Reference Architecture•5 min read
Designing Fault-Tolerant Distributed Systems Cheat Sheet
Comprehensive resiliency cheat sheet covering Circuit Breakers, Bulkheads, Retries with Jitter, Dead-Letter Queues, and Graceful Degradation.
FreeCheat SheetAWSArchitecture
Comprehensive resiliency cheat sheet covering Circuit Breakers, Bulkheads, Retries with Jitter, Dead-Letter Queues, and Graceful Degradation.
1. The 6 Resiliency Pillars
Interactive Architecture DiagramSynthesizing vector architecture diagram...
2. Resiliency Patterns Cheat Sheet
Pattern 1: Circuit Breaker State Machine
Interactive Architecture DiagramSynthesizing vector architecture diagram...
Pattern 2: Bulkhead Isolation
Isolate resources into discrete pools so that a failure in one non-critical domain cannot starve thread pools or memory in critical domains:
- Connection Pool Bulkhead: Separate database connection pools for read-heavy catalog searches vs. payment checkouts.
- Thread Pool Bulkhead: Isolate third-party webhook dispatchers from user-facing login threads.
Pattern 3: Exponential Backoff with Full Jitter
Never retry immediately on network failures; synchronized retries cause a "thundering herd" or "retry storm" that prevents struggling services from recovering.
sleep_duration = random_between(0, min(max_backoff, base * 2 ^ attempt_count))
Interactive Architecture DiagramSynthesizing vector architecture diagram...
3. Graceful Degradation Strategies
| Component Failure | Naive Failure Behavior | Graceful Degradation Strategy |
|---|---|---|
| Personalized Recommendations Down | Entire home page crashes with HTTP 500 error. | Fallback to cached static top-10 popular items list. |
| Real-Time Inventory Lock Service Down | User checkout fails immediately. | Accept order provisionally into an async reconciliation queue. |
| Search Autocomplete Inverted Index Down | Search bar input errors out. | Fallback to simple SQL LIKE 'query%' or client-side history. |
| Image Transcoding Worker Queue Jammed | User upload rejected. | Store raw original in S3; show placeholder while queue drains. |
4. Health Check Probing Matrix
| Probe Type | Frequency | Purpose | Action on Consecutive Failures |
|---|---|---|---|
| Liveness Probe | Every 5-10s | Checks if process is deadlocked or stuck in infinite loop. | Restart / kill container immediately. |
| Readiness Probe | Every 2-5s | Checks if service is warm and capable of accepting traffic. | Remove instance from Load Balancer target group. |
| Startup Probe | On Boot | Checks if slow initialization (JVM warmup, cache pre-fill) done. | Pause liveness/readiness evaluation until complete. |