Reliability Pillar
The ability of a workload to perform its intended function correctly and consistently when it's expected to, including operating and testing the workload through its total lifecycle.
Official Questions
Best Practices
High Risk if Missing
AWS Verbatim
Cell-based architecture, shuffle sharding, circuit breakers, exponential backoff with full jitter, end-to-end idempotency, and disaster recovery (RPO/RTO).
In AWS Well-Architected, questions do not have single-choice trick answers. Every listed Best Practice represents an official architectural answer you must incorporate into your workload. When asked these questions in an Amazon System Design interview, your score is evaluated by how many of these best practices you proactively articulate and defend with trade-offs.
Verbatim Questions & Architecture Answers
How do you manage service quotas and constraints?
Maintain awareness of default quotas, manage constraints across accounts and regions, and accommodate fixed limits through architecture.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Audit default quotas for network throughput, IOPS, and API rate limits across all dependencies.
Ensure quota increases are replicated across all production accounts and failover regions.
Design systems using cellular architecture and partitioning when hard unchangeable limits are reached.
Proactively track usage against thresholds using AWS Service Quotas alerts.
Use Service Quotas APIs to programmatically request increases prior to anticipated traffic spikes.
Verify surviving AZs or Regions have sufficient quota to absorb 100% of shifted failover load.
Pre-allocating high quotas avoids abrupt 429 throttling during traffic surges but does not replace client-side backpressure.
Requesting higher soft quotas is free in AWS, but over-provisioning reserved concurrency (e.g. Lambda) incurs idle cost.
Cellular partitioning around fixed limits introduces routing complexity but permanently bounds blast radius.
How do you plan your network topology?
Design resilient, multi-AZ network foundations with redundant connectivity, private subnets, and non-overlapping IP address spaces.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Use redundant Direct Connect connections or multi-tunnel IPsec VPNs.
Avoid single points of failure in hybrid cloud gateway routing.
Allocate sufficiently sized CIDR blocks (/16) to prevent IP exhaustion during auto-scaling.
Keep databases and compute isolated from public internet exposure across 3+ Availability Zones.
Ensure VPC CIDRs do not conflict across interconnected accounts and Transit Gateways.
Multi-AZ placement ensures fault tolerance but introduces 1-2ms inter-AZ latency for synchronous cross-AZ calls.
Cross-AZ traffic costs $0.01/GB each way ($0.02/GB total). Design for AZ affinity where possible.
Transit Gateway simplifies hub-and-spoke routing across hundreds of VPCs, eliminating complex VPC peering meshes.
How do you design your workload service architecture?
Build microservices focused on specific business domains, establish explicit contracts, and enforce stateless compute tiers.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Segment into decoupled microservices or service-oriented architectures based on domain boundaries.
Adhere to Single Responsibility and Domain-Driven Design (DDD) to isolate change impact.
Expose explicit, backward-compatible REST or gRPC contracts to decouple clients from implementations.
Microservices introduce network RPC latency overhead compared to monolith in-process method calls.
Separate compute fleets per microservice can have higher baseline idle cost without serverless scale-to-zero.
Microservices allow independent deployments and localized failure domains, dramatically accelerating team velocity.
How do you design interactions in a distributed system to prevent failures?
Prevent cascading failures using loose coupling, asynchronous queues, constant work design, and idempotent operations.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Distinguish between hard real-time synchronous RPC and soft/asynchronous dependencies.
Use queues (Amazon SQS) and event streams (Kinesis) to buffer and isolate components.
Design systems that perform uniform work regardless of traffic state, preventing sudden CPU spikes.
Ensure retries with the same idempotency token produce identical results without side effects.
Asynchronous queuing converts synchronous latency (blocking) into immediate 202 Accepted (<30ms).
Requires queuing infrastructure (SQS/Kinesis) and worker pools, but prevents expensive cascading outages.
Idempotency keys require a fast deduplication store (DynamoDB conditional writes with TTL) but eliminate double-charge bugs.
How do you design interactions in a distributed system to mitigate or withstand failures?
Implement circuit breakers, rate limiting, exponential backoff with full jitter, deadlines, and stateless services.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Fallback to cached or simplified data when non-critical downstream dependencies fail.
Use token bucket rate limiters at the API Gateway to prevent overload and shed excess load.
Apply exponential backoff randomized uniformly between 0 and backoff cap to prevent thundering herds.
Use circuit breakers and bounded queue lengths to drop stale requests rather than queueing indefinitely.
Propagate end-to-end deadlines so downstream services discard expired requests immediately.
Store session state in external distributed caches (ElastiCache) so any instance can handle any request.
Circuit breakers fail fast in <1ms instead of tying up threads waiting for 30-second socket timeouts.
Prevents runaway compute scaling triggered by retrying against a dying backend service.
Requires careful tuning of timeout thresholds and circuit breaker failure ratios.
How do you monitor workload resources?
Continuously observe infrastructure, compute saturation, network metrics, and user-facing KPIs with automated alerting.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Track CPU, memory, disk I/O, network bandwidth, and queue depths across all tiers.
Measure real business impact (checkout completion rate, error budget burn rate).
Trigger auto-healing actions via CloudWatch Alarms and EventBridge.
Telemetry agents (CloudWatch EMF, OpenTelemetry) consume <1% CPU when configured with asynchronous batching.
High-resolution custom metrics and detailed logging cost money; filter and sample non-critical events.
Dramatically reduces Mean Time to Detect (MTTD) from hours to seconds.
How do you design your workload to adapt to changes in demand?
Scale dynamically to meet unexpected spikes without over-provisioning through predictive and reactive auto-scaling.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Target tracking policies on CPU or request count to add instances smoothly.
Provision all resources via Infrastructure as Code (AWS CDK / CloudFormation).
Ensure no shared state or localized file storage prevents adding more nodes.
Schedule capacity warm-ups in advance of marketing campaigns or flash sales.
Auto-scaling prevents queue buildup and latency degradation during demand surges.
Auto-scaling down prevents paying for idle compute during off-peak hours.
Eliminates manual operator intervention during unexpected traffic spikes.
How do you implement change?
Deploy changes using automated CI/CD pipelines, canary rollouts, and automated rollback alarms.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Eliminate manual production deployments through automated CI/CD stages.
Validate integration, performance, and contract tests in identical staging environments.
Expose changes to 10% of traffic first and monitor automated rollback alarms.
Canary rollouts ensure regressions only impact a tiny slice of users while metrics are evaluated.
Blue/Green requires duplicate compute fleets during cutover; Canary deployment minimizes extra compute cost.
Automated rollbacks prevent outages when faulty code passes unit testing.
How do you back up data?
Perform automated, encrypted backups of critical data stores and regularly test restoration procedures.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Ensure databases, configuration files, and state stores are systematically captured.
Enforce KMS encryption and cross-account backup vault isolation against ransomware.
Use AWS Backup and continuous Point-in-Time Recovery (PITR) in DynamoDB and Aurora.
Regularly execute restoration drills to prove recovery time objectives (RTO).
Continuous PITR in modern cloud databases (Aurora, DynamoDB) has zero performance impact on live OLTP queries.
Backup retention policies (pruning snapshots older than 30 days) save substantial storage fees.
Centralized AWS Backup policies eliminate bespoke cron jobs and backup scripts.
How do you use fault isolation to protect your workload?
Limit blast radius using Multi-AZ deployments, Cell-Based Architectures, and Shuffle Sharding.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Distribute stateless compute and database replicas across 3 independent AZs.
Ensure secondary failover region is geographically separated from primary.
Partition fleets into independent, complete miniature copies (cells) to bound blast radius.
Assign customers to overlapping subsets of nodes so single poison pill queries affect <0.01% of users.
Cellular routing adds a lightweight proxy routing hop (<2ms) while keeping internal cell traffic local.
Running multiple cells requires minimal extra cost if compute scales on-demand per cell.
Eliminates whole-platform outages; troubleshooting is scoped to a single isolated cell.
How do you design your workload to withstand component failures?
Automate failure detection, enable self-healing node replacement, and handle graceful failovers.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Use health checks (ALB / ECS) to automatically drain and replace faulty tasks.
Configure multi-master or automated read replica promotion in databases.
Return partial responses or cached defaults if a secondary recommendation engine crashes.
Fast failure detection (3 missed heartbeats in 15s) minimizes client-visible disruption.
N+1 redundancy in compute fleets ensures capacity during rolling node failures without 2x over-provisioning.
Self-healing architectures prevent nighttime on-call pages for transient single-node crashes.
How do you test reliability?
Validate resilience under chaos experiments, simulate network latency, and conduct GameDays.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Standardize runbooks for incident triage and root cause investigation.
Conduct Amazon 5-Whys Correction of Errors reviews to eliminate systemic root causes.
Simulate regional failover under live traffic during scheduled business hours.
Inject synthetic CPU spikes, packet drops, and AZ outages to verify automated recovery.
Identifies hidden concurrency bottlenecks before real customers experience outages.
Controlled chaos testing saves millions in lost revenue by preventing major Prime Day outages.
Transforms theoretical disaster recovery plans into battle-tested automated systems.
How do you plan for disaster recovery (DR)?
Define quantitative RTO and RPO objectives and select matching Backup/Restore, Pilot Light, Warm Standby, or Active-Active strategies.
Recommended Answers (Official AWS Best Practices Checklist)
Every best practice below is an official answer to incorporate into your system design:
Quantify acceptable downtime (RTO) and acceptable data loss (RPO) based on business SLAs.
Match architecture to objectives: Backup & Restore ($), Pilot Light ($$), Warm Standby ($$$), Active-Active ($$$$$).
Design reverse data synchronization to return traffic to primary region once recovered.
Active-Active provides near-zero RTO but requires asynchronous replication and conflict resolution (LWW).
Active-Active costs 2.5x-3x more in idle infrastructure and cross-region egress; Pilot Light provides high resilience at ~1.2x cost.
Pilot Light is far easier to operate and test without split-brain reconciliation risks.