Design a Distributed Job Scheduler & Task Execution Queue

1. Problem Statement & Scope

System Mission

Design a distributed, highly available, fault-tolerant Job Scheduling and Task Execution platform (similar to Temporal, Quartz, or AWS EventBridge Scheduler) capable of scheduling and executing billions of delayed, recurring (cron), and one-off tasks with exact-once semantics, priority queuing, dynamic worker auto-scaling, and execution status tracking.

Functional Requirements

1. Schedule Delayed & Cron Tasks (POST /v1/jobs): Schedule jobs with specific future execution timestamps (delay), recurrence intervals (cron expressions), or instant execution.
2. Prioritized Task Dispatch: Execute tasks based on priority tiers (CRITICAL, HIGH, DEFAULT, LOW).
3. Task State & Lifecycle Tracking: Track job lifecycle states (SCHEDULED, DISPATCHED, RUNNING, SUCCEEDED, FAILED, CANCELLED).
4. Automated Retries & Exponential Backoff: Automatically retry failed tasks with configurable retry policies and Dead-Letter Queue (DLQ) routing.
5. Worker Heartbeating & Lease Renewal: Ensure long-running tasks (> 10 minutes) renew execution leases; un-heartbeated tasks are automatically rescheduled.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.999\%$ uptime SLA.
• Execution Timing Accuracy: Tasks execute within $< 500\text{ ms}$ of scheduled execution time.
• Durability: $99.999999999\%$ (11 9s) zero task loss guarantee before execution.
• Scale: Support $100\text{M}$ active scheduled tasks and $50,000\text{ tasks/sec}$ peak dispatch rate.

2. Capacity & Scale Estimation

Traffic Calculations

• Active Scheduled Tasks: 100 Million tasks ($10^8$).
• Daily Completed Task Volume: 500 Million tasks/day.
• Average Dispatch Throughput: $\text{Dispatch QPS} = \frac{500,000,000}{86,400} \approx \mathbf{5,787\text{ tasks/sec}}$
• Peak Dispatch Throughput ($5\times$ top of the hour burst): $\mathbf{30,000\text{ tasks/sec}}$.

Storage & Memory Estimation

• Task Definition Size:
  • job_id (16 Bytes), tenant_id (16 Bytes), target_url / handler (128 Bytes), payload_json (512 Bytes), schedule_cron (32 Bytes), next_run_epoch_ms (8 Bytes), retry_policy (64 Bytes) $\approx \mathbf{1\text{ KB per task}}$.
• Database Storage (100M active tasks): $100\times 10^6 \times 1\text{ KB} = \mathbf{100\text{ GB active storage}}$
• Historical Execution Log (30-Day Retention in S3): $500\text{M tasks/day} \times 1\text{ KB} \times 30\text{ days} = \mathbf{15\text{ TB/month}}$

3. AWS-First High-Level Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

4. API Interface Design

1. Schedule a Job

httpCopy
POST /v1/jobs
Host: scheduler.api.aws.internal
Content-Type: application/json
Authorization: Bearer <jwt_token>
Idempotency-Key: idemp_job_8492019

{
  "job_name": "generate_monthly_invoice",
  "schedule_type": "CRON",
  "cron_expression": "0 0 1 * *",
  "target": {
    "type": "HTTP_WEBHOOK",
    "endpoint": "https://billing.internal.aws/invoices/generate",
    "method": "POST",
    "timeout_seconds": 120
  },
  "payload": { "tenant_id": "tenant_992", "currency": "USD" },
  "priority": "HIGH",
  "retry_policy": { "max_retries": 3, "backoff_multiplier": 2.0, "initial_interval_seconds": 10 }
}

Response: 201 Created
{
  "job_id": "job_9847291048",
  "status": "SCHEDULED",
  "next_run_time_ms": 1718000000000
}