Design Google Drive File Sync & Storage

1. Problem Statement & Scope

System Mission

Design an enterprise-grade, cross-platform cloud file storage and multi-device synchronization engine capable of managing billions of files, supporting content-defined chunking (CDC), block-level differential synchronization (delta sync), global data deduplication, real-time cross-device mutation push, and ACID-compliant directory namespace management.

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Functional Requirements

1. Hierarchical File Management: Support nested folder hierarchies, renaming, moving, and soft/hard deletions (Trash).
2. Block-Level Differential Delta Sync: Split files into content-defined chunks ($4\text{ MB}$ average), compute cryptographic hashes (SHA-256), and transfer only modified blocks over the network.
3. Global Content Addressable Storage (CAS) Deduplication: Deduplicate identical blocks across all files and users globally to minimize raw storage footprint.
4. Real-Time Multi-Device Fanout: Instantly notify all connected devices (desktop, mobile, web) when files are modified, added, or deleted.
5. Revision History & Conflict Resolution: Maintain granular 30-day revision history with automatic creation of "Conflicted Copies" during concurrent offline edits.

Non-Functional Requirements (SLAs/SLOs)

• High Availability: $99.999\%$ uptime SLA for file read/write APIs ($< 5.26\text{ minutes}$ downtime/year).
• Data Durability: 11 9s ($99.999999999\%$) durability via Amazon S3 multi-AZ storage.
• Sync Propagation Latency: Delta sync notification delivered to online peer devices in $< 500\text{ ms}$ globally ($P95$).
• Network Bandwidth Optimization: Delta sync must achieve $\ge 90\%$ bandwidth savings on incremental edits of large documents.
• Client Resource Footprint: Local client RAM $< 150\text{ MB}$, background CPU $< 1\%$ when idle.

Out-of-Scope

• Collaborative real-time rich-text co-authoring engines (OT / CRDT character-by-character live document typing as in Google Docs; covered in collaborative editing blueprints).
• Server-side full-content Optical Character Recognition (OCR) and deep AI semantic video indexing.

2. Capacity & Scale Estimation

Traffic & Scale Assumptions

• Total Registered Users: $500\text{ Million}$ users.
• Daily Active Users (DAU): $100\text{ Million}$ active users.
• Active Connected Devices: Average $2.5\text{ devices/DAU} \implies \mathbf{250\text{ Million Concurrent [WebSocket](/components/18-websocket-sse-and-long-polling) Connections}}$.
• Daily File Modifications / Sync Commits: Average $10\text{ sync events/user/day} \implies 1\text{ Billion sync commits/day}$. $\text{Average Sync Commit QPS} = \frac{10^9 \text{ commits}}{86,400 \text{ s}} \approx 11,574 \text{ QPS (Peak: } 35,000\text{ QPS)}$

Storage & Delta Bandwidth Math

• Average File Size: $5\text{ MB}$. Total user storage quota average $= 15\text{ GB}$. $\text{Total Raw User Data Stored} = 500 \times 10^6 \times 15\text{ GB} = 7,500\text{ PB} = \mathbf{7.5\text{ Exabytes}}$
• Deduplication Ratio: Content-defined chunking yields an estimated $3.5\times$ storage reduction via cross-user block deduplication. $\text{Actual S3 Physical Storage} \approx \frac{7.5\text{ EB}}{3.5} \approx \mathbf{2.14\text{ Exabytes}}$
• Delta Sync Network Bandwidth Savings:
  • Without Delta Sync (Full file upload): $10^9 \text{ edits} \times 5\text{ MB} = 5\text{ PB/day} \approx \mathbf{463\text{ Gbps Bandwidth}}$.
  • With Block-Level Delta Sync: Only 1 modified $4\text{ MB}$ chunk uploaded per commit: $10^9 \times 4\text{ MB} = 4\text{ PB/day}$.
  • For small edits ($100\text{ KB}$ diff inside a $100\text{ MB}$ file), delta sync reduces network payload from $100\text{ MB} \to 4\text{ MB}$ ($96\%$ reduction).

3. High-Level Architecture & AWS Component Mapping

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

Component Responsibility Breakdown

4. API Interface Design & Wire Protocols

1. Delta Sync Commit & Block Hash Negotiation

httpCopy
POST /v1/files/sync/commit HTTP/1.1
Host: api.drive.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "file_id": "file_88129384bc",
  "base_version": 14,
  "file_name": "quarterly_financial_model.xlsx",
  "parent_folder_id": "folder_root_001",
  "file_size_bytes": 20971520,
  "chunk_hashes": [
    "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2",
    "a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e",
    "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8",
    "4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a"
  ]
}

Response: 200 OK (Only Missing Chunks Require Upload)

jsonCopy
{
  "file_id": "file_88129384bc",
  "new_version": 15,
  "sync_action": "UPLOAD_REQUIRED",
  "missing_chunks": [
    {
      "chunk_hash": "a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e",
      "presigned_put_url": "https://s3.us-east-1.amazonaws.com/drive-cas-blocks/a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e?X-Amz-Signature=..."
    }
  ]
}

2. Finalize Commit Once Chunks Uploaded

httpCopy
POST /v1/files/sync/finalize HTTP/1.1
Host: api.drive.aws.internal
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "file_id": "file_88129384bc",
  "new_version": 15,
  "uploaded_chunk_hashes": [
    "a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e"
  ]
}

Response: 200 OK

jsonCopy
{
  "file_id": "file_88129384bc",
  "status": "COMMITTED",
  "version": 15,
  "updated_at": 1718000500123
}

5. Data Models & Storage Architecture

Interactive Architecture Diagram
100%Fit100%
Synthesizing vector architecture diagram...

DynamoDB Single-Table Schema (GoogleDriveCoreTable)