Design Figma's 100x Multi-Tenant Postgres & Real-Time Multiplayer Architecture

1. Problem Statement & Scope Clarification

System Mission

Design Figma's horizontal multi-tenant database scaling architecture and real-time multiplayer document synchronization engine. The system must support $40\text{M}+$ monthly active designers collaborating simultaneously on complex vector scene graphs, maintain sub-16ms ($60\text{ FPS}$) multiplayer editing latency across WebSockets, eliminate PostgreSQL connection exhaustion and lock contention by horizontally sharding databases across hundreds of RDS instances via a custom Rust SQL Proxy Router, and persist multi-gigabyte document snapshots reliably to Amazon S3.

Synthesizing vector architecture diagram...

Functional Requirements

1. Real-Time Multiplayer Canvas Synchronization: Multiplex hundreds of simultaneous editors on a single design file over WebSockets, synchronizing vector transforms, text edits, and cursor positions with $P99 < 30\text{ ms}$.
2. Horizontal PostgreSQL Multi-Tenant Sharding: Scale relational database storage and throughput by $100\times$ by horizontally partitioning tenants across independent Amazon Aurora PostgreSQL shards based on org_id and team_id. Tenant placement is directory-based (an explicit org_id -> shard_id map), never hash(org_id) % N: a static modulo would force a full re-hash of every tenant when a shard is added and would make moving one hot org to its own shard (§11, Q3) impossible.
3. Rust SQL AST Proxy Routing: Intercept all application SQL traffic; parse queries into Abstract Syntax Trees (ASTs), automatically inject shard routing keys, and strictly forbid cross-shard distributed transactions.
4. Deterministic Conflict Resolution (CRDT / OT): Reconcile concurrent tree structural operations (e.g., User A groups shapes while User B modifies color) deterministically without data loss or canvas corruption.
5. Periodic Snapshot Compaction & S3 Offload: Compact ephemeral in-memory operation streams into immutable binary scene graph snapshots flushed to Amazon S3, minimizing file load times on cold boot.

Non-Functional Requirements (SLAs & SLOs)

• Multiplayer Sync Latency: $P50 < 10\text{ ms}$, $P95 < 25\text{ ms}$, $P99 < 50\text{ ms}$ round-trip broadcast time between active editors.
• Relational Metadata Latency: $P95 < 5\text{ ms}$, $P99 < 15\text{ ms}$ for sharded PostgreSQL point lookups through the Rust proxy.
• System Availability: $99.99\%$ uptime for document read/write sessions ($< 52.6$ minutes downtime per year).
• Zero Cross-Shard Deadlocks: $100\%$ of write transactions must execute against a single physical database shard; cross-shard joins are rejected at the proxy layer.
• Data Durability Guarantee: 11 Nines ($99.999999999\%$) for canvas snapshots in Amazon S3; zero uncommitted operation loss via local sync server Write-Ahead Logs (WAL).
• CAP / PACELC Classification:
  • Multiplayer Sync Engine: AP / PA/EL system. Real-time canvas editing prioritizes availability and low latency; concurrent edits converge eventually using deterministic CRDT/OT ordering.
  • Relational Org/Billing Metadata: CP / PC/EC system enforcing strict ACID transactions on sharded PostgreSQL.

Out-of-Scope

• Complex server-side 3D ray tracing or real-time raster video rendering.
• Offline-only desktop native file storage formats (e.g., local .fig file import/export converters).
• Font licensing distribution and third-party plugin sandboxing runtimes.

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

Traffic & Multiplayer Concurrency Calculations

• Active Users: $40,000,000$ Monthly Active Users (MAU); $5,000,000$ Daily Active Users (DAU).
• Active Design Documents: $100,000,000$ design files stored across $500,000$ organizations.
• Peak Concurrent Multiplayer Editing Sessions: $250,000\text{ active collaborative sessions simultaneously}.$
• Average Active Editors per Session: $4\text{ editors/file}$ (with peak collaboration sessions reaching $500\text{ users}$ during design critiques).
• Multiplayer Operation Ingress Throughput: Each active editor generates an average of $5\text{ operations/sec}$ (object drags, vector point shifts, text typing): $\text{Peak Real-Time Op Ingress} = 250,000\text{ sessions} \times 4\text{ editors} \times 5\text{ ops/sec} = \mathbf{5,000,000\text{ ops/sec}}$
• WebSocket Egress Broadcast Bandwidth: Each op ($\approx 120\text{ bytes}$) is broadcast to other active editors in the room ($N-1 = 3$ recipients): $\text{Broadcast Egress} = 5,000,000\text{ ops/sec} \times 3 \times 120\text{ bytes} \approx 1.8\text{ GB/sec} = \mathbf{14.4\text{ Gbps}}$

Relational PostgreSQL Shard Sizing

• Relational Metadata Queries (File directory, permissions, user profiles): $\text{Average Ingress QPS} = \mathbf{80,000\text{ QPS}} \quad (\text{Peak Ingress QPS} = \mathbf{250,000\text{ QPS}})$
• Why Monolithic Postgres Collapsed at 100x Scale: A single large AWS RDS PostgreSQL instance (db.r6i.32xlarge) hits hard limits at:
  • Connection limits: $> 5,000$ concurrent connections saturate backend worker processes.
  • Lock contention: pg_locks table thrashing on shared organization tables.
  • Autovacuum failure: High write volume causes table bloat; autovacuum cannot keep up without disk I/O starvation.
• Horizontal Sharding Topology: Dividing $250,000\text{ peak QPS}$ across database shards (target $< 2,500\text{ QPS}$ per shard primary): $\text{Postgres Shards Required} = \frac{250,000\text{ QPS}}{2,500\text{ QPS/shard}} = \mathbf{100\text{ Shards (Each shard: 1 Primary + 1 Multi-AZ Replica)}}$

Canvas Scene Graph & Snapshot Storage Sizing

• Average Compacted File Snapshot: $2.5\text{ MB}$ compressed binary format.
• Active Files Edited Daily: $2,000,000\text{ files/day}$.
• Daily S3 Snapshot Storage: $\text{Daily Snapshot Volume} = 2,000,000 \times 2.5\text{ MB} = \mathbf{5.0\text{ TB/day}}$
• 3-Year Snapshot Horizon (with S3 Lifecycle Tiering): $\text{3-Year Raw Snapshots} = 5.0\text{ TB/day} \times 1,095\text{ days} \approx \mathbf{5.47\text{ Petabytes in Amazon S3}}$ Lifecycle policy transitions snapshots older than 90 days to Amazon S3 Glacier Instant Retrieval, saving $> 65\%$ on storage costs.

---

3. AWS-First High-Level Architecture

Synthesizing vector architecture diagram...

Follow an edit from the top. The client keeps a WebSocket through the NLB to the one sync node that owns the open file. In the "Sync Node Architecture" panel, an event loop applies each user's operation to the file's in-memory scene graph, merging concurrent edits so all users converge, and appends it to a local write-ahead log before acknowledging. In the "Snapshot Offload & Object Storage Tier" panel, the node writes a compacted snapshot to S3 every 30 s and queues operation records to SQS. In the "Rust Database Proxy & Sharded PostgreSQL Tier" panel, workers write that metadata through Figma's Rust proxy, which maps org_id to one of about 100 Aurora PostgreSQL shards. Real-time editing runs entirely in memory; the databases receive only batched, durable summaries, which is what lets them scale.

Data Flow Walkthrough

1. Multiplayer Session Initialization:
   • The user opens a Figma document. The client establishes a persistent WebSocket connection through AWS NLB to the assigned Multiplayer Sync Server.
   • If the document is not currently loaded in memory, the sync server fetches the latest compacted scene graph snapshot from Amazon S3 and replays any pending operations from the changelog table in sharded PostgreSQL.
2. Real-Time Operation Broadcast Path:
   • The designer drags a rectangle on the canvas. The client emits an atomic operation (Op: MoveNode(node_id, dx=10, dy=20)).
   • The sync server's non-blocking epoll thread receives the binary frame, applies the transformation to the in-memory scene graph, resolves any OT/CRDT ordering conflicts, and commits the mutation to an ephemeral local NVMe write-ahead log.
   • The server immediately broadcasts the operation to all other connected clients in the session room over their open WebSockets ($< 15\text{ ms}$).
3. Periodic Snapshot Flushing:
   • Every 30 seconds (or after 100 uncompacted operations), the sync server serializes the in-memory scene graph into a compact binary buffer and writes it asynchronously to Amazon S3.
4. Relational Metadata Routing (Rust DB Proxy):
   • When application microservices query tenant metadata (e.g., SELECT * FROM files WHERE org_id = 'org_492'), the query routes through the custom Rust DB Proxy.
   • The proxy parses the SQL into an AST, extracts the org_id sharding key, looks the tenant up in the shard directory (org_id -> shard_id, the shard_id column on organizations, replicated into every proxy's memory), and forwards the query to the dedicated Aurora PostgreSQL primary node.
   • If a query lacks a sharding key or attempts a cross-shard join (SELECT * FROM users JOIN teams ON ...), the proxy immediately throws an error: CROSS_SHARD_TRANSACTION_PROHIBITED.

---

End-to-End Request Tracing Walkthrough

---

4. API Interface Design & Wire Protocol

1. Multiplayer WebSocket Binary Wire Protocol (SyncMessage.proto)

protobufCopy
syntax = "proto3";

package figma.multiplayer.v1;

enum OpType {
  NODE_INSERT = 0;
  NODE_UPDATE = 1;
  NODE_DELETE = 2;
  CURSOR_MOVE = 3;
  SELECTION_CHANGE = 4;
}

message Vector2D {
  float x = 1;
  float y = 2;
}

message CanvasOp {
  OpType op_type = 1;
  string node_id = 2;
  string parent_id = 3;
  uint64 client_sequence = 4;
  uint64 server_sequence = 5;
  bytes serialized_properties = 6; // Compact binary vector/color payload
}

message SyncFrame {
  string file_id = 1;
  string user_id = 2;
  uint64 session_epoch_ms = 3;
  repeated CanvasOp operations = 4;
  Vector2D cursor_position = 5;
}

---

2. REST Metadata API: Create Design File (POST /v1/files)

httpCopy
POST /v1/files HTTP/1.1
Host: api.figma.com
Content-Type: application/json
X-Figma-Org-Id: org_8492019284
X-Figma-Team-Id: team_301928471
Authorization: Bearer <oauth_jwt_token>
Idempotency-Key: 7f8a9b2c-3d4e-5f6a-7b8c-9d0e1f2a3b4c

{
  "name": "Design System 3.0",
  "folder_id": "fld_92019283",
  "default_canvas_color": "#1E1E1E"
}

Response: 201 Created

jsonCopy
{
  "file_id": "file_01HZX894KMNPQ9283",
  "org_id": "org_8492019284",
  "team_id": "team_301928471",
  "name": "Design System 3.0",
  "assigned_shard_id": 42,
  "sync_endpoint_wss": "wss://multiplayer.figma.com/v1/connect?file_id=file_01HZX894KMNPQ9283",
  "created_at_epoch_ms": 1773648000000
}

---

3. Status Codes & Error Contracts

---

5. Data Models & Storage Architecture

Sharded PostgreSQL Relational Schema

All tables enforce a strict multi-tenant partitioning hierarchy: org_id $\to$ team_id $\to$ file_id. Every single table in the schema contains org_id to enable deterministic single-shard routing.

Synthesizing vector architecture diagram...

Read the hierarchy top-down: an ORGANIZATION contains TEAMs, a team owns FILEs, and each file has SNAPSHOTs (checkpoints stored in S3) and an OPERATION_LOG (the edits since the last snapshot, ordered by op_sequence). The key detail is that every table carries org_id, even where a parent ID would be enough, and ORGANIZATION records its shard_id. Since the proxy routes every query by org_id, all of an organization's rows land on one shard, so its joins, foreign keys and transactions stay local. Loading a file means reading its latest snapshot and replaying only the operations after it.

Production PostgreSQL DDL: Sharded Files & Snapshots

sqlCopy
-- Deployed on each of the 100 Aurora PostgreSQL Shards
CREATE TABLE files (
    file_id VARCHAR(64) NOT NULL,
    org_id BIGINT NOT NULL,            -- Sharding Key: Guarantees co-location
    team_id BIGINT NOT NULL,
    name VARCHAR(255) NOT NULL,
    current_version BIGINT NOT NULL DEFAULT 1,
    is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (org_id, file_id)      -- Composite PK includes Shard Key
);

CREATE TABLE file_snapshots (
    snapshot_id BIGSERIAL,
    org_id BIGINT NOT NULL,
    file_id VARCHAR(64) NOT NULL,
    version BIGINT NOT NULL,
    s3_uri VARCHAR(512) NOT NULL,      -- e.g., 's3://figma-snapshots/org_42/file_109/v45.bin'
    byte_size BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (org_id, snapshot_id),
    CONSTRAINT fk_file FOREIGN KEY (org_id, file_id) REFERENCES files (org_id, file_id) ON DELETE CASCADE
);

CREATE TABLE file_operations_log (
    op_id BIGSERIAL,
    org_id BIGINT NOT NULL,
    file_id VARCHAR(64) NOT NULL,
    op_sequence BIGINT NOT NULL,
    operation_payload BYTEA NOT NULL,  -- Compact binary Protobuf operation
    committed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (org_id, op_id),
    CONSTRAINT uq_file_seq UNIQUE (org_id, file_id, op_sequence)
);

CREATE INDEX idx_files_team ON files (org_id, team_id);

---