Design Nearby Friends Real-Time Location Service

1. Problem Statement & Scope Clarification

System Mission

Design a planetary-scale, real-time location sharing and proximity alerting platform (equivalent to Apple Find My, Snapchat Snap Map, or Zenly) capable of continuously ingesting live GPS coordinates from tens of millions of concurrent mobile devices, evaluating mutual friendship graphs, filtering spatial proximity ($< 5\text{ km}$), and delivering real-time map updates with sub-2-second propagation latency while rigorously defending mobile device battery longevity.

At a glance:

Functional Requirements

1. Periodic Location Broadcast (UpdateLocation): Ingest high-frequency GPS coordinates (lat, lon, accuracy, speed, timestamp) every 30 seconds from active mobile clients running in the foreground or authorized background.
2. Real-Time Nearby Friends Discovery (GetNearbyFriends): Retrieve a real-time list of mutual friends within a $5\text{ km}$ radius, including exact or blurred distance, bearing angle, and last-heartbeat timestamp.
3. Proximity Push Notifications: Trigger real-time push notifications when a mutual friend crosses a configurable proximity threshold (e.g., enters a $1\text{ km}$ geofence).
4. Granular Privacy & Ghost Mode Controls: Support per-user and per-friend privacy settings: Precise Location (exact coordinates), Fuzzy / City-Level (randomized within $\pm 2\text{ km}$), and Ghost Mode (completely invisible).
5. Battery-Aware Kinematic Polling: Dynamically throttle device GPS hardware polling intervals based on velocity and physical movement states (Stationary, Walking, Driving).

Non-Functional Requirements (SLAs & SLOs)

• High Ingestion Throughput: Ingest an average of $333,333\text{ updates/sec}$ with headroom for a $500,000\text{ updates/sec}$ peak.
• Propagation Latency: P95 end-to-end delivery of friend location updates across WebSockets $< 1.0\text{ second}$, $P99 < 2.0\text{ seconds}$.
• Availability: $99.999\%$ uptime SLA for the WebSocket Gateway and Ingestion tiers ($< 5.26\text{ minutes}$ unscheduled downtime/year).
• Data Ephemerality: Live GPS coordinates are strictly ephemeral (stored in-memory with a 10-minute TTL). Zero unencrypted raw GPS trails are persisted to permanent storage without explicit user opt-in.
• Battery & Network Efficiency: GPS polling and network transmission must consume $< 3\%$ battery per hour of active background tracking.

---

2. Capacity & Scale Estimation (Back-of-the-Envelope Math)

User Scale & Ingestion QPS

• Daily Active Users (DAU): $100,000,000\text{ users}$ ($100\text{M DAU}$).
• Concurrent Active Broadcasters: $10\%$ of DAU active simultaneously = $10,000,000\text{ active devices}$.
• Broadcast Frequency: 1 location update every 30 seconds ($1/30\text{ Hz}$).
• Average Ingest QPS: $\text{QPS}_{\text{ingest, avg}} = \frac{10,000,000\text{ devices}}{30\text{ sec}} \approx \mathbf{333,333\text{ updates/sec}}$
• Peak Ingest QPS ($1.5\times$ diurnal weekend evening spike): $\text{QPS}_{\text{ingest, peak}} = 333,333 \times 1.5 \approx \mathbf{500,000\text{ updates/sec}}$

The Quadratic Fan-Out Disaster Proof

Assume an average user has $400\text{ mutual friends}$. If every location update is naively broadcast to all friends:

$$\text{Naive Fan-Out Delivery Rate} = 333,333\text{ updates/sec} \times 400\text{ friends} = \mathbf{133,333,200\text{ msgs/sec}}$$

Delivering $133\text{ Million}$ WebSocket frames per second would require hundreds of gigabits of outbound bandwidth, massive Redis Pub/Sub cluster saturation, and millions of dollars in monthly cloud infrastructure costs.

Mathematical Mitigation via Mutual Cell Pub/Sub Pruning

Instead of indiscriminate global fan-out, we apply a two-stage filter:

1. Online Status Filter: Only $\approx 10\%$ of a user's friends are online and connected to a WebSocket gateway at any given moment: $N_{\text{online}} = 400 \times 10\% = 40\text{ friends}$
2. Spatial Proximity Filter: Only $\approx 5\%$ of online friends are located in the same or adjacent geographic cell ($< 5\text{ km}$ away): $N_{\text{nearby}} = 40 \times 5\% = \mathbf{2\text{ mutual nearby friends}}$

$$\text{Pruned Fan-Out Delivery Rate} = 333,333\text{ updates/sec} \times 2 = \mathbf{666,666\text{ msgs/sec}}$$

This architectural pruning achieves a $200\times$ message volume reduction (from $133\text{M}$ down to $666\text{K msgs/sec}$), making real-time delivery completely tractable on a production AWS deployment.

Network Bandwidth Sizing

• Uplink Payload Size (Client $\to$ Server): user_id (16B), lat (8B), lon (8B), accuracy (4B), speed (4B), timestamp (8B), seq (4B) $\approx 52\text{ bytes}$ (with JSON/Protobuf envelope $\approx 100\text{ bytes}$).
• Downlink Payload Size (Server $\to$ Client): friend_id (16B), lat (8B), lon (8B), distance_m (4B), bearing_deg (4B), timestamp (8B) $\approx 120\text{ bytes}$.
• Peak Ingress Bandwidth: $\text{Bandwidth}_{\text{in}} = 500,000\text{ updates/sec} \times 100\text{ bytes} = 50\text{ MB/sec} = \mathbf{400\text{ Mbps}}$
• Peak Egress Bandwidth: $\text{Bandwidth}_{\text{out}} = (666,666 \times 1.5)\text{ msgs/sec} \times 120\text{ bytes} = 120\text{ MB/sec} = \mathbf{960\text{ Mbps}}$

In-Memory Live State Sizing (ElastiCache Redis)

• Live Coordinate Hash per Active User: Key: loc:usr:<user_id> (32B), Fields: lat (8B), lon (8B), h3 (8B), ts (8B) $\approx 64\text{ bytes}$ (with Redis hash overhead $\approx 180\text{ bytes}$).

$$\text{Live Coordinate RAM} = 10 \times 10^6\text{ users} \times 180\text{ bytes} \approx \mathbf{1.8\text{ GB RAM}}$$

• Social Graph In-Memory Cache (Redis Sets): Cached friend lists for active DAU (10M users $\times$ 400 friend UUIDs $\times$ 16B) $\approx \mathbf{64\text{ GB RAM}}$.
• Total Redis Footprint: $\approx \mathbf{80\text{ GB RAM}}$, provisioned across a 6-node ElastiCache Redis Cluster (cache.r6g.xlarge nodes with 26.32 GiB RAM each).

Compute Fleet Sizing (WebSocket Gateway Fleet)

• Peak Concurrent Connections: $10,000,000\text{ [WebSockets](/components/18-websocket-sse-and-long-polling)}$.
• An optimized Netty / Rust async WebSocket worker container (4 vCPU, 8 GB RAM) reliably handles $50,000\text{ concurrent persistent TCP connections}$.

$$\text{Required WebSocket Tasks} = \frac{10,000,000\text{ connections}}{50,000\text{ connections/task}} = \mathbf{200\text{ ECS Fargate Tasks}}\text{ (67 per AZ)}$$

---

3. AWS-First High-Level Architecture

The architecture comprises three primary planes: 1) L4/L7 WebSocket Gateway Tier, 2) Ingestion, Spatial Discretization & Mutual Filtering Tier, and 3) Hexagonal Cell Redis Pub/Sub Backplane.

Synthesizing vector architecture diagram...

Follow Alice's location update to Bob's screen. Alice's phone sends it over its WebSocket to a gateway, which forwards it to the location router. In the "Location Ingestion & Social Filtering Tier" panel, the router saves her latest coordinates in Redis (with a 10-minute TTL, so stale users disappear) and knows her friend list from the graph cache. In the "Hexagonal Cell Pub/Sub Backplane" panel, it publishes the update to the channel of Alice's H3 cell; each gateway subscribes only to the cells around its connected users, so the update reaches just the gateways that might care. Bob's gateway checks that Alice is his friend, computes the distance, and pushes "Alice is 1.2 km away". Publishing by place instead of to every friend means each update reaches a handful of gateways, not all of them, which keeps the system scalable.

Data Flow Walkthrough

1. WebSocket Handshake & Channel Subscription: When Bob opens the mobile app, he establishes a persistent WSS connection via NLB to a WebSocket Gateway task (WS_Node3). Bob's client sends his initial coordinate (37.7749, -122.4194). The gateway computes Bob's Uber H3 Resolution 7 Cell (e.g., 872830828ffffff) and the $k=2$ ring around it (center + 6 + 12 = 19 cells, the smallest ring that fully contains a $5\text{ km}$ radius). WS_Node3 subscribes to these 19 Redis Pub/Sub channels on Bob's behalf; because subscriptions are multiplexed per gateway (Section 8.6), nearby users on the same gateway share them.
2. Location Ingestion Ping: Alice's device broadcasts her location update frame (lat, lon) over her open WebSocket to WS_Node1. WS_Node1 forwards the raw ping to the Location Router service.
3. Live State Update & Ephemeral TTL: The Location Router writes Alice's coordinates to ElastiCache Redis (HSET loc:usr_alice lat ... lon ... followed by EXPIRE loc:usr_alice 600 in one pipeline; HSET has no EX option of its own), resetting her 10-minute TTL heartbeat.
4. Hexagonal Spatial Publishing: The Location Router converts Alice's coordinate to an H3 Resolution 7 index and publishes the location payload exclusively to that specific cell's Redis Pub/Sub channel: PUBLISH channel:h3:872830828ffffff { user_id: Alice, lat: ..., lon: ... }
5. Gateway Mutual Friendship Gating: WS_Node3 receives the message because Bob is subscribed to that H3 cell. Before forwarding the update, WS_Node3 evaluates mutual friendship via an atomic in-memory check: SISMEMBER friends:usr_bob usr_alice
6. Haversine Distance Filter & Push: If mutual friendship is verified, WS_Node3 calculates the exact spherical Haversine distance between Alice and Bob. If $d \le 5\text{ km}$, WS_Node3 encodes the downlink frame and pushes it down Bob's open WebSocket connection in $< 500\text{ ms}$.
7. Proximity Geofence Alert: If Alice crosses from $> 1\text{ km}$ to $\le 1\text{ km}$ distance, the Location Router fires an asynchronous event to Amazon SNS, pushing an alert to Bob's notification center ("Alice is nearby!").

Core Request Tracing Execution Walkthrough

---

4. API Interface Design & Wire Protocols

1. Client Location Update Frame (WebSocket JSON)

Dispatched by mobile clients every 30 seconds over the persistent WSS connection.

jsonCopy
{
  "action": "UPDATE_LOCATION",
  "coordinates": {
    "latitude": 37.774929,
    "longitude": -122.419416,
    "altitude_meters": 42.5,
    "accuracy_meters": 8.0,
    "speed_mps": 1.4,
    "bearing_degrees": 182.0
  },
  "motion_state": "WALKING",
  "client_timestamp_epoch_ms": 1767225600120,
  "sequence_number": 4820
}

2. Server Downlink Location Broadcast Frame (WebSocket JSON)

Delivered from the gateway to connected mutual friends in the same spatial cell.

jsonCopy
{
  "event": "FRIEND_LOCATION_UPDATE",
  "friend_id": "usr_alice_99a8b7c6",
  "coordinates": {
    "latitude": 37.778291,
    "longitude": -122.418294
  },
  "distance_meters": 384,
  "bearing_degrees": 24.5,
  "privacy_mode": "PRECISE",
  "last_updated_epoch_ms": 1767225600150
}

3. Update Privacy Mode (PUT /v1/users/me/privacy)

Configures user location visibility and obfuscation.

httpCopy
PUT /v1/users/me/privacy HTTP/1.1
Host: api.nearby.platform.aws.internal
Authorization: Bearer jwt_usr_alice_99a8b7c6
Content-Type: application/json

{
  "global_privacy_mode": "FUZZY_CITY",
  "ghost_mode_until_epoch_ms": null,
  "friend_exceptions": [
    {
      "friend_id": "usr_bob_77182930",
      "privacy_mode": "PRECISE"
    }
  ]
}

• Modes:
  • PRECISE: Exact coordinates broadcast to mutual friends.
  • FUZZY_CITY: Coordinates obfuscated with randomized Gaussian noise ($\pm 2\text{ km}$) and snapped to nearest landmark.
  • GHOST: Completely invisible. All live Redis keys immediately deleted.

Status Codes & Error Contracts

• 200 OK: Successful privacy update or REST snapshot retrieval.
• 400 Bad Request: Out-of-bounds coordinates or invalid motion state string.
• 401 Unauthorized: Missing or expired WebSocket handshake JWT.
• 429 Too Many Requests: Client broadcasting faster than once every 10 seconds.

---

5. Data Models & Storage Architecture

Database Selection Justification

1. Amazon ElastiCache Redis 7 Cluster: Master datastore for active ephemeral locations, H3 cell channels, and active friendship cache. Redis delivers sub-millisecond in-memory read/write performance, native key expiration (10-minute TTL for dead connections), and horizontal pub/sub sharding.
2. Amazon DynamoDB: Maintains optional 30-day location history trails for opt-in auditing. DynamoDB handles 500k writes/sec with automated TTL purging and zero operational management.
3. Amazon Aurora PostgreSQL Multi-AZ: Master persistence for permanent user profiles, mutual friendship graph edges, and privacy settings.

1. Redis In-Memory Geospatial Data Structures

textCopy
1. Live Ephemeral Coordinate Hash:
   Key:    loc:usr:<user_id>
   Type:   HASH
   Fields: {
             "lat": "37.774929",
             "lon": "-122.419416",
             "h3": "872830828ffffff",
             "ts": "1767225600120"
           }
   TTL:    600 seconds (10 Minutes)

2. Mutual Friend Graph Set:
   Key:    friends:usr:<user_id>
   Type:   SET
   Value:  [ "usr_bob_101", "usr_charlie_102", "usr_david_103", ... ]
   TTL:    86400 seconds (Refreshed on friendship mutations)

3. H3 Hexagonal Cell Pub/Sub Channel:
   Channel: channel:h3:<h3_resolution_7_index>
   Example: channel:h3:872830828ffffff
   Payload: { "user_id": "usr_alice", "lat": 37.7749, "lon": -122.4194, "ts": 1767225600120 }

2. DynamoDB Location History Schema (LocationHistoryTable)

---