The Semantic Search That Exhausted All GPU RAM
The Semantic Search That Exhausted All GPU RAM
Your enterprise knowledge base powers an AI-driven RAG (Retrieval-Augmented Generation) search engine over 25 million documents. Each document chunk is embedded into a 1536-dimensional vector using OpenAI text-embedding-3-large (6 KB of raw float32 data per vector, totaling ~150 GB for raw embeddings). The engineering team chooses a state-of-the-art Hierarchical Navigable Small World (HNSW) graph index to achieve 98% recall with sub-10ms Approximate Nearest Neighbor (ANN) search. When the cluster is loaded into memory, the instances run out of memory and crash: the HNSW graph consumes over 650 GB of RAM — more than 4x the raw vector payload — due to multi-layer graph edge pointers, neighbor lists (M=32, efConstruction=200), and index metadata. Furthermore, as 50,000 new documents are inserted hourly, online graph node insertion locks cause p99 query latency to balloon from 8ms to 450ms. You are tasked with restructuring the vector storage and indexing architecture to slash RAM consumption by 70% while preserving high recall and sub-20ms search latencies.
The Semantic Search That Exhausted All GPU RAM
Your enterprise knowledge base powers an AI-driven RAG (Retrieval-Augmented Generation) search engine over 25 million documents. Each document chunk is embedded into a 1536-dimensional vector using OpenAI text-embedding-3-large (6 KB of raw float32 data per vector, totaling ~150 GB for raw embeddings). The engineering team chooses a state-of-the-art Hierarchical Navigable Small World (HNSW) graph index to achieve 98% recall with sub-10ms Approximate Nearest Neighbor (ANN) search. When the cluster is loaded into memory, the instances run out of memory and crash: the HNSW graph consumes over 650 GB of RAM — more than 4x the raw vector payload — due to multi-layer graph edge pointers, neighbor lists (M=32, efConstruction=200), and index metadata. Furthermore, as 50,000 new documents are inserted hourly, online graph node insertion locks cause p99 query latency to balloon from 8ms to 450ms. You are tasked with restructuring the vector storage and indexing architecture to slash RAM consumption by 70% while preserving high recall and sub-20ms search latencies.
Provide 1–2 precise sentences for each architectural dimension. Each box guides you on what staff-level interviewers evaluate.
Define SLA targets, hard consistency constraints, and conditions the system must never violate.
Quantify throughput (QPS/RPS), read:write ratios, and peak burst multipliers.
Step-by-step path: client ingress → API gateway → queues → background workers → persistence.
Database engine, table schema, partition keys (PK/SK), and durability strategy.
What resource hits saturation first under 10x traffic? (CPU, disk IOPS, connection pools, network).
Worker crashes, network partitions, split-brain, poison pill DLQ, retries, and idempotency.
What did you sacrifice in exchange and why? (e.g. eventual consistency vs latency, cost vs redundancy).