LEANN: The Vector Index That Deletes Its Embeddings

Berkeley Sky Lab, the group behind vLLM and SGLang (and earlier Spark and Ray), just shipped a vector index that deletes the embedding store entirely.

60 million Wikipedia chunks fit in 4 GB on a laptop. It comes at a price. Standalone vector search runs around 50 times slower than a normal HNSW index.

In a normal vector database (FAISS, Milvus, pgvector), embeddings are precious data: stored, indexed, paged from disk. LEANN treats them as functions. The same chunk through the same embedding model produces the same vector. So why store it?

LEANN keeps three things: a pruned proximity graph (the topology), a tiny Product Quantization codebook (PQ, short approximate codes that stand in for full embeddings), and the raw text chunks. Full embeddings are never persisted. They are recomputed on demand during graph traversal, but only for the top few percent of candidates. The rest get cheap PQ approximate scores.

Two structural tricks make the trade-off tractable.

First, the search is two-level. Cheap PQ distances for every neighbor visited, full embedding recomputation only for the most promising candidates. Best-first traversal on an HNSW-style graph touches O(log N) nodes, so the number of expensive recomputations per query stays small.

Second, the graph itself is compressed. Around 2 percent of nodes (the small-world hubs) keep their full connectivity, the rest get aggressively pruned. Stored in CSR format with 4-byte neighbor IDs. Graph metadata drops from 15 GB to 2 GB on a 76 GB corpus.

Result on the paper's 60M Wikipedia benchmark: 188 GB down to 4 GB. The README's 97 percent number rounds up the baseline. The real figure is 47×.

PQ alone at the same compression collapses below BM25 accuracy. The graph traversal is what salvages recall.

The catch worth knowing. End-to-end RAG latency only stays comparable because LLM generation in the paper's setup already takes more than 20 seconds. Retrieval overhead becomes around 10 percent of total. For semantic search UI on a sub-50 millisecond budget, LEANN breaks.

It shines on personal RAG (email, browser history, Claude exports) and as the MCP semantic search backend for Claude Code, where no incumbent ships a Claude-Code-first integration. It breaks on multi-tenant cloud SaaS at 10k QPS (use Milvus or Qdrant) and on real-time write churn, where soft deletion degrades the graph until rebuild. The embedder also has to stay local and deterministic; switching it means rebuilding the index.

The principle, even if you never touch LEANN: compute versus storage is an architectural choice, not a constraint. When something downstream already dominates your latency budget, you can pay in compute to free up disk. Same logic applies to KV cache compression, agent memory layers, and on-device personalization.

The question is architectural: where does latency live in your pipeline, and what can you trade against it?

Paper: arXiv 2506.08276. Repo: yichuan-w/LEANN (MIT).