Prefix Caching vs LMCache

Two months ago, I wrote that KV cache was becoming infrastructure. Here's what happened since.

Every time an LLM processes a prompt, it computes key-value pairs for every token. That's the KV cache, the model's working memory. On a 70B model with 8K context, that's about 2.5 GB per request. Serve 32 users and the cache alone eats 80 GB, more than the model weights themselves. And it's thrown away after each request.

Now imagine a RAG system where 50 different queries retrieve the same document. Without caching, the model recomputes that document's KV tensors 50 times. Same bytes, same math, 50 times the GPU cost.

LMCache's answer: prefill each text once. Store the KV cache across GPU memory, CPU RAM, disk, or S3. When the same text shows up again, pull from cache instead of recomputing. Up to 15x throughput improvement on document analysis workloads.

Most caching systems can only do this for prefixes, meaning your input has to start with the same text as a previous request. That works for system prompts. But in RAG, retrieved documents land in different positions across queries. Prefix caching can't help.

LMCache reuses cached KV tensors from any matching text segment, regardless of position. Their CacheBlend paper (Best Paper, EuroSys 2025) describes how: reuse the cached values, selectively recompute roughly 15% of tokens to account for the preceding context. The rest is served from cache.

Think of it as a CDN for KV tensors. A CDN pre-computes HTML and serves it from the closest edge node. LMCache pre-computes KV tensors and serves them from the closest memory tier. The project actually calls this a Knowledge Delivery Network.

The adoption reflects it. Google Cloud, CoreWeave, GMI Cloud run it in production. vLLM, SGLang, and NVIDIA Dynamo integrate it natively. Redis, Weka, and PliOps provide storage backends. Nearly 8,000 GitHub stars, 4 peer-reviewed papers.

In my original post, I noted that every player was building their own KV cache stack with no interop guarantees. The fragmentation is still real, but one project is accumulating gravity faster than the others.

If you're running vLLM with repeated context (RAG, multi-turn, shared system prompts), this is worth benchmarking. pip install lmcache, Apache 2.0.