The KV Cache Is Becoming Infrastructure

Debating whether LLM APIs need better abstractions is the right question.

But right now, something is happening one level below: the KV cache is being extracted from inference engines and turned into managed infrastructure.

NVIDIA announced a four-tier storage hierarchy for it. LMCache is making it the connective tissue between engines. llm-d routes requests to the GPU that already has your prefix cached. vLLM is experimenting with running prefill and decode on separate hardware.

Providers already cache KV tensors behind the scenes. But their caching is opaque. 5 to 60 minutes, then gone.

What's being built now is different: explicit, persistent, routable KV management. The kind of shift data engineering went through with the move from raw files to lakehouses.

Except this time, convergence isn't guaranteed.