NVIDIA Dynamo: Distributed LLM Serving

NVIDIA just shipped the v1 of an open-source repo really interesting.

I wrote a post not long ago about disaggregated prefill and the massive cost savings it enables on inference. Well, NVIDIA just took it to datacenter scale in an open-source repo.

It's called Dynamo. It sits on top of vLLM, TensorRT-LLM, SGLang. Turns them into a coordinated distributed system. 6,355 stars on GitHub, Rust-based orchestration core with Python bindings.

So what does it actually change?

Four things:

1/ Disaggregated serving Prefill and decode on separate GPU pools. vLLM already supported this. Dynamo turns it into a first-class distributed primitive, with independent autoscaling and cluster-level coordination.

2/ Multi-tier KV cache The cache doesn't stay stuck on a single GPU. Dynamo cascades it: GPU → CPU → SSD → remote storage. And the routing is KV-aware, it sends your request to the node that already has the right cache. Baseten measured 2x faster time to first token with KV-aware routing on Qwen3-Coder 480B.

3/ ModelExpress GPU-to-GPU weight streaming via NVLink. Want to spin up a new replica? 7x faster than loading from disk. For autoscaling, that changes everything.

4/ SLA-driven Planner You define your SLA (target latency), the Planner profiles your workload and adjusts resources automatically. No manual GPU babysitting.

Result: 7x higher throughput per GPU on DeepSeek R1 with GB200 NVL72.

At some point, serving LLMs stops being a model problem and becomes a distributed systems problem. And that shift is happening faster than most teams expect.

When to go distributed? → 1 model, moderate traffic: vLLM alone is enough → Multi-model + long contexts: you need an orchestrator → Datacenter scale: Dynamo