Multi-Token Prediction Goes Mainstream

In December 2024, DeepSeek shipped Multi-Token Prediction.

Eighteen months later, every major open-weights LLM ships it. Qwen 3.6, Gemma 4, GLM-OCR, DeepSeek V4-Flash. Llama doesn't. Mistral doesn't. The same reason that explains those absences also explains why vLLM just deprecated general speculative decoding in v1.

Start with what makes inference slow. A model generates text one token at a time, each token requiring a full pass through the neural network. For years the trick to speed this up was speculative decoding: train a small draft model to guess the next few tokens, then have the big model verify them in one pass. Right guesses save time. Wrong ones fall back to the slow path.

What DeepSeek changed is where the draft lives. Instead of a separate small model alongside, they attached two or three tiny prediction heads directly onto the main model. Each head learns to predict a token further into the future: one for two ahead, one for three ahead, and so on. The heads reuse the main transformer's work; only the small output layers per look-ahead step are new.

At inference, those heads propose several future tokens, and the main model verifies them in a single pass. Same idea as speculative decoding, except the drafter is the model itself. Skip the heads entirely and the model is still better than it would have been without them, because training to predict further ahead forced the neural network to encode richer features. The speedup is optional. The quality lift is permanent.

That optionality drove fast convergence. By late 2025, SGLang, vLLM, and TensorRT-LLM all supported MTP. Apple released a retrofit recipe for existing models. Then Qwen3-Next, Qwen 3.6, and Gemma 4 shipped with native heads of their own.

The local speedup is real but more conservative than the loudest numbers. On Python code generation about 90 percent of tokens are accepted, which translates to roughly 2 to 3x faster output. On translation, around 1.8x faster. Apple Silicon with Qwen 3.6 lands near 1.5x, a Google H100 with Gemma 4 near 1.9x.

At production scale, the trick LOSES throughput. The reason is batching. When you serve many users at once the GPU is already busy with their requests, and the extra compute the heads need is no longer free. Recent measurements show alignment overhead eating 40 to 47 percent of compute at batch sizes of 8 to 16. vLLM, the most-used open-source inference engine, deprecated general speculative decoding in v1 for that exact reason.

So the apparent contradiction isn't really one. Open-weights AI converged on MTP as a training-time architecture choice, and that's permanent. As a runtime speedup it shines on a single laptop or one-user GPU and breaks at high concurrency.

Architecture and deployment economics are different things.