Trained in 4-bit vs Rounded to 4-bit

Kimi K3, the biggest open model ever announced: 2,800 billion parameters in a projected 1.4 TB file (weights land July 27). Llama 3.1 405B needs 0.81 TB for nearly 7 times fewer parameters. How is that possible?

🧮 The math first. A model file is its parameters stored as numbers. In the standard format each number takes 2 bytes, so Llama 3.1 is 405 billion parameters × 2 bytes = 0.81 TB. At that rate Kimi K3 would be 2,800 billion × 2 bytes = 5.6 TB. Moonshot AI stores each weight in 4 bits instead. Four bits is half a byte, and 2,800 billion × 0.5 = 1.4 TB, the projected file. Nearly 7 times the parameters of Llama, for less than twice the file.

🎯 The cost of compressing after training. A number stored in 4 bits can only take 16 different values. The usual approach, called post-training quantization, takes a fully trained model and rounds its weights to those 16 values. The model was trained with precise weights, so the rounding introduces an error it cannot correct, and quality drops.

Kimi K3 does it differently. From the fine-tuning stage onward, the model trains with its weights already stored in 4-bit format, a method called quantization-aware training. It adjusts to the 16 available values while it is still learning, so the rounding costs it much less.

⚙️ The format and the hardware. Kimi K3 uses a 4-bit format called MXFP4. Each weight is stored as a 4-bit floating point number, and each small block of weights shares one scaling factor, so 16 values can still cover a wide range. NVIDIA Blackwell chips and AMD's latest accelerators support the format directly in hardware.

That matters because generating each token means reading weights from memory again, and the speed limit is how fast those bytes move, not how fast the chip multiplies. Half a byte per weight means half the memory traffic. Worth noting, K3 is a mixture-of-experts model, so each token only uses a fraction of the 2,800 billion parameters. The 1.4 TB is what you must store, not what one token computes.

Until recently the hardware support was missing. The chips that handle 4-bit floats natively shipped over the last year, and OpenAI's GPT-OSS showed the method works in 2025, at 120 billion parameters. Kimi K3 is the first to apply it at the trillion scale.

⚠️ For info. The weights are not out yet. Moonshot has promised them for July 27 under a Modified MIT license, text not yet published. Today Kimi K3 is API only. The benchmarks are from Moonshot, and their own table shows K3 behind Claude Fable 5 and GPT-5.6 Sol. The one independent signal so far, from Artificial Analysis, scores it behind Fable 5.

💾 The takeaway.

▸ Compress after training and the model gets an error it cannot correct ▸ Train inside the constraint and the model compensates while it learns ▸ Weights land July 27, and Moonshot has already contributed the serving support to vLLM