Why Quantization Works

Quantization removes 87.5% of a weight's precision. 32 bits → 4 bits. And somehow, the model still works.

Three things make this possible:

1. 95% of weights live near zero.

After training, most weights in a neural network cluster around ±0.1. Tight bell curve, almost nothing at the extremes.

32 bits (4 bytes) per weight means 2³² possible values, over 4 billion. Multiply by 70 billion parameters: 280 GB.

4 bits (half a byte) per weight means 2⁴ = 16 possible values. Same 70 billion parameters: 35 GB.

In practice, most models ship in BF16 (16 bits, 2 bytes, 140 GB). But the quantization math is the same: fewer bits, fewer ticks, same snap.

When you go from billions of ticks to 16, the absolute rounding error stays small for the vast majority of weights. They snap to the nearest tick, and the network barely notices.

But "barely notices" only works for the bulk. Not for everything.

2. The outliers are the real problem.

Less than 1% of weights sit at the extremes. They look negligible on a histogram.

They're not. These rare outlier weights carry disproportionate importance for model quality. Crush them with naive rounding, and perplexity explodes. The model becomes incoherent.

This is why INT4 alone isn't enough. Algorithms like GPTQ and AWQ exist specifically to protect these critical weights during quantization. And NF4 (used in QLoRA) goes further: it spaces the 16 values non-linearly, allocating more precision near zero where the bell curve peaks.

3. Overparameterization absorbs the rest.

70 billion parameters means massive redundancy. When quantization introduces small rounding errors (quantization noise), the network's redundancy compensates. Slightly shift one weight and thousands of others cover the gap.

It's why quantization is possible at all.

The result:

280 GB → 35 GB. Your 70B model runs on a MacBook.

Not because precision doesn't matter, but because smart math meets redundant networks.

The visual shows this step by step. Same 55 weights snapping from 4 billion ticks down to 16. Watch the dots cluster.