The PCIe Wall: 10 Years of Layer Offloading

DeepSeek-R1 671B. A single RTX 4090. 14 tokens per second.

Yes, but...

The GPU is the cheapest part. The system that hits 14 tok/s costs around $10,000. An Intel Xeon Gold server CPU ($3,200), 382 GB of DDR5 server RAM ($2,500+), and an RTX 4090 ($1,600). Swap in a consumer AMD chip with 96 GB desktop RAM: 3-4 tok/s.

Still. A 671B model on anything short of an H100 would have been unthinkable two years ago. The technique has been quietly evolving for a decade.

In 2016, a Samsung research team figured out something fundamental: a transformer is a stack of blocks that run one after another, not in parallel. Block 1 finishes, its output feeds Block 2, then Block 3. At any moment, the GPU only works on one block. The rest just sit in VRAM doing nothing.

So why keep them there? Store the waiting blocks in CPU RAM, keep only the active one on GPU. While the GPU computes Block N (30-50ms), load Block N+1 over PCIe (~6ms). The GPU never waits.

Five generations refined this:

2016, vDNN: swap activations to CPU. 89% VRAM savings (on CNNs). 2021, ZeRO-Infinity (Microsoft): offload across GPU, CPU, and NVMe. 2023, FlexGen (Stanford): OPT-175B on a single T4 16GB. About 1 tok/s, but it ran. 2024, PowerInfer: neuron-level routing instead of full layers. 2025, KTransformers (SOSP): MoE experts in INT4 on CPU with Intel AMX, attention in FP8 on GPU.

What caught my attention isn't the speed.

Ampere: PCIe 4.0 ~25 GB/s, VRAM ~900 GB/s. A 36x gap. Blackwell: PCIe 5.0 ~50 GB/s, VRAM ~1,800 GB/s. Still 36x.

Both sides got faster at the same rate and we have the same gap since 2016.

Every innovation above hides the same wall. Reduce data crossing the bus, overlap transfers with compute, or compute on CPU where the data already lives.

One trap: putting a small fraction of layers on GPU can be slower than full CPU. The curve is U-shaped.

When to quantize vs offload?

Model fits in VRAM after Q4: quantize. Fastest. Need bit-perfect precision: layer offloading. 1-3 tok/s for 70B. Slow but lossless. 200B+ MoE: hybrid. Dense on GPU, sparse experts on CPU. This is what KTransformers does.

Five generations, same 36x wall. The hardware didn't solve it. The software learned to hide it.