The AI Inference Iceberg

You call an API. You get a response in a couple of seconds. That's all you see. But underneath?

9 layers of engineering running in silence.

Here's the iceberg:

Inside the model:

1. Tokenization Your text is split into subwords. "unhappiness" becomes ["un", "happi", "ness"]. The model doesn't read text. It reads numbers.

2. Embedding Lookup Each token becomes a high-dimensional vector. A single sentence turns into a massive matrix.

3. KV Cache As the model generates tokens, it stores the computed Key and Value tensors so it doesn't recompute them for every new token. Without this, generation would be quadratically slower.

4. Attention Q×Kᵀ/√d → Softmax → ×V. This is where the model weighs relationships between tokens. The formula that changed AI.

5. Feed-Forward Network SwiGLU activations, billions of parameters. The layer where knowledge stored in weights gets activated.

Serving infrastructure:

6. GPU Scheduling Your request is batched with others (continuous batching) to maximize GPU utilization. You're sharing your H100 with dozens of users.

7. Memory Management PagedAttention allocates vRAM dynamically, like virtual memory for attention. Without it, a single model would eat the entire server memory.

8. Quantization BF16 or INT8 weights, pre-compressed before deployment. Less precision, faster math. The art of losing the right information.

9. Hardware H100 × 8, connected via NVLink, 640GB of HBM3. The bottom of the iceberg is silicon at $30K+ per chip.

Do you need to understand all of this to use an API? No.

But does understanding these layers make you better at debugging latency, choosing a provider, or optimizing costs? Absolutely.

Next time you get a response in a couple seconds, remember: 9 layers of engineering just ran a marathon at sprint speed.