How LLMs Actually Run: Prefill vs Decode

Here's a claim that sounds wrong until you've stared at a GPU utilisation chart during a long generation: your GPU is often idle 80% of the time while the model is "thinking." Not because the model is slow. Because generation and prompt processing are two completely different problems, and GPUs are only really good at one of them.

If you work on LLM serving — or you're about to — this distinction is the foundation everything else sits on. Continuous batching, PagedAttention, chunked prefill, speculative decoding: they all exist because of it.

---

The problem: one model, two workloads

When someone sends a request to an LLM, two things happen in sequence:

1. The model reads the entire prompt.
2. The model writes tokens one by one until it stops.

We call these prefill and decode. On paper they look like the same forward pass. On a GPU, they behave nothing alike.

That's the problem. A serving stack that treats them as one workload will leave throughput on the table, create unfair latency spikes, and make every later optimisation feel mysterious. Get this right, and the rest of inference engineering starts to make sense.

---

Prefill: the part GPUs love

Prefill is what happens when the prompt arrives. Say the user sent 2,000 tokens. The model needs to process all of them before it can emit the first output token.

The important detail: those tokens can be handled in parallel. Attention over the prompt, matrix multiplies through every layer — it's a big pile of dense linear algebra. That's exactly what GPUs are built for.

During prefill:

• You do a huge amount of math per byte loaded from memory
• GPU compute cores stay busy
• Utilisation is high

In other words, prefill is compute-bound. The bottleneck is how fast the GPU can multiply matrices, not how fast it can fetch weights.

This is also why time-to-first-token (TTFT) scales with prompt length. A 200-token system prompt is cheap. A 20K-token document dump is not — you're filling the GPU with real work.

---

Decode: the part that feels broken

Once prefill is done, generation starts. The model produces token 1, then token 2, then token 3… one at a time. Each new token depends on the previous one, so you can't parallelise across the output sequence.

Here's what each decode step actually does:

1. Load the full model weights from GPU memory
2. Load the growing KV cache (keys and values for everything seen so far)
3. Do a relatively small amount of math for one token
4. Write the new KV entries back
5. Repeat

You still touch most of the model. You just don't get much arithmetic out of that trip to memory. The GPU cores spend a lot of time waiting for data to arrive.

Decode is memory-bound. The bottleneck is bandwidth, not FLOPs.

That's why utilisation looks terrible during generation. The chip isn't "underpowered." It's underfed. You're paying the full cost of loading weights for a tiny amount of useful work per step.

---

Arithmetic intensity, without the academic fluff

A useful way to frame this is arithmetic intensity: roughly, how many operations you get for every byte you load from memory.

High arithmetic intensity → GPU cores stay busy → prefill.
Low arithmetic intensity → cores wait on HBM → decode.

Once you see it that way, a lot of "weird" serving behaviour stops being weird. Batching helps decode because you amortise one weight load across many sequences. Speculative decoding helps because you try to do more useful work per memory pass. Quantisation helps decode partly because smaller weights mean less data to move.

Same model. Same GPU. Completely different limiting factor depending on the phase.

---

A concrete picture

Imagine serving a 7B model on a single GPU.

Prefill of a 2,048-token prompt

• Matrix multiplies across thousands of tokens at once
• High FLOPs utilisation
• Feels "fast" relative to how much work is getting done

Decode of the next 256 tokens

• 256 separate forward passes
• Each pass loads weights again (plus KV cache)
• GPU utilisation drops
• Tokens per second is limited by how fast you can stream weights, not by peak FLOPs

This is also why adding users doesn't always look like linear slowdown in the way people expect. During decode, a larger batch can improve GPU efficiency — you're finally doing enough work to justify the memory traffic. During a long prefill, one fat request can monopolise the GPU and stall everyone else's decode. (That's a story for chunked prefill later in this series.)

---

Why this distinction matters in practice

If you only remember one thing from this post: every serious inference optimisation is a response to prefill ≠ decode.

A few examples you'll see again and again:

• Continuous batching — keep the GPU busy by mixing sequences that are in different phases, instead of waiting for a whole batch to finish.
• KV cache management — decode's cost grows with cached context; memory layout becomes a first-class product decision.
• Chunked prefill — stop one long prompt from freezing decode for every other user.
• Speculative decoding / Medusa-style heads — try to raise arithmetic intensity during generation so memory bandwidth isn't wasted on single-token steps.
• vLLM and friends — their scheduling, paging, and batching policies all exist because these two phases fight over the same GPU in different ways.

If someone asks "why is my tokens/sec bad?" and you don't first ask whether you're looking at prefill-heavy or decode-heavy traffic, you're debugging blind.

---

Key takeaways

• Prefill processes many prompt tokens in parallel. It's compute-bound and keeps GPU cores busy.
• Decode generates one token at a time. It's memory-bound: you load full weights (and KV cache) for relatively little math.
• Arithmetic intensity (ops per byte loaded) is the clean mental model for why the GPU feels "idle" during generation.
• Serving systems don't optimise "the model" in the abstract — they optimise around this split.
• Almost every trick in modern stacks (including vLLM) traces back to making decode less wasteful, or stopping prefill from starving decode.

---

Next up

Once you see that decode lives and dies by memory traffic, the next question is obvious: what exactly is eating your VRAM? In the next post we'll break down the GPU memory budget every ML engineer should know — and why adding one more user can take down an otherwise healthy inference server.