Chapter 25 · Hardware

GPUs, the Memory Wall, and Why Inference Is Slow

A GPU generating a token from a 70-billion-parameter model spends over 99% of its time waiting for memory and almost none of it doing arithmetic. This chapter explains why that is true, why it is not a bug, and what every serving technique and architectural fashion of the last three years has been doing about it.

37 sections H100 · Llama 3 · Gemma 2/3/4 Every number derived from published specs 14 drills Reading time ~75 min

Contents

25.1 The number that explains everything

Take a 70-billion-parameter language model, quantised to 8 bits so it fits, and put it on an NVIDIA H100 — roughly the most capable inference card widely deployed today. Ask it to generate one token. Here is where the time goes.

The arithmetic takes 0.071 milliseconds. Reading the weights out of memory so that arithmetic can happen takes 21.1 milliseconds. The GPU spends 99.7% of the step waiting, and 295.0 times longer fetching numbers than using them.

This is not a misconfiguration. It is not a bad kernel, an old driver, or a framework overhead you can tune away. It is what the hardware does when you ask it to do this particular job, and every serious technique in LLM serving — batching, paged attention, quantisation, speculative decoding, mixture of experts — is a response to this one fact.

Most engineers meet this backwards. They learn that GPUs are fast at matrix multiplication, observe that transformers are mostly matrix multiplication, and conclude that inference should be compute-bound. Then they watch a production endpoint produce forty tokens a second on hardware rated for a thousand trillion operations a second, and have no framework for the gap.

This chapter builds that framework from the hardware up. By the end you should be able to predict, on paper and before deploying anything, roughly how fast a given model will run on a given card, what will break first as you add users, and which of the dozen available optimisations actually addresses your problem rather than someone else's.

25.2 The bet a GPU makes

A CPU and a GPU are given the same problem — silicon is finite, memory is slow — and they answer it in opposite ways. Understanding the two answers is most of understanding why inference behaves as it does.

A CPU is built so that one thread runs as fast as possible. When that thread asks for something not in cache, everything stops until it arrives: hundreds of cycles of nothing. So a CPU spends most of its transistor budget on not waiting — large caches, branch prediction, speculative and out-of-order execution. All of it exists to keep a small number of threads from stalling.

A GPU gives up on that entirely. It assumes stalls are unavoidable and makes them irrelevant by having enormously many threads available. When one warp stalls on memory, the scheduler runs another. With 270,336 threads resident on an H100 across 132 streaming multiprocessors, there is essentially always something ready to run.

25.3 Inside a streaming multiprocessor

The H100's 132 streaming multiprocessors are the actual units of execution. Each SM holds up to 2048 resident threads, its own register file, its own shared memory, and two quite different kinds of arithmetic hardware.

CUDA cores do ordinary scalar arithmetic — one multiply-add per cycle per core, on ordinary floating-point numbers. They are general-purpose and, by modern standards, slow: about 67.0 TFLOP/s in FP32 across the whole chip, which sounds enormous until you see the next number.

Tensor cores do one thing: they multiply small matrices and accumulate the result, as a single instruction. Not element by element — a whole tile at once. That specialisation buys about 989.0 TFLOP/s at FP16 — 14.8× the CUDA-core rate, and the reason a single card can claim nearly a thousand trillion operations a second.

25.4 Warps, and why 32 is everywhere

Threads on a GPU do not execute individually. They are grouped into warps of exactly 32 threads, and a warp is the true unit of scheduling: all 32 threads issue the same instruction in the same cycle, each on its own data. NVIDIA calls this SIMT — single instruction, multiple threads.

This is why 32 appears constantly in GPU programming advice. A block of 100 threads does not run as 100 threads; it runs as four warps, the last of which has 28 threads sitting idle doing nothing while the other four work. Block sizes that are not multiples of 32 waste a fraction of the machine for no reason at all.

Each SM can hold 64 warps resident at once, which across the chip is 8448 warps in flight. The scheduler picks, every cycle, from whichever of them is ready. That pool is the entire latency-hiding mechanism.

25.5 Warp divergence

The lockstep has a cost, and it appears the moment your code branches.

If some threads in a warp take the if and others take the else, the hardware cannot run both at once. It runs the if branch with the else threads disabled, then the else branch with the if threads disabled. Both paths execute, serially. A two-way branch inside a warp costs you 50% of that warp's throughput.

25.6 The memory hierarchy

Four levels, each roughly an order of magnitude slower and larger than the one above it. Everything in this chapter is ultimately about the distance between the top and the bottom.

Two things in that table deserve a second look, because they are unlike a CPU.

First, the register file is 33.0 MB. That is the same order of magnitude as the L2 cache, where on a CPU registers are measured in bytes against megabytes of cache. Each SM's registers are roughly 512.0× a typical CPU core's architectural register budget. This is what it costs to keep 270,336 threads resident: every one of them needs its state kept live, because the whole point is switching between them for free.

Second, per thread that generosity evaporates — about 128.0 bytes each. GPU threads are not small CPU threads. They are extremely numerous and individually impoverished.

25.7 Coalescing: the cheapest own goal

HBM is not read a byte at a time. It is read in sectors of 32 bytes, and this single implementation detail decides whether a memory-bound kernel runs at full speed or at an eighth of it.

When the 32 threads of a warp read 32 consecutive 4-byte values, the hardware coalesces them into 4 transactions and every byte fetched gets used. When those same threads read with a large stride — each landing in a different sector — the hardware issues 32 transactions and discards most of what it fetched. Identical arithmetic, identical thread count, 8× the memory traffic.

25.8 Occupancy and hiding latency

Now the pieces combine into the mechanism the whole design depends on.

A warp requests data from HBM and stalls for several hundred cycles. On a CPU this would be a disaster. On a GPU the scheduler simply switches to another resident warp, at zero cost, because that warp's registers were never swapped out — this is what the enormous register file bought. If enough warps are resident, the memory latency of any one of them is completely hidden behind the execution of the others.

Occupancy is the fraction of the maximum resident warps you actually achieve. Low occupancy means too few warps to hide stalls, so the SM idles waiting. Occupancy is usually limited by registers per thread or shared memory per block: use too much of either and fewer blocks fit, so fewer warps are resident, so less latency is hidden.

25.9 Compute outran memory

Everything so far describes a machine that is very good at arithmetic provided the data arrives. This section is about the data not arriving.

Between 2016 and 2024, peak compute on NVIDIA's flagship datacentre parts grew by a factor of 106.1. Memory bandwidth over the same period grew by a factor of 10.9. Both improved enormously, which is precisely why the problem is easy to miss: nothing got worse in absolute terms. What changed is the ratio between them, and the ratio is what determines whether your kernel runs at peak.

Read that chart as a demand rather than an achievement. Each bar is the number of arithmetic operations you must perform on every byte you fetch, just to keep that generation's maths units busy. On a P100 you needed 29.0 operations per byte. On an H100 you need 281.0. Fall short and the tensor cores idle, however impressive the number on the datasheet.

25.10 Arithmetic intensity

The quantity those bars measure has a name, and it is the single most useful number in this chapter.

Compare the extremes. Adding two vectors reads 8 bytes, does 1 addition, writes 4 bytes: an intensity of about 0.08, hopelessly memory-bound on any hardware. Multiplying two large square matrices does n operations for every element it loads, giving an intensity that grows with the matrix — which is exactly why matrix multiplication is the operation GPUs were optimised for.

Now the question that decides everything: how much arithmetic intensity does your workload have, and how much does the hardware demand?

25.11 The roofline model

Williams, Waterman and Patterson gave us the standard way to answer that in 2009, and it remains the right mental model. Plot achievable performance against arithmetic intensity and you get two regimes joined at a corner.

On the left, performance is limited by bandwidth: you are moving bytes as fast as the memory system allows and the arithmetic units are partly idle. The slope of that line is the memory bandwidth. On the right, performance is capped by the arithmetic units themselves — the flat roof.

The corner between them is the ridge point: the arithmetic intensity at which the two limits are equal. For an H100 at FP8 it sits at 590.7 FLOP per byte. Below it you are memory-bound; above it you are compute-bound. That one number tells you which half of the machine you are actually using.

25.12 Tensor cores move the ridge right

Here is the conclusion people resist, and it follows directly from the definition.

The ridge point is peak compute divided by peak bandwidth. Make the compute faster without touching the memory system, and the ridge moves right: you now need more arithmetic per byte before the hardware is saturated. On the H100 the ridge sits at 20.0 FLOP/byte for ordinary FP32 on CUDA cores, 295.3 for FP16 tensor cores, and 590.7 for FP8.

25.13 What a forward pass actually moves

Enough theory. Take Llama 3 70B, quantised to 8 bits, and ask what physically happens when it produces one token.

Every weight in the model participates. There is no way to produce a token without consulting all 70.6 billion parameters, so all 65.8 GB of them must travel from HBM into the SMs. At 3.35 TB/s that trip takes 21.1 ms and cannot be made faster by any amount of compute.

The arithmetic performed on those weights is about two operations per parameter — one multiply, one add — so roughly 141.0 GFLOP. At the FP8 tensor-core rate that takes 0.071 ms.

An intensity of 2.0 against a ridge of 590.7 puts this workload 295.0× into the memory-bound region. Not slightly below the ridge. Not marginally misconfigured. Two and a half orders of magnitude away from the regime the hardware was designed for.

25.14 Prefill and decode

Generation has two phases with completely different characters, and conflating them is the most common source of confused performance analysis.

Prefill processes your prompt. Every token of it is available at once, so they go through the model together as a matrix, not a vector. The weights are still read once — but now they serve hundreds or thousands of tokens.

Decode generates the response, one token at a time, each depending on the last. There is no way to parallelise across tokens you have not generated yet. The weights are read once per token.

The same weights, on the same card, in the same request, cross the ridge somewhere around a prompt of 512 tokens. Prefill is a compute problem. Decode is a memory problem. They want different batch sizes, different parallelism strategies and different hardware, and section D is largely the story of serving systems slowly accepting that.

25.15 Why decode is memory-bound — the short version

If you remember one paragraph from this chapter, this is it.

25.16 The KV cache

Weights are not the only thing decode must read. Attention needs the keys and values of every previous token, and recomputing them each step would be quadratic madness, so they are cached. That cache is the second great consumer of memory, and unlike the weights it grows.

For Llama 3 70B the cache costs 320.0 KB per token. That figure is already the product of a memory-saving design: with ordinary multi-head attention it would be 2560.0 KB, and grouped-query attention — sharing each key-value head across several query heads — cuts it by 8.0×.

A single 128k-token conversation needs 40.0 GB of KV cache — a substantial fraction of an entire 80 GB card, for one user. This is why long context is expensive in a way that has nothing to do with the model being large and everything to do with memory the model never had at training time.

25.17 What actually fits on one card

Now put the two consumers together, and a widely-believed piece of folklore falls over.

Llama 3 70B at 16-bit precision is 131.5 GB of weights. An H100 has 80.0 GB of HBM. The model does not fit. It overflows by 51.5 GB — more than half a card — before a single byte of KV cache, and needs 2 cards minimum.

With 14.2 GB of headroom, KV cache at 320.0 KB per token allows about 5 simultaneous 8k-token conversations — and 0 at 128k. Not few. Zero. One long conversation does not fit alongside the model it is talking to.

25.18 TTFT, TPOT, and goodput

Because the two phases differ, one latency number cannot describe a serving system. The standard vocabulary separates them.

25.19 MFU and MBU

One more pair, because using the wrong one will send you optimising the wrong half of the machine.

MFU — model FLOPs utilisation — is the fraction of peak arithmetic you achieved. MBU — model bandwidth utilisation — is the fraction of peak memory bandwidth you achieved. Take a server producing a perfectly respectable 25.0 tokens per second on our 70B model:

Both are correct. The MFU is genuinely near zero, because decode is not trying to use the arithmetic units and never was. Judging a decode server by MFU is like judging a lorry by its top speed: you will conclude something true and entirely beside the point.

25.20 Batching, the only real lever

If decode is slow because reading the weights costs 21.1 ms and the arithmetic is nearly free, there is one obvious response: make that read serve more than one token.

Run n sequences at once. The weights are still read once. Each read now produces n tokens instead of one, so arithmetic intensity becomes 2n instead of 2, and throughput rises almost linearly while the cost per user stays flat.

Batch 64 delivers 65.0× the throughput of batch 1 with no loss in per-user speed — the closest thing to a free lunch anywhere in this chapter. It is free precisely because the machine was idle: you are filling waiting time, not stealing from anyone.

It stays free until batch 512, where intensity finally crosses the ridge and decode becomes compute-bound. Past that point per-user speed starts falling, because the arithmetic units are genuinely saturated and users are now competing for them.

25.21 Static batching wastes the card

The naive implementation collects n requests, runs them together and returns when all are finished. It is simple, and it wastes most of what batching just bought.

Requests finish at different times. One asks for twenty tokens, another for two thousand. Under static batching the short request's slot sits idle until the longest one completes, and a batch of 32 that started full might be doing useful work in three slots by the end. Meanwhile new requests wait for the whole batch to drain.

25.22 Continuous batching

Orca (Yu et al., OSDI 2022) made the fix that now underpins every serious serving engine: schedule at the granularity of an iteration rather than a request.

After every single decode step, the scheduler removes finished sequences and admits waiting ones. The batch is rebuilt continuously, so a slot freed by a completed request is reused on the very next step rather than at the end of the batch. No sequence waits for another to finish.

One complication: attention cannot be batched the way the feed-forward layers can, because each sequence has a different length and a different KV cache. So engines use selective batching — batch the big matrix multiplications where the weights are shared, run attention per sequence.

25.23 Chunked prefill

Continuous batching creates a new problem. Prefill and decode now share a scheduler, and they are not comfortable neighbours.

A 4,000-token prefill occupies the GPU for a long, compute-bound stretch. Every user currently streaming tokens stalls for the duration. One arriving long prompt produces a visible hitch in everyone else's output — a latency spike with no obvious cause in the logs of the request that suffered.

Sarathi (Agrawal et al., 2023) proposed splitting prefill into fixed-size chunks and mixing a chunk into each iteration alongside the decodes. A step now carries a slice of prefill work and the ongoing decodes, so prefill makes steady progress without ever monopolising a step.

25.24 Prefill/decode disaggregation

Chunked prefill manages the conflict. The more radical answer is to stop running the two phases on the same hardware at all.

We established in 25.14 that prefill is compute-bound and decode is memory-bound. They want different batch sizes, different parallelism and arguably different cards. DistServe (Zhong et al., 2024) and NVIDIA's Dynamo run two pools: prefill workers produce the KV cache, ship it over the interconnect, and decode workers stream the output.

25.25 Paged attention and prefix caching

Since KV cache is the binding constraint, how it is allocated matters as much as how big it is.

The naive approach reserves a contiguous block per sequence, sized for the maximum possible length. A request that might generate 4,000 tokens but actually generates 200 wastes 95% of its reservation, and because the blocks are contiguous, the free memory between them fragments into unusable gaps.

PagedAttention (Kwon et al., 2023 — the paper that launched vLLM) borrowed the answer from operating systems: allocate the KV cache in fixed-size pages, keep a block table mapping logical positions to physical pages, and let a sequence's cache be scattered across memory. Waste falls to at most one partial page per sequence, and fragmentation disappears.

25.26 Engines and orchestrators

Two categories that get conflated constantly, to the confusion of anyone choosing between them.

You need both, and they solve different problems. An engine cannot help you when one replica holds the cached prefix for a conversation and the request lands on another; that is a routing decision. An orchestrator cannot help you when your kernels are inefficient.

25.27 Quantisation

Section D fought for more tokens per weight read. Section E attacks the read itself.

If decode time is the time to stream the weights, then halving the bytes per weight halves decode time. Not approximately — exactly, because the relationship is direct.

Going from 16-bit to 8-bit gives 2.0× the decode speed and frees 65.7 GB for KV cache, which raises the batch size, which raises throughput again. The two effects compound.

25.28 GQA, and Gemma's sliding window

Quantisation shrinks the weights. The KV cache needs its own answer, and this is where model architecture starts making decisions on the hardware's behalf.

Grouped-query attention is the now-universal first step: share each key-value head across several query heads. Llama 3 70B uses 8 KV heads for 64 query heads, cutting KV cache by 8.0× against full multi-head attention.

Gemma asks a more radical question. Every layer in a standard transformer keeps a cache that grows without bound — but does every layer need to see the whole conversation? Gemma's answer is no, and it interleaves two kinds of layer: local layers that attend only within a fixed window, and global layers that see everything. Only the global ones keep a cache that grows.

Gemma 3 holds that conversation in 3.7× less KV cache than Llama 3 70B, and the difference is qualitative rather than incremental: on the same leftover HBM, Llama fits 0 such conversations and Gemma 3 fits 1. Cannot, versus can.

Notice also the direction of travel between generations. Gemma 2 used a 1:1 pattern with a 4096-token window, saving 1.9×. Gemma 3 moved to 5:1 with a 1024-token window, saving 5.8×. The same team, tightening the same screw, one release later.

25.29 Gemma 4: three more attacks

Gemma 4 keeps the 5:1 pattern and adds three further mechanisms, each aimed at the same bottleneck. They are worth understanding individually, because they are independent and you will meet them separately in other models.

Asymmetric head dimensions. Global layers use a head dimension of 512 while local layers use 256 — 2× wider for the layers that must remember everything. This makes the global layers more expensive, which looks backwards until you notice there are only 5 of them out of 30. Spend capacity where memory is needed; withhold it where the window already caps the cost.

K=V sharing. Setting keys and values to the same tensor halves the cache exactly — 2.0×, by construction.

Shared KV layers. Consecutive layers can reuse one cache rather than each keeping its own. Combined with K=V sharing this reaches 2.5×.

25.30 Mixture of experts

The last idea in this section is the sharpest illustration of the chapter's whole thesis, and Gemma 4 ships it as 26B-A4B.

A mixture-of-experts model replaces each feed-forward block with many "experts" and routes each token to a small subset. The model has 26.0 billion parameters in total, but any given token only activates 4.0 billion of them — 6.5× sparsity.

Now apply everything this chapter has established. Decode time is the time to read the weights you need. A dense 26.0B model would read 24.2 GB per token, taking 7.8 ms. The MoE reads only 3.7 GB, taking 1.2 ms — 6.5× faster.

25.31 Training: where the constraint flips

Everything so far has been about inference, where bandwidth is the binding constraint and capacity is a secondary irritation. Training inverts that, and the inversion catches people who learned their intuitions from serving.

Training is compute-bound in the way inference is not. A training step processes a whole batch of sequences at once, so arithmetic intensity is high and the tensor cores genuinely work. The rule of thumb is 6N FLOPs per token for a model of N parameters — roughly 2N for the forward pass and 4N for the backward pass, which must compute gradients with respect to both activations and weights.

What breaks instead is capacity. The weights are the smallest part of what a training step must hold.

25.32 Optimiser state and activations

Train a 70.6B model with mixed precision and Adam, and count what must be resident:

That is 16.0 bytes per parameter against the 2 bytes inference needs — and it requires at minimum 14 H100s before you have stored a single activation or processed a single token.

Activations are the fourth consumer, and the one you can actually trade against compute. Every intermediate tensor from the forward pass is needed during the backward pass. Gradient checkpointing discards most of them and recomputes them on the way back — typically around 30% more compute for a large reduction in memory. On the memory-constrained side of the problem, that is usually a bargain.

25.33 The parallelism menu

Once one card is not enough, you must split the work, and the choice of how is dictated almost entirely by what each option costs in communication.

Tensor parallelism communicates at every layer of every step, so it is only viable where bandwidth between GPUs is enormous — which means inside a single NVLink-connected node. Pipeline parallelism only passes activations at stage boundaries, so it tolerates slower links and is what you use to cross between nodes. That is not a preference; it follows from the numbers in the next section.

25.34 Interconnect, and why topology decides

Every step away from the silicon costs an order of magnitude.

NVLink is 7.0× faster than PCIe, and HBM is another 3.7× faster than NVLink. This hierarchy explains the physical design of AI datacentres: eight GPUs in a node on NVLink, nodes connected by InfiniBand, and a parallelism strategy chosen to keep the chattiest communication inside the fastest tier.

25.35 A decision guide

The chapter's practical residue. Diagnose first, then choose.

25.36 Key takeaways

25.37 Drills

Fourteen questions in the order the chapter built them. Work out the answer before opening each one — several have a plausible wrong answer that is the whole reason the question is worth asking.

Expand all