Serving with vLLM

6.1 Why a serving engine at all

You can run a model with fifty lines of PyTorch. Doing that in production wastes most of a very expensive GPU, and understanding exactly how is the foundation for every parameter in this chapter.

The naive loop looks reasonable: take a request, tokenise it, run the model forward one token at a time until it emits a stop token, return the text. It is correct. It is also perhaps 5% efficient, for three separate reasons that compound.

1. The GPU sits idle between tokens Generating one token requires reading every weight in the model from memory. For a 31B model in bf16 that is roughly 58 GiB read per token. The arithmetic performed on that data is trivial by comparison, so the GPU's compute units spend nearly all their time waiting for memory. Serving one request at a time means paying that entire read to produce a single token.
2. Requests wait for the slowest sibling Batch naively — group eight requests, run them together — and the batch finishes when its longest member finishes. Seven requests that wanted 50 tokens sit completed but unreturned while the eighth grinds through 2,000. Their KV cache stays allocated the whole time.
3. Memory is reserved for the worst case A naive implementation allocates KV cache for the maximum possible sequence length, because it cannot know how long the output will be. A request that generates 100 tokens holds space for 30,000. At scale this is where the GPU actually goes.

6.2 PagedAttention

The idea vLLM was built around, borrowed wholesale from how operating systems manage memory. It is worth understanding properly, because it explains why several later parameters exist at all.

The problem it solves is fragmentation. A KV cache must grow as a sequence generates tokens, and a naive implementation reserves one contiguous block per sequence sized for the worst case. Reserve 30,000 tokens of space for a request that produces 200 and you have wasted 99% of that allocation — and because it is contiguous, no other request can use the hole.

6.3 Continuous batching

Traditional batching groups requests and waits. Continuous batching lets each request leave the moment it is done, and admits a new one into the vacant slot on the next step.

Because generation happens one step at a time for every sequence simultaneously, there is a natural boundary between steps at which the batch can be rebuilt. vLLM rebuilds it every step. A sequence that emits its stop token is removed and its blocks are freed immediately; a waiting request is admitted into the freed capacity.

6.4 Prefill and decode are different machines

Inference has two phases with opposite performance characteristics. Almost every tuning mistake comes from treating them as one thing.

6.5 Where 96 GB actually goes

Before touching a single flag, work out your memory budget on paper. Almost every serving problem is a memory problem wearing a disguise, and five minutes of arithmetic prevents a day of guessing.

An H100 96 GB card divides into four claims, in this order:

1. Model weights — fixed, and you pay for all of them Parameters times bytes per parameter. At full bf16 precision that is 2 bytes each. For a 31B dense model, about 57.7 GiB. For the 26B MoE it is about 48.4 GiB, not 7.5 GiB — every expert must be resident even though only a few run per token. Sparsity buys compute, never memory. This is the single most expensive misunderstanding in MoE planning.
2. Activations and workspace — roughly 2 to 4 GiB Intermediate tensors for the current forward pass, plus CUDA graph capture buffers and the allocator's own headroom. It scales with batch size and hidden dimension rather than with model size.
3. KV cache — whatever is left, and the only part you control vLLM measures free memory after loading weights and claims essentially all of it. This number determines your maximum concurrency, and therefore your throughput.
4. The reserve — what gpu-memory-utilization holds back Space deliberately left unallocated so that fragmentation and transient spikes do not trigger an out-of-memory failure mid-request.

"""KV cache and batch sizing for Gemma 4 on an H100 96 GB.

Every number quoted in chapter 6 comes from here rather than from memory.
The arithmetic is simple but extremely easy to get wrong by a factor of two
(the classic error is forgetting the factor of 2 for K *and* V), and a wrong
KV figure means a wrong max_num_seqs, which means either wasted GPU or
crashes under load.

Model shapes are stated as assumptions at the top so a reader can substitute
the real config.json values for their checkpoint.
"""
from __future__ import annotations

from dataclasses import dataclass

GIB = 1024 ** 3


@dataclass(frozen=True)
class ModelShape:
    """The handful of config.json fields that decide KV cache size.

    Note it is the KV-head count that matters, not the attention-head count.
    Grouped-query attention shrinks KV heads while leaving query heads alone,
    and that ratio is exactly the KV cache saving.
    """

    name: str
    total_params_b: float          # billions, for weight memory
    active_params_b: float         # billions; differs from total only for MoE
    layers: int
    kv_heads: int                  # num_key_value_heads
    head_dim: int

    def weight_bytes(self, bytes_per_param: int = 2) -> int:
        """Weights are sized by TOTAL parameters, even for MoE.

        A common and expensive planning error: an MoE with 4B active still
        needs all 26B resident in HBM. Sparsity buys compute, not memory.
        """
        return int(self.total_params_b * 1e9 * bytes_per_param)

    def kv_bytes_per_token(self, bytes_per_elem: int = 2) -> int:
        """2 (K and V) x layers x kv_heads x head_dim x bytes."""
        return 2 * self.layers * self.kv_heads * self.head_dim * bytes_per_elem


# Shapes assumed for the worked examples. Substitute your checkpoint's values.
GEMMA4_31B = ModelShape("Gemma 4 31B (dense)", 31.0, 31.0, 48, 8, 128)
GEMMA4_26B = ModelShape("Gemma 4 26B-A4B (MoE)", 26.0, 4.0, 40, 8, 128)


def kv_budget_gib(shape: ModelShape, gpu_gib: int = 96,
                  gpu_util: float = 0.92, bytes_per_param: int = 2,
                  overhead_gib: float = 3.0) -> float:
    """GiB left for KV cache after weights and working memory.

    vLLM carves out gpu_memory_utilization of the card, loads weights, then
    gives essentially all the remainder to the KV cache. The overhead term
    covers activations, CUDA graphs and the allocator's own fragmentation.
    """
    usable = gpu_gib * gpu_util
    weights = shape.weight_bytes(bytes_per_param) / GIB
    return usable - weights - overhead_gib


def concurrent_seqs(shape: ModelShape, context_len: int, kv_gib: float,
                    kv_bytes_per_elem: int = 2) -> float:
    """How many sequences of this length fit in the KV budget at once."""
    per_seq = shape.kv_bytes_per_token(kv_bytes_per_elem) * context_len
    return kv_gib * GIB / per_seq


def report(shape: ModelShape, contexts=(15_000, 30_000), tp: int = 1) -> None:
    per_tok = shape.kv_bytes_per_token()
    print(f"\n{shape.name}   (TP={tp})")
    print(f"  weights (bf16, total params) : {shape.weight_bytes()/GIB:7.1f} GiB")
    print(f"  active params (compute cost) : {shape.active_params_b:7.1f} B")
    print(f"  KV per token                 : {per_tok/1024:7.1f} KiB")

    # With TP the weights and the KV cache both shard across GPUs.
    kv_gib = kv_budget_gib(shape) * tp + (tp - 1) * 0  # aggregate budget
    if tp > 1:
        usable = 96 * 0.92 * tp
        kv_gib = usable - shape.weight_bytes() / GIB - 3.0 * tp
    print(f"  KV budget (aggregate)        : {kv_gib:7.1f} GiB")

    for ctx in contexts:
        per_seq_gib = per_tok * ctx / GIB
        seqs = concurrent_seqs(shape, ctx, kv_gib)
        print(f"    ctx {ctx:>6,}: {per_seq_gib:5.2f} GiB/seq  ->  "
              f"{seqs:6.0f} concurrent sequences")


if __name__ == "__main__":
    for shape in (GEMMA4_31B, GEMMA4_26B):
        for tp in (1, 2, 4):
            report(shape, tp=tp)
    print("\n-- 31B with fp8 KV cache (1 byte per element) --")
    kv = kv_budget_gib(GEMMA4_31B)
    for ctx in (15_000, 30_000):
        print(f"    ctx {ctx:>6,}: "
              f"{concurrent_seqs(GEMMA4_31B, ctx, kv, 1):6.0f} concurrent sequences")

Running that for both models at TP 1, 2 and 4 produces the table this entire chapter leans on. The single-GPU rows are the ones that surprise people:

6.6 gpu-memory-utilization

The fraction of each GPU vLLM is allowed to claim. The most consequential single number you will set, and the one most often left at a value someone copied from a blog post.

The mechanism matters for understanding when to change it. vLLM profiles a forward pass at startup, measures what weights and activations consume, then allocates the remainder of its budget to KV cache and holds it for the process lifetime. It does not grow later. So this single number sets your concurrency ceiling permanently at boot.

6.7 max-model-len

The longest sequence the engine will accept, input plus output combined. It looks like a safety limit. It is really a memory allocation decision, and leaving it at the default is one of the most expensive mistakes available.

Setting it for the two scenarios

Your deployments are defined as input plus output totalling 15k and 30k. That makes the setting straightforward, with one caveat worth respecting.

6.8 KV cache dtype

You asked to run the models themselves at full precision. That constraint applies to the weights; the KV cache is a separate decision, and quantising it is the cheapest concurrency you can buy.

6.9 max-num-seqs

The ceiling on how many sequences can be in flight at once. Your batch size, and therefore the direct lever on decode throughput.

How to arrive at the right value

1. Compute the memory ceiling KV budget divided by (KV bytes per token × max-model-len). That is the hard limit, and it is what the calculator in 6.5 prints.
2. Subtract a safety margin Set max-num-seqs to roughly 80–90% of that ceiling. Running at exactly 100% guarantees preemption whenever requests run long, because your average request is shorter than the maximum but some are not.
3. Verify against the startup log vLLM reports the GPU blocks it allocated. Blocks × block size gives total cached tokens; divide by max-model-len for the true concurrency. If this disagrees with your arithmetic, trust the log and find out why.
4. Load-test and watch for preemption Sustained preemption warnings mean the value is too high. Zero preemption under peak load with memory to spare means it is too low, and you are leaving throughput on the table.

6.10 max-num-batched-tokens

How many tokens the scheduler may process in a single forward step. The companion to max-num-seqs, and the one that governs the prefill/decode balance you asked about.

Both limits apply simultaneously. The scheduler fills a step until it hits either the sequence cap or the token budget, whichever comes first. Understanding the interaction is what makes the setting tractable:

max_num_batched_tokens = 8192      max_num_seqs = 32

step N:  40 sequences decoding    -> capped at 32 by max_num_seqs
         32 decode tokens used
         8160 tokens of budget left over  -> spent on prefill chunks

step N+1: 32 decode tokens + one 8160-token prefill chunk
          = 8192 tokens, budget exactly full

6.11 Chunked prefill

The mechanism that stops a long prompt from freezing everyone else's generation. On by default in 0.27.1 — but the default token budget it operates under is not tuned for 15k or 30k contexts, so it needs your attention rather than your trust.

The problem it solves

Without chunking, a prefill is atomic. A 15,000-token prompt occupies an entire forward step, and every sequence currently generating waits for all of it. At perhaps 400 ms for that prefill, thirty users see their token stream freeze for nearly half a second because one other person submitted a long document.

Arriving at the right chunk size

Since the chunk is whatever the token budget leaves after decode, setting the chunk size means setting max-num-batched-tokens. Here is how to derive it rather than guess.

1. Decide your inter-token latency budget Interactive chat wants a step under about 50 ms so generation feels fluid. An offline job has no such constraint and should use large chunks.
2. Measure how many tokens fit in that budget Run a prefill-only benchmark at several chunk sizes and record milliseconds per step. On an H100 with a 31B model, expect very roughly 4,000–8,000 tokens in a 50 ms step; measure rather than trusting that range, because it moves with TP size and attention backend.
3. Add the decode tokens The budget must also cover one token per decoding sequence, so add max-num-seqs to your chunk target.
4. Round to a power of two and verify Then load-test with a realistic mix of long prompts and active generations, watching both time to first token and inter-token latency. Those two move in opposite directions; you are looking for the knee, not a maximum.

6.12 Prefix caching

Reuse the KV cache of a prompt prefix that has already been computed. For RAG workloads with a shared system prompt this is close to free throughput, and in 0.27.1 it is on by default.

6.13 Async scheduling and CUDA graphs

Two settings that reduce the overhead between forward passes rather than the cost of the passes themselves. Both matter more than they sound, because at high batch sizes the gaps add up.

Async scheduling

The gap it closes is small per step — a few milliseconds of Python deciding which sequences to run and building tensors. But with steps arriving many times per second, those milliseconds are a real fraction of your throughput. Leave this enabled. Disable it only when debugging, because it makes the ordering of events easier to follow.

CUDA graphs

6.14 TP, PP, DP and EP

Four ways to spread a model across GPUs. They solve different problems and combine, and choosing wrongly is expensive in a way that is hard to see afterwards.

6.15 MoE versus dense: 26B-A4B against 31B

Your two models look similar in size and behave completely differently. Understanding why decides which one to deploy for which job.

6.16 How speculative decoding works

The one technique that breaks the sequential nature of decoding. Done right it is close to free throughput; done wrong it makes things slower, and the difference is one number you have to measure.

Recall from 6.4 that decode is memory-bandwidth-bound: a forward pass reads every weight to produce one token, leaving the compute units idle. That idle compute is the opportunity. If you could guess the next few tokens cheaply, you could verify all of them in a single forward pass — for almost the same cost as verifying one.

drafter proposes 4:   "the"  "customer"  "should"  "return"
target verifies all in ONE forward pass:
                      "the"  "customer"  "can"     ...
                        OK      OK       reject
result: 3 tokens produced this step ("the", "customer", "can")
        instead of 1.  The last draft token is thrown away.

6.17 Assistant draft models

The most straightforward form of speculation: a small model from the same family proposes, the big model verifies. Gemma 4 ships purpose-built assistant checkpoints for exactly this.

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 4 \
  --speculative-config '{
      "model": "google/gemma-4-31B-it-assistant",
      "num_speculative_tokens": 4,
      "draft_tensor_parallel_size": 1
  }'

6.18 MTP and the other methods

A separate draft model is the obvious approach, not the best one. Multi-token prediction removes the second model entirely, and vLLM 0.27.1 supports it for Gemma 4 specifically.

6.19 Tuning acceptance

Speculation lives or dies on one measured number. This section is how to find it and what to do with it.

Choosing num_speculative_tokens

Every drafted token costs drafting time and makes verification slightly more expensive. The value of an extra draft token falls as you add more, because it only pays off if all preceding drafts were accepted. Acceptance compounds against you: at 70% per token, the fourth token is only accepted 24% of the time.

--speculative-config '{
    "model": "google/gemma-4-31B-it-assistant",
    "num_speculative_tokens": 5,
    "num_speculative_tokens_per_batch_size": [
        [1,  8, 5],     # quiet: draft aggressively
        [9,  24, 3],    # moderate load: back off
        [25, 64, 1],    # busy: minimal speculation
        [65, 512, 0]    # saturated: stop entirely
    ]
}'

6.20 The 15k deployment

Input plus output capped at 15,000 tokens. Everything from the previous nineteen sections, assembled into one command line with the reasoning for each flag.

vllm serve google/gemma-4-26B-A4B-it \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --max-num-seqs 108 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype auto \
  --speculative-config '{
      "method": "ngram",
      "num_speculative_tokens": 3,
      "prompt_lookup_max": 4,
      "num_speculative_tokens_per_batch_size": [
          [1, 16, 3], [17, 48, 2], [49, 512, 0]
      ]
  }'

6.21 The 30k deployment

Doubling the context does not double the difficulty — it roughly halves your concurrency, which changes several decisions at once. This is why it deserves a separate deployment rather than a larger limit on the same one.

vllm serve google/gemma-4-26B-A4B-it \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --max-num-seqs 54 \
  --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.94 \
  --kv-cache-dtype fp8 \
  --long-prefill-token-threshold 10240 \
  --speculative-config '{
      "method": "ngram",
      "num_speculative_tokens": 3,
      "prompt_lookup_max": 4,
      "num_speculative_tokens_per_batch_size": [
          [1, 12, 3], [13, 32, 2], [33, 512, 0]
      ]
  }'

6.22 Sizing a ten-million-request job

A different problem from serving. Nobody is waiting for any individual answer, so the objective is total wall-clock for the whole batch — and several settings invert.

The first job is an estimate good enough to choose a fleet size. The model below is deliberately simple: prefill is compute-bound and scales with input tokens, decode is bandwidth-bound and scales with output tokens divided by batch size. Both include a realistic efficiency factor rather than assuming peak hardware numbers.

"""Throughput and wall-clock planning for a 10-million-request batch job.

The question "how long will 10M inferences take" has no single answer, but
it has a defensible estimate, and the estimate is what tells you whether to
buy 8 GPUs or 64. This model is deliberately simple and states its
assumptions loudly: it is for choosing a fleet size and a configuration, not
for promising an SLA.

Two regimes matter and they behave completely differently:
  - PREFILL is compute-bound. Cost scales with total input tokens.
  - DECODE is memory-bandwidth-bound. Cost scales with output tokens and is
    helped enormously by batching, because the weights are read once per
    step regardless of how many sequences are in flight.
"""
from __future__ import annotations

from dataclasses import dataclass

H100_BF16_TFLOPS = 989.0     # dense bf16 with sparsity off, marketing peak
H100_HBM_TBS = 3.35          # HBM3 bandwidth, TB/s
MFU_PREFILL = 0.45           # realistic fraction of peak achieved on prefill
MBU_DECODE = 0.70            # realistic fraction of peak bandwidth on decode


@dataclass(frozen=True)
class Workload:
    requests: int
    input_tokens: int
    output_tokens: int


@dataclass(frozen=True)
class Deployment:
    name: str
    active_params_b: float       # drives BOTH flops and bytes moved per step
    total_params_b: float
    gpus: int
    batch: int                   # sequences decoded concurrently
    accept_rate: float = 0.0     # speculative tokens accepted per draft token


def prefill_seconds(w: Workload, d: Deployment) -> float:
    """Total prefill time across the fleet.

    Forward pass costs roughly 2 FLOPs per active parameter per token. For an
    MoE only the ACTIVE experts run, which is the entire point of the
    architecture.
    """
    total_tokens = w.requests * w.input_tokens
    flops = 2 * d.active_params_b * 1e9 * total_tokens
    fleet_flops = H100_BF16_TFLOPS * 1e12 * MFU_PREFILL * d.gpus
    return flops / fleet_flops


def decode_seconds(w: Workload, d: Deployment) -> float:
    """Total decode time across the fleet.

    Decode reads the whole weight matrix once per forward step and serves the
    entire batch from that single read, which is why batching is nearly free
    on decode and why small batches waste the GPU so badly.

    A tensor-parallel group of 8 GPUs behaves as one fast engine, not eight
    independent ones: each holds a shard and they read in parallel. So the
    fleet is modelled as (gpus / tp) independent replicas, each running its
    own batch.
    """
    tp = 8                                   # one replica per 8-GPU node
    replicas = max(1, d.gpus // tp)
    steps_per_seq = w.output_tokens / (1 + d.accept_rate)
    steps_per_replica = w.requests * steps_per_seq / d.batch / replicas
    # Each GPU reads its own 1/tp shard of the weights concurrently.
    bytes_per_step = d.total_params_b * 1e9 * 2 / tp
    seconds_per_step = bytes_per_step / (H100_HBM_TBS * 1e12 * MBU_DECODE)
    return steps_per_replica * seconds_per_step


def summarise(w: Workload, deployments: list[Deployment]) -> None:
    print(f"workload: {w.requests:,} requests x "
          f"({w.input_tokens:,} in / {w.output_tokens:,} out)")
    print(f"{'deployment':<34}{'prefill':>10}{'decode':>10}{'total':>10}"
          f"{'req/s':>10}")
    print("-" * 74)
    for d in deployments:
        p, dec = prefill_seconds(w, d), decode_seconds(w, d)
        total = p + dec
        print(f"{d.name:<34}{p/3600:9.1f}h{dec/3600:9.1f}h{total/3600:9.1f}h"
              f"{w.requests/total:10.1f}")


if __name__ == "__main__":
    JOB = Workload(requests=10_000_000, input_tokens=12_000, output_tokens=3_000)

    summarise(JOB, [
        Deployment("26B-A4B MoE, 8 GPU, batch 48", 4.0, 26.0, gpus=8, batch=48),
        Deployment("26B-A4B MoE, 8 GPU, batch 48, spec", 4.0, 26.0, gpus=8,
                   batch=48, accept_rate=1.6),
        Deployment("31B dense, 8 GPU, batch 32", 31.0, 31.0, gpus=8, batch=32),
        Deployment("31B dense, 8 GPU, batch 32, spec", 31.0, 31.0, gpus=8,
                   batch=32, accept_rate=1.6),
        Deployment("26B-A4B MoE, 64 GPU, batch 48", 4.0, 26.0, gpus=64, batch=48),
    ])

workload: 10,000,000 requests x (12,000 in / 3,000 out)
deployment                           prefill    decode     total     req/s
--------------------------------------------------------------------------
26B-A4B MoE, 8 GPU, batch 48           74.9h    481.2h    556.1h       5.0
26B-A4B MoE, 8 GPU, batch 48, spec     74.9h    185.1h    260.0h      10.7
31B dense, 8 GPU, batch 32            580.5h    860.7h   1441.1h       1.9
31B dense, 8 GPU, batch 32, spec      580.5h    331.0h    911.5h       3.0
26B-A4B MoE, 64 GPU, batch 48           9.4h     60.2h     69.5h      40.0

What the numbers say

1. The MoE wins decisively, and mostly on prefill 75 hours of prefill against 580 for the dense model — nearly eight times — exactly the ratio of active parameters. Since this workload is prefill-heavy, that difference dominates. Total job time is roughly 2.6× better even before speculation.
2. Decode dominates anyway, so batch size is the main lever Even for the MoE, decode is 481 hours against 75 for prefill. Decode time is inversely proportional to batch size, which means every memory decision that buys concurrency — TP, fp8 KV cache, a tighter max-model-len — converts directly into hours saved.
3. Speculation more than halves decode here 556 hours falls to 260. This is the one context where speculation is unambiguously a throughput win rather than a latency one: with 10M queued requests you can keep batches moderate and still saturate the fleet, so the idle compute speculation needs actually exists.
4. Scaling out is close to linear 8 GPUs to 64 takes 556 hours to 69.5 — a factor of exactly 8. Data parallelism has no inference-time communication, so replicas scale almost perfectly. This is the reliable lever: tune one replica properly, then multiply it.

6.23 Batch-mode tactics

With no user waiting, several settings move in the opposite direction from everything recommended so far. Here is the full inversion, and the tactics that only make sense offline.

Tactics that only apply offline

1. Sort requests by length before submitting The single highest-value trick in this section. Batching a 30k prompt alongside a 2k one wastes the padding difference on every step they share. Sorting into length-homogeneous groups can lift throughput 20–30% for nothing but a sort() before the job starts. This is only possible offline, because it requires seeing all the work before beginning it.
2. Group by shared prefix to exploit caching If requests share system prompts or retrieved documents, sort so identical prefixes are adjacent. Prefix caching from 6.12 then hits constantly rather than being evicted between unrelated requests. Combine with the length sort: group by prefix first, then sort by length within each group.
3. Use the offline LLM class, not the HTTP server LLM.generate() with a list of prompts skips HTTP, JSON serialisation and per-request overhead entirely. For 10M requests that overhead is real money. The server exists for interactive traffic; a batch job does not need it.
4. Cap max-tokens tightly and honestly The scheduler reserves against the maximum a request might generate. If your answers are 500 tokens, setting max_tokens=3000 "just in case" costs concurrency on every single request. Measure your real output distribution and set the cap just above the 99th percentile.
5. Checkpoint continuously A 40-hour job will meet a node failure. Write results incrementally, keyed by request id, and make the driver resumable so a restart skips completed work. This is ordinary batch engineering, and it is the difference between a delay and a disaster.

6.24 Defaults: what to keep, what to set

You asked which knobs to leave free and what goes wrong if you do. Here is the complete answer in one place, sorted by how much damage the default can do.

Always set these

Set these deliberately, after measuring

Leave these alone

6.25 Key takeaways

6.26 Interview drills

vLLM questions test whether you understand the memory-bandwidth argument or have only copied flags from a README. Every strong answer below explains the mechanism first and states the number second.

Expand all

Where this leaves you

You can now read a vLLM config with real intent behind it: work out a memory budget on paper, explain why a 26B MoE and a 31B dense model behave nothing alike despite similar parameter counts, size the scheduler to your actual context length rather than the model's advertised ceiling, and decide when speculation is a genuine win versus an expensive habit.

The thread through this chapter is the one from section 6.1: decoding is memory-bandwidth-bound, so almost every parameter is ultimately about making the batch as large as the memory allows, or about spending idle compute on something useful while that batch runs. Chunked prefill, prefix caching, tensor parallelism, fp8 KV cache and speculative decoding are five different answers to the same underlying question. Once you see them that way, a new flag in a future vLLM release is something you can reason about rather than something you have to look up.

That is also the honest limitation of everything here: defaults and even flag names change release to release, faster than almost anything else in this stack. The method in this chapter — compute the KV budget, understand which phase a workload spends its time in, measure acceptance and hit rates rather than assuming them — will outlive the specific numbers quoted for 0.27.1. Rerun the calculators in this chapter against your own checkpoint's config.json before trusting a number that was computed against a different model or a different vLLM version.