Chunked Prefill

Chunked prefill is a model serving technique that splits a long input prompt into smaller groups of tokens and processes them over several steps, mixed with token generation for other requests. Without it, an inference engine that admits a long prompt spends a whole step on that prompt, and every response already streaming to other users pauses until it finishes. The 2023 SARATHI paper introduced chunked prefills, and its follow-up Sarathi-Serve built "stall-free" scheduling on them. DeepSpeed-FastGen's Dynamic SplitFuse uses the same idea, and chunked prefill is now on by default in vLLM and available in SGLang and TensorRT-LLM, the engines that serve many open-weight large language models (LLMs).

Prefill, Decode, and Generation Stalls#

Autoregressive transformer inference has two phases:

• Prefill: The model processes every prompt token in parallel, computes their keys and values, and stores them in the KV cache. Prefill does many operations per byte of weights read, so it is usually compute-bound.
• Decode: The model generates one token per step for each request, reading the whole cache each time. Decode does little arithmetic per byte read, so it is usually limited by memory bandwidth.

A GPU step that contains a 20,000-token prefill takes far longer than a step of decodes, so every request in the batch waits for it. Sarathi-Serve calls these pauses generation stalls, and they show up as spikes in inter-token latency for users whose own prompts are short.

How Chunked Prefill Works#

The scheduler gives each step a token budget, set in vLLM by max_num_batched_tokens. It first adds one token for every running decode, then fills the remaining budget with chunks of waiting prompts. A prompt that does not fit continues in the next step, extending its KV cache chunk by chunk until the whole prompt is processed and it starts decoding. Mixing the two phases in one step also uses the GPU better, since decodes leave compute idle that prefill chunks can fill:

Chunking changes only when prompt tokens are processed. Each chunk attends to the cached keys and values of every earlier chunk plus its own tokens under a causal mask, so the result equals a one-pass prefill, as one attention layer in PyTorch shows:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
prompt_len, chunk = 1000, 256
q, k, v = torch.randn(3, 1, 8, prompt_len, 64).unbind(0)  # one layer: 8 heads, 64 dims each

# One-shot prefill: all prompt tokens attend causally in a single pass
full = F.scaled_dot_product_attention(q, k, v, is_causal=True)

# Chunked prefill: each chunk attends to the cached keys and values of earlier chunks plus itself
outputs = []
for start in range(0, prompt_len, chunk):
    end = min(start + chunk, prompt_len)
    mask = torch.ones(end - start, end, dtype=torch.bool).tril(diagonal=start)  # causal over absolute positions
    q_chunk, k_cached, v_cached = q[..., start:end, :], k[..., :end, :], v[..., :end, :]
    outputs.append(F.scaled_dot_product_attention(q_chunk, k_cached, v_cached, attn_mask=mask))
chunked = torch.cat(outputs, dim=2)

print(f"{len(outputs)} chunks, same result as one pass: {torch.allclose(full, chunked, atol=1e-5)}")

It prints 4 chunks, same result as one pass: True: the 1,000-token prompt is processed as chunks of 256, 256, 256, and 232 tokens, and the outputs match the single pass.

Related Serving Techniques#

• Continuous batching: Adds and removes requests at every step instead of waiting for a whole batch to finish. Chunked prefill makes long prompts schedulable inside that loop.
• PagedAttention: Stores the KV cache in fixed-size pages, so a prompt that is prefilled over several steps can grow its cache without reserving memory for its full length up front.
• Prompt caching: Reuses the cached keys and values of a repeated prompt prefix, which removes that part of the prefill entirely.
• Disaggregated prefill and decode: Runs the two phases on separate GPU pools and transfers the KV cache between them, so each pool can be scaled and tuned for its own bottleneck.
• Speculative decoding: Speeds up the decode phase by drafting several tokens and verifying them in one pass.

Applications#

Chunked prefill matters most where long and short prompts share a server. A retrieval-augmented generation (RAG) assistant that fills its context window with a 50-page report can prefill it in chunks while other users' chats keep streaming. Vision-language models (VLMs) add hundreds or thousands of image tokens to each prompt, so camera pipelines that send frames for description produce exactly that mix. The Ultralytics LLM interface sends text and images to any OpenAI-compatible endpoint, including a self-hosted vLLM server, where a YOLO26 pipeline's requests to describe the frames in which it detects people are prefilled in chunks alongside other users' traffic.

Tuning the Token Budget#

The budget trades two latency metrics against each other. A small budget leaves little room for prefill in each step, which keeps inter-token latency low for streaming users and delays the time to first token for new prompts, and a large budget does the reverse. Each later chunk of a long prompt also re-reads the growing KV cache of the earlier chunks, so very small chunks add memory traffic. Operators tune the budget against median and tail inference latency, time to first token, and throughput on realistic prompt lengths. The vLLM optimization guide cites 2,048 tokens as a budget that favors inter-token latency and recommends more than 8,192 tokens for maximum throughput with smaller models on large GPUs.