Speculative Decoding
Speculative decoding has a small draft model propose tokens that the large model checks in one pass, speeding up LLM output without changing it.
Speculative decoding is an inference technique that speeds up text generation in large language models (LLMs) by letting a small, fast draft model guess several upcoming tokens and having the large target model check all of them in a single forward pass. Every accepted guess is a token the large model did not have to produce one step at a time, and an acceptance rule guarantees that the output follows exactly the same probability distribution as the target model alone. Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google introduced it in Fast Inference from Transformers via Speculative Decoding in November 2022, reporting 2-3x faster T5-XXL generation with identical outputs. Charlie Chen and colleagues at DeepMind published the same idea as speculative sampling in February 2023, with a 2-2.5x speedup on the 70-billion-parameter Chinchilla.
Why Decoding Leaves Room to Speculate#
Autoregressive generation produces one token per forward pass, and every pass reads all of the model's weights from memory. For a single sequence, that memory traffic sets the pace: a GPU spends most of each step waiting on memory bandwidth, so scoring five tokens in one pass takes almost as long as scoring one. Speculative decoding spends that idle compute on verification. Many tokens are easy to predict, such as the rest of a common phrase, a closing bracket in code, or a name copied from the prompt, and a draft model a small fraction of the target's size gets them right most of the time. The result is lower inference latency for each response.
The Draft-and-Verify Loop#
Each round has three steps. The draft model generates k candidate tokens, which is cheap because it is small. The target model then processes all k drafts in one pass, reusing its KV cache for the earlier context, and returns its own probability for every drafted position. Finally, the drafts are checked from left to right. A draft token x is accepted with probability min(1, p(x) / q(x)), where p is the target's probability and q is the draft's. At the first rejection, a replacement token is sampled from the leftover distribution max(0, p - q), renormalized, and the remaining drafts are discarded. If all k drafts survive, the target's prediction for the next position comes free from the same pass, so one round yields between 1 and k + 1 tokens.
With greedy decoding, the test reduces to a match check that keeps drafts while they equal the target's top token. If each draft is accepted with probability a, a round produces on average (1 - a^(k+1)) / (1 - a) tokens, so at a = 0.8 and k = 4 the target model advances about 3.4 tokens per forward pass.
This PyTorch simulation applies the acceptance rule to one decoding step and counts which tokens come out:
import torch
torch.manual_seed(0)
p = torch.tensor([0.50, 0.30, 0.15, 0.05]) # target model's next-token probabilities
q = torch.tensor([0.40, 0.20, 0.20, 0.20]) # draft model's probabilities for the same step
# Draft 100,000 tokens from q and accept each with probability min(1, p / q)
draft = torch.multinomial(q, 100_000, replacement=True)
accepted = torch.rand(len(draft)) < (p[draft] / q[draft]).clamp(max=1)
# A rejected token is replaced by a sample from the leftover mass max(0, p - q)
residual = (p - q).clamp(min=0)
fallback = torch.multinomial(residual / residual.sum(), len(draft), replacement=True)
tokens = torch.where(accepted, draft, fallback)
print(f"accepted drafts: {accepted.float().mean():.3f}")
print(f"output frequencies: {torch.bincount(tokens) / len(tokens)}")The run accepts 80.1% of drafts, matching the overlap between the two distributions, the sum of min(p, q), which is 0.80. The output frequencies of 0.5003, 0.2985, 0.1503, and 0.0510 reproduce the target distribution p, even though most tokens were proposed by the draft.
Draft Sources and Related Techniques#
The draft does not have to be a separate network:
- Small language model from the same family: A model with the same tokenizer, released alongside the large one, is the classic choice.
- Medusa heads: Extra decoding heads on the target model predict several future tokens from its last hidden state and verify them with tree attention.
- Prompt lookup: Draft tokens are copied from matching n-grams in the prompt, which works well for summarization, retrieval-augmented answers, and code edits that repeat their input.
Knowledge distillation trains a small model to stand in for a large one, and it is also a common way to train drafts whose guesses the target accepts more often. Model quantization lowers the precision of the target's own weights, which changes its outputs slightly, and quantized drafts are a frequent pairing.
Serving engines such as vLLM, SGLang, TensorRT-LLM, and llama.cpp ship speculative decoding, and Hugging Face Transformers offers it as assisted generation. A YOLO26 pipeline that hands its detections to a language model through the Ultralytics LLM interface can point base_url at such a local server, so its generated reports stream back sooner with the same output distribution. The gain is largest at small batch sizes: under heavy traffic, continuous batching already fills the GPU with other requests, leaving little spare compute for verification.










