Prompt Compression

Prompt compression shortens the input to a large language model (LLM) while keeping the information the model needs to answer, so a request uses fewer tokens, runs faster, and fits inside the context window. Methods range from deleting predictable words with a small language model, to summarizing long histories, to training compact learned tokens that stand in for whole passages. Compression pays off when prompts are long and sent often, as in retrieval-augmented generation (RAG) pipelines that insert many retrieved documents, AI agents whose histories grow with every step, and applications with long instructions and examples. Providers bill per input token, and the compute of attention grows with input length, so a prompt compressed by half roughly halves input cost and reduces inference latency.

How Prompt Compression Works#

Hard prompt compression turns text into shorter text that any LLM, including a closed API model, can read. Most methods score each token with a small language model and drop the tokens it finds easy to predict, since those carry the least information. Selective Context applies this idea to words and phrases, and Microsoft's LLMLingua adds a budget controller that assigns different compression ratios to the instruction, examples, and question, plus iterative token-level pruning, reaching up to 20x compression with little loss in performance. LLMLingua-2 recasts compression as token classification with a bidirectional encoder such as XLM-RoBERTa, trained on compression examples distilled from a larger LLM, which runs 3x to 6x faster than earlier methods and speeds up end-to-end latency by 1.6x to 2.9x at 2x to 5x compression. Abstractive methods ask a model to summarize the context instead, which compresses more but can drop details the question needs.

Soft prompt compression turns text into a few learned embeddings that the model reads in place of the original tokens. Gist tokens train LLaMA-7B and FLAN-T5-XXL to squeeze an instruction into a handful of special tokens, achieving up to 26x compression and up to 40% fewer FLOPs. These methods require access to the model's weights, so they apply to open models.

The core of token-level compression fits in a few lines of PyTorch. This example uses the 124-million-parameter GPT-2 from Hugging Face Transformers to compute each token's self-information, the negative log probability of the token given everything before it, and keeps the more surprising half:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
prompt = (
    "The quarterly maintenance report states that, as of the end of the quarter, a total of three conveyor "
    "belts on line 4 were found to have worn bearings, and all of them were replaced on March 12 by the night shift."
)
ids = tokenizer(prompt, return_tensors="pt").input_ids[0]

# Self-information of each token, -log p(token | preceding tokens), under a small language model
with torch.no_grad():
    logits = model(ids[None]).logits[0, :-1]
info = -logits.log_softmax(-1).gather(1, ids[1:, None]).squeeze(1)

# Keep the first token and the more surprising half of the rest, in their original order
keep = torch.cat([torch.tensor([True]), info > info.median()])
compressed = tokenizer.decode(ids[keep])
print(f"{len(ids)} -> {keep.sum().item()} tokens: {compressed}")

It prints 46 -> 23 tokens: The quarterly maintenance report states, a three convey on line 4 found worn bearings all replaced on March 12 by night shift. The facts survive, including line 4, worn bearings, March 12, and the night shift, while predictable filler such as "as of the end of the quarter" disappears. The fragment "convey", left from "conveyor belts", is a typical artifact of pruning single tokens, which is why some methods prune whole words or phrases.

Compressing Images#

Vision-language models turn each image into hundreds of visual tokens, many of them redundant. FastV prunes half of the image tokens after the second layer and cuts FLOPs by 45% for LLaVA-1.5-13B without sacrificing performance, and token merging applies a similar idea inside vision transformers. An object detector can also act as a compressor for images: Ultralytics YOLO26 reduces a photo to Results.summary(), a list with each object's class name, confidence, and box coordinates, so an LLM pipeline that needs only what objects are present and where can send a few dozen tokens per object instead of an image that costs hundreds to more than a thousand tokens.

Prompt Compression vs. Related Techniques#

• Prompt caching: Stores the model's computed states for a repeated prompt prefix so they are not recomputed. The prompt keeps its length, and the two techniques combine well, compressing the variable part of a prompt and caching the fixed part.
• KV cache compression: Evicts or quantizes the key-value states inside the model during generation. Prompt compression works on the input before the model sees it.
• Prompt engineering: People write clear, concise prompts by hand. Compression is automatic and applies to content no one edits, such as retrieved documents.
• Context engineering: The broader practice of deciding what enters a model's context at each step. Compression is one of its tools, alongside retrieval and filtering.
• Prompt enrichment: Adds retrieved or generated context to a prompt. Pipelines often enrich first and compress the result.

Real-World Applications#

• RAG pipelines: Retrieved passages are compressed before generation, which fits more sources into the budget and keeps the model focused on the sentences that answer the question.
• Agent histories: Coding and research agents compact their history when the context fills, replacing old tool outputs and turns with a summary, because accuracy degrades as long contexts fill with stale material.
• Few-shot prompts: Long lists of worked examples compress well. LLMLingua's budget controller first drops whole demonstrations, ranked by perplexity, then prunes individual tokens while keeping 85% of the instruction and 90% of the question in the paper's default settings.