Hallucination (in LLMs)

A hallucination in a large language model (LLM) is generated content that reads as fluent and confident but is false, unsupported by the model's sources, or inconsistent with its own input. Hallucinations range from an invented citation or a wrong date to a summary that adds claims the source document never made, and they are hard to catch because a wrong answer carries the same tone as a right one. In Mata v. Avianca (2023), a US federal court sanctioned two lawyers for filing a brief that cited six court decisions ChatGPT had invented. Confabulation is a synonym preferred by some researchers, because the model fills gaps in its knowledge with plausible text.

Types of Hallucination#

• Intrinsic hallucinations: Output that contradicts the provided source, such as a summary that gives a different approval year than the document it summarizes. The survey by Ji et al. uses this term and the next to classify errors in summarization, dialog, and translation.
• Extrinsic hallucinations: Output that adds information the source cannot confirm, which may or may not be true.
• Factual errors in open-ended answers: Without any source, the model states wrong facts, fabricates references, or blends details from several people into one biography.
• Package hallucinations: Code assistants import libraries that do not exist. A study of 576,000 generated code samples from 16 models found that at least 5.2% of suggested packages from commercial models and 21.7% from open-source models were hallucinated, covering 205,474 unique fake package names that attackers could register with malicious code.
• Object hallucinations: Vision-language models (VLMs) describe objects that are not in the image.

Why LLMs Hallucinate#

An LLM generates text by predicting the next token from probabilities learned during pretraining, with no built-in check against a database of facts. Kalai et al. (2025) at OpenAI trace hallucinations to two statistical pressures. In pretraining, facts seen only once, such as an obscure person's birthday, cannot be learned reliably: if 20% of birthday facts appear exactly once in the training data, a base model should be expected to hallucinate on at least 20% of birthday questions. After training, most benchmarks grade answers as right or wrong and give "I don't know" zero points, so a model that guesses outscores one that abstains, and post-training tunes models into confident guessers.

Other factors add to these. Sampling with a high temperature or wide nucleus sampling can pick low-probability continuations, accuracy drops as relevant facts get buried in long inputs (context rot), and models trained on human approval tend to accept a false premise in the question, a form of AI sycophancy.

Detecting and Reducing Hallucinations#

• Retrieval-augmented generation (RAG): The system retrieves passages from a trusted collection and instructs the model to answer from them. A reranker improves which passages reach the prompt, though the model can still misread or ignore them.
• Grounding with citations: Each claim must cite a retrieved passage, and a checker, often a second model, confirms that the passage supports it.
• Semantic entropy: The model answers the same question several times, the answers are clustered by meaning, and high entropy across clusters flags a likely confabulation.
• Chain-of-verification (CoVe): The model drafts an answer, writes verification questions about its own claims, answers them independently, and revises the draft.
• Rewarding abstention: Training with RLHF or other feedback that penalizes confident errors more than "I don't know," and scoring evaluations the same way, removes the incentive to guess.

A grounded question-answering system combines several of these checks in a loop:

Hallucination vs. Related Concepts#

• Bias in AI: A systematic skew in outputs, such as favoring one group. A response can be free of bias and still hallucinated, or factually correct and still biased.
• Overfitting: Memorizing training examples at the cost of generalization. Hallucinations often come from the opposite situation, where the model fills in a fact it saw too rarely to memorize.
• Model collapse: The loss of rare knowledge when models train on generated data, which can make later models hallucinate more on long-tail facts.

Object Hallucination in Vision-Language Models#

VLMs lean on language priors, so a captioning model may mention a fork on a dining table that has none because forks so often appear next to plates in training captions. Object hallucination in image captioning is measured with the CHAIR metric, the fraction of objects mentioned in a caption that are absent from the image's ground-truth annotations. When a VLM draws boxes instead, each hallucinated object becomes a false positive, and the VLM benchmark results report each model's precision on real-world detection datasets. One mitigation is to pass the language model a list of objects that a detector actually found. The snippet below runs YOLO26 on an image and turns its detections into a prompt constraint:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Per-class detection counts as text, used as facts the language model must stay within
detections = result.verbose()
prompt = f"Describe this image. Mention only these detected objects: {detections}"
print(prompt)

result.verbose() returns one count per detected class, so every object named in the prompt has a bounding box behind it, and an answer that mentions any other object can be flagged before it reaches the user.