Chain of Verification (CoVe)

Chain of Verification (CoVe) is a prompting method that reduces hallucinations in large language model (LLM) output by having the model fact-check its own draft. The model writes a first answer, plans short verification questions that test each factual claim in it, answers those questions independently of the draft, and then writes a final answer that keeps only the claims the verification supports. Shehzaad Dhuliawala and colleagues at Meta AI introduced the method in the 2023 paper Chain-of-Verification Reduces Hallucination in Large Language Models. It works because models answer short, focused questions more accurately than they state the same facts inside a long answer: in the paper's list-question experiments, only about 17% of the entities Llama 65B listed were correct, yet the same model answered about 70% of the individual verification questions correctly.

How Chain of Verification Works#

CoVe runs four steps with the same model:

1. Draft a baseline response: The model answers the query normally. Asked to name politicians born in New York City, it might list Hillary Clinton, Donald Trump, and Michael Bloomberg.
2. Plan verification questions: The model reads its draft and writes one question per checkable claim, such as "Where was Hillary Clinton born?" and "Where was Michael Bloomberg born?"
3. Answer each question independently: The model answers the questions in fresh prompts that do not contain the draft, so it cannot simply repeat its earlier mistake. Clinton was born in Chicago and Bloomberg in Boston, while Trump was born in Queens.
4. Write the final revised response: The model revises the draft against the answers, dropping or correcting every claim that failed, which here leaves Trump on the list.

How strictly the third step isolates each question matters. The paper compared four variants: joint, where planning and answering happen in one prompt that still contains the draft; two-step, which answers all questions in a second prompt without the draft; factored, which answers every question in its own prompt; and factor+revise, which adds an explicit check of each answer against the original claim. Every variant that kept the draft out of the verification prompts hallucinated less than the joint variant, because a model that can see its own draft tends to repeat its errors.

Results#

With Llama 65B, CoVe raised precision on Wikidata list questions from 0.17 to 0.36 and lifted F1 on the closed-book MultiSpanQA benchmark from 0.39 to 0.48. On long-form biographies, the factor+revise variant raised FActScore, the percentage of atomic facts supported by Wikipedia, from 55.9 to 71.4, above the 58.7 reported for ChatGPT and the 61.6 for the retrieval-based PerplexityAI. The gains come at a cost in inference latency: the factored variants need one extra model call per verification question.

CoVe vs. Related Techniques#

• Chain-of-thought prompting: Asks the model to reason step by step before answering, which helps with logic and math. CoVe audits factual claims after a draft exists.
• Retrieval-augmented generation (RAG): Grounds the answer in retrieved documents from the start. The original CoVe relies only on the model's own knowledge, and the two combine naturally when verification questions are answered from a search engine or document index.
• Semantic entropy: Detects likely hallucinations by sampling several answers and measuring how much their meanings disagree. CoVe goes further and repairs the answer.
• AI debate: Two models argue opposite sides so a judge can spot errors. CoVe is a single model checking itself without an adversary.

Verifying Claims With Tools#

The verification questions do not have to be answered from the model's memory. Pipelines route them to tools that hold the ground truth: a search engine for public facts, a vector database of company documents, or a vision model for claims about an image. When a vision-language model writes an image caption, an Ultralytics YOLO26 object detection model can answer each "how many" question from the caption, independently of the draft:

from collections import Counter

from ultralytics import YOLO

# Claims extracted from a draft caption: "A bus, four people, and a bicycle on a street"
draft_counts = {"bus": 1, "person": 4, "bicycle": 1}

# Answer each verification question with a detector that never sees the draft
model = YOLO("yolo26n.pt")
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]
detected = Counter(result.names[int(c)] for c in result.boxes.cls)

# Keep claims the detector confirms and mark the rest for revision
for name, claimed in draft_counts.items():
    verdict = "keep" if detected[name] == claimed else f"revise to {detected[name]}"
    print(f"How many '{name}'? Draft: {claimed}, detector: {detected[name]} -> {verdict}")

Each printed line is one verification question with the draft's claim, the detector's independent answer, and the decision that the final caption applies.

Real-World Applications#

• Healthcare: Assistants that summarize medical literature verify drug names, dosages, and interactions with separate questions before a clinician sees the summary.
• Question answering: Search and support assistants check list answers, dates, and biographical facts, the claim types where the paper measured the largest gains.
• Manufacturing reports: An LLM drafting a shift report verifies each defect it mentions against the inspection system's logs before the report is filed.
• Document processing: Values extracted from contracts or invoices are re-asked one field at a time and compared with the first extraction.

Limitations#

CoVe catches errors the model makes inconsistently, and it cannot catch a false belief the model holds in every context, since the independent answer repeats the same mistake. It checks factual claims and leaves faulty reasoning between correct facts untouched, and it multiplies the number of model calls per answer. Even the best variant in the paper left 28.6% of the atomic facts in its biographies unsupported, the remainder of its 71.4 FActScore.