Retrieval Augmented Generation (RAG)

Retrieval-augmented generation (RAG) is a technique in which a large language model (LLM) answers a question using passages retrieved at query time from an external knowledge source and inserted into its prompt next to the question. An LLM's training data stops at a cutoff date and never included a company's internal documents, so without retrieval it either lacks the facts or invents plausible ones, a failure known as hallucination. The term comes from the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and colleagues at Facebook AI Research, which paired a pretrained BART generator with a neural retriever over a dense vector index of Wikipedia and set the state of the art on three open-domain question answering tasks. Today the name covers any system that retrieves first and generates second.

How a RAG Pipeline Works#

RAG runs in two stages. Indexing happens ahead of time: documents are split into passages, for example by semantic chunking, each passage is converted into an embedding, and the vectors are stored in a vector database. At query time, the question is embedded with the same model, vector search returns the closest passages, and the system builds a prompt from instructions, those passages, and the question. The LLM then writes an answer grounded in the supplied text and can cite the passage behind each claim.

Production systems add steps around this core. Hybrid search combines vector similarity with keyword matching so exact terms such as part numbers are not missed, a reranker reorders the top candidates with a more precise model, and contextual retrieval has an LLM prepend a short note to each chunk that situates it within its document before indexing. Frameworks such as LangChain and LlamaIndex package these components.

The retrieval and augmentation steps fit in a few lines of Python with the Sentence Transformers library:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Knowledge base: passages the LLM never saw during training
passages = [
    "Badge access to Building C requires a safety briefing completed within the last 12 months.",
    "The cafeteria in Building A serves breakfast from 7:00 to 9:30 on weekdays.",
    "Forklift operators must renew their certification every three years.",
]
passage_embeddings = model.encode(passages)

# Retrieve: embed the question and keep the most similar passage
question = "How often do I need to redo forklift training?"
scores = model.similarity(model.encode(question), passage_embeddings)[0]
context = passages[int(scores.argmax())]

# Augment: put the retrieved passage in the prompt the LLM will answer from
prompt = f"Answer using only this context.\nContext: {context}\nQuestion: {question}"
print(prompt)

It prints a prompt containing the forklift certification passage, which the question matches by meaning: it asks about redoing training, and the passage talks about renewing certification. Any LLM can answer from that prompt, and the passages can change daily without retraining the model.

RAG vs. Fine-Tuning and Long Context#

• Fine-tuning: Updates the model's weights on new examples. It changes behavior, style, and output format well, but it is a slow way to add facts that change often and leaves no source to cite, so many systems fine-tune for format and use RAG for knowledge.
• Long context windows: Models that accept a million tokens can read a large document set directly, but every query pays for every token, and accuracy drops as inputs grow longer, an effect known as context rot. RAG sends only the passages that matter.

Several variants extend the basic pattern: GraphRAG retrieves from a knowledge graph, agentic RAG lets an agent decide when and where to search, and multimodal RAG retrieves images, video, and audio as well as text.

Where RAG Is Used#

• Enterprise chatbots: Internal assistants answer from policies, wikis, and past support tickets and link the paragraph each answer came from.
• Customer support: Assistants retrieve the manual for a customer's exact product model and that customer's order history before replying.
• Healthcare: Clinical assistants retrieve current guidelines and drug references, so answers point to a checkable source instead of the model's memory.

RAG also answers questions about video. An LLM cannot watch hours of footage for every question, so combining RAG with computer vision turns footage into retrievable records first. Ultralytics YOLO26 in tracking mode reports each object's class, box, and persistent track ID in every frame, which a pipeline stores as rows, so a question such as "How many trucks entered dock 3 this morning?" retrieves those rows instead of video.

Why RAG Answers Go Wrong#

RAG fails in two places. A retrieval failure means the passage holding the answer never reached the prompt, because chunking split it, the embedding model missed a synonym, or it ranked just below the top-k cutoff; it is measured as recall at k over a set of test questions. A generation failure means the right passage arrived but the model ignored, misread, or contradicted it. Evaluation libraries such as Ragas score both sides: context recall for the retriever, and faithfulness for the generator, defined as the share of claims in the answer that the retrieved context supports.