LlamaIndex

LlamaIndex is an open-source framework for connecting large language models (LLMs) to private data: it loads documents, databases, and APIs, indexes their content, and retrieves the passages a model needs to answer a question. Most applications built with it use retrieval-augmented generation (RAG), such as assistants that answer from a company's manuals, contracts, or tickets, and AI agents that read and act on documents. Jerry Liu released the project as GPT Index in November 2022 and renamed it LlamaIndex in February 2023. The LlamaIndex framework is available in Python and TypeScript, and its company also runs LlamaParse, a hosted service for turning complex PDFs, tables, and slides into text that indexes well.

How LlamaIndex Works#

A LlamaIndex application has two phases. Ingestion runs once per data update: connectors from LlamaHub load files and services as Document objects, a node parser splits each into chunks called Nodes, and an embedding model converts every Node into a vector, its embedding, stored in an index. Querying runs per question: a retriever finds the Nodes closest to the question, postprocessors such as a reranker reorder or filter them, and a response synthesizer places the survivors in the LLM's context window with instructions to answer from them.

The VectorStoreIndex is the default, storing embeddings in memory or in an external vector database. Other indexes send every chunk to the model for summaries, build a knowledge graph of entities and relations for GraphRAG-style queries, or match keywords. A query engine answers a single question, a chat engine keeps a conversation, and Workflows, LlamaIndex's event-driven orchestration layer, connect retrieval, tools, and several agents into agentic RAG systems that decide what to search and when.

Indexing Images by What They Contain#

Retrieval works on any content that can be written as text. Running Ultralytics YOLO26 over a photo collection turns each image into a searchable description: Results.verbose() returns the per-class counts of a predict mode result as a short string, which becomes a Document whose metadata keeps the image URL. A local Hugging Face embedding model then lets the index run without an API key:

from llama_index.core import Document, Settings, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from ultralytics import YOLO

Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
model = YOLO("yolo26n.pt")

# Describe each image by its detections; the URL stays in metadata and out of the embedding
docs = []
for url in ["https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/zidane.jpg"]:
    results = model(url)
    text = f"A photo containing {results[0].verbose()}"
    docs.append(Document(text=text, metadata={"image": url}, excluded_embed_metadata_keys=["image"]))

# Index the descriptions, then retrieve the image that best matches a question
index = VectorStoreIndex.from_documents(docs)
retriever = index.as_retriever(similarity_top_k=1)
nodes = retriever.retrieve("Which photo shows a bus?")
print(nodes[0].node.metadata["image"], round(nodes[0].score, 3))

The retriever returns the bus image with its cosine similarity score, found through the detector's description rather than the file name. Adding a response synthesizer on top answers questions about the collection in prose, the pattern behind multimodal RAG systems that pair computer vision with document search.

LlamaIndex vs. Related Tools#

• LangChain: A general framework for LLM applications and agents with a broad set of integrations. Both projects now cover retrieval and agents; LlamaIndex goes deeper on ingestion, parsing, and index types, and teams often use its retrievers inside LangChain or LangGraph agents.
• Vector databases: Store vectors and run vector search. LlamaIndex sits above them, deciding how documents are parsed, chunked, embedded, retrieved, and passed to the model, and it supports dozens of vector stores as backends.
• Semantic search: The retrieval step on its own. LlamaIndex adds generation, so results arrive as an answer with the source Nodes attached for citation.