LLMOps

LLMOps (large language model operations) is the set of practices and tools for building, releasing, and running applications on large language models (LLMs) in production. It covers versioning prompts and model choices, evaluating generated text before and after release, tracing multi-step requests through retrieval and tool calls, controlling latency and token costs, and guarding against failures such as hallucinations and prompt injection. LLMOps is the branch of machine learning operations (MLOps) for these applications, and the term came into wide use in 2023 as companies moved chatbot prototypes into products. Most teams adapt a hosted or open-weight foundation model through prompt engineering, retrieval-augmented generation (RAG), or fine-tuning instead of training one from scratch, which moves most of the operational work from training pipelines to everything around the model.

How LLMOps Differs from MLOps#

The release unit is larger than a set of weights. One change can touch the prompt template, the model name and version, sampling settings, the retrieval index, and the tool definitions, and each piece is versioned because any one of them alters behavior. Prompt management tools store templates with version numbers that an application fetches at runtime, so a prompt edit can ship and roll back without a code release. Hosted models can also change underneath an application, so teams pin dated model snapshots where providers offer them and rerun their tests before switching. Data work changes shape too: a RAG application versions the documents, chunking settings, and embedding model behind its search index, and switching embedding models means re-embedding the whole corpus. Fine-tuned variants are often stored as small LoRA adapters that one server loads on top of a shared base model.

Evaluation is harder because generated text rarely has one correct answer. Teams keep a golden dataset of representative inputs with reference answers or grading rubrics, and score outputs with exact checks, human review, or LLM-as-a-judge, in which a strong model grades responses. In the 2023 paper Judging LLM-as-a-Judge, Lianmin Zheng and colleagues found that GPT-4 agreed with human preferences more than 80% of the time, as often as humans agree with each other. Every candidate release runs through the same suite, and production traffic keeps adding cases:

Cost and latency are managed per request. Hosted models bill per token, so teams cut spend with prompt caching, semantic caching, and model routing that sends easy requests to smaller models. A related pattern puts a cheap model in front of an expensive one: the YOLO → VLM Monitor template in Ultralytics Platform Agents runs YOLO26 first and calls the vision-language model only for images in which YOLO detects a person, with language-model usage billed to the workspace's own provider key. Safety work runs alongside: AI guardrails filter inputs and outputs on every request, and red teaming probes for jailbreaks before each release.

Tracing, Gateways, and Serving#

Observability for LLM applications is built on traces. Each request becomes a tree of spans for the prompt, every model call, each retrieval from a vector database, and each tool call by an AI agent, with token counts, latency, and cost attached. OpenTelemetry's generative AI semantic conventions standardize the attribute names for models and token usage, and tools such as LangSmith, Langfuse, Arize Phoenix, and MLflow Tracing store and search the traces. Traces also feed online evaluation: sampled production answers are scored with the same judges and rubrics as the offline suite, and each user's thumbs-up or thumbs-down is attached to the trace that produced the answer.

An AI gateway sits between applications and model providers, holding API keys and enforcing rate limits, fallbacks, and spend budgets in one place. Teams that host open-weight models run them on inference servers such as vLLM and SGLang, which use continuous batching and paged KV cache memory to serve many concurrent requests on each GPU.