Foundation Model

A foundation model is a large machine learning model trained on broad data at scale that can be adapted to a wide range of downstream tasks. Stanford researchers introduced the term in the 2021 report On the Opportunities and Risks of Foundation Models, choosing "foundation" to describe models that are central to many applications yet incomplete on their own. Earlier models were usually trained for one task on one labeled dataset. A foundation model learns general patterns once, in an expensive pretraining run, and then serves as the starting point for chatbots, search, translation, image classification, segmentation, and many other applications built by teams that never train such a model themselves. Large language models (LLMs) such as GPT, Claude, Gemini, and Llama are foundation models, as are vision models such as CLIP, DINOv3, and Meta's Segment Anything Model (SAM).

How Foundation Models Work#

Pretraining uses objectives that need no human labels, so the training set can grow to trillions of words of text or billions of images. Language models learn by predicting the next token, vision encoders by self-supervised learning methods such as masked image modeling and self-distillation, and vision-language models by contrastive learning on image-caption pairs. Nearly all foundation models use the transformer architecture, and their performance improves predictably with parameters, data, and compute, as described by scaling laws.

After pretraining, the same model is adapted to each task in one of several ways. Prompting steers it with instructions or examples and trains nothing, fine-tuning or parameter-efficient fine-tuning (PEFT) updates its weights on task data, and a small head trained on its frozen embeddings turns it into a classifier or retriever:

The report highlighted two consequences of this pattern. Emergence means abilities appear at scale without being trained for directly, as when GPT-3 learned to perform new tasks from a few examples in its prompt. Homogenization means many applications now build on a few models, so a model's errors and biases pass into everything adapted from it. Regulators now address these models directly: the EU AI Act calls them general-purpose AI models, and its obligations for their providers have applied since August 2, 2025.

Foundation Models in Computer Vision#

Vision foundation models are trained on far larger and broader data than task-specific detectors. CLIP learned from 400 million image-caption pairs and classifies images into categories named only in text, the basis of zero-shot learning in vision. Meta's DINOv3 produces frozen features for depth estimation, segmentation, and satellite mapping from a 7-billion-parameter model trained on 1.7 billion unlabeled images. SAM, whose authors described it as a foundation model for image segmentation, was trained on the SA-1B dataset of 11 million images and 1.1 billion masks and segments any object indicated by a point or box, and SAM 3 adds text prompts that find every instance of a named concept. Vision-language-action models extend the pattern to robotics, mapping camera images and instructions to robot actions.

Ultralytics Platform uses these models to speed up data annotation: in Smart annotation, a click prompts SAM 2.1, SAM 3, or SAM 3.1 for an object's mask, and on detection datasets, class-prompted models such as SAM 3.1 Text and Grounding DINO find every object of a class from its name. Meta's SAM 2.1 also runs in the Ultralytics Python package, where a box prompt replaces any task-specific training:

from ultralytics import SAM

# Meta's SAM 2.1, a segmentation foundation model, used as-is with no training
model = SAM("sam2.1_b.pt")

# A box prompt around the bus defines the task; the model returns that object's mask
results = model("https://ultralytics.com/images/bus.jpg", bboxes=[22, 231, 805, 757])
result = results[0]
result.save(filename="sam_bus_mask.jpg")

Foundation Models vs. Related Terms#

• Large language model (LLM): A foundation model for text. Foundation models also cover images, video, audio, tabular data, and robot control.
• Generative AI: Systems that create text, images, audio, or video. Many are built on foundation models, and some foundation models, such as DINOv3 and SAM, produce features and masks instead of new content.
• Transfer learning: The technique of reusing a pretrained model for a new task. A foundation model is a pretrained model built to be reused this way across many tasks, while a model trained for one task with a fixed label set, such as a COCO object detector with 80 classes, transfers mainly to closely related tasks.
• Frontier AI models: The most capable foundation models at a given time. Under the EU AI Act, a general-purpose AI model trained with more than 10^25 floating-point operations is presumed to pose systemic risk and faces extra obligations.