Large Concept Models (LCMs)

A large concept model (LCM) is a language model that predicts the next sentence-level embedding, called a concept, instead of the next token. Meta FAIR introduced the architecture in December 2024 in Large Concept Models: Language Modeling in a Sentence Representation Space, treating each sentence as one concept and modeling a document as a sequence of sentence vectors. Those vectors come from SONAR, a fixed sentence embedding space that supports text input and output in 200 languages and speech input in 76, so the model in the middle reasons over meaning without being tied to one language's wording.

How Large Concept Models Work#

A conventional large language model (LLM) runs tokenization and predicts one subword at a time. An LCM splits the input into sentences, encodes each with the frozen SONAR encoder into a vector of 1,024 numbers, and feeds that sequence to a transformer that predicts the next sentence vector. Text generation repeats that step until the model signals the end of the document, and the frozen SONAR decoder turns each predicted vector back into a sentence in any supported language:

The core is an ordinary causal transformer whose sequence elements are sentence vectors. The simplest variant, Base-LCM, trains it to regress each next vector with a mean squared error loss, as in this PyTorch sketch of one training step:

import torch
from torch import nn

torch.manual_seed(0)

# A document of 6 sentences, each already encoded as a 1024-number SONAR-sized concept vector
concepts = torch.randn(1, 6, 1024)

# A causal transformer over sentence vectors, the core of a Base-LCM
layer = nn.TransformerEncoderLayer(d_model=1024, nhead=16, batch_first=True)
lcm = nn.TransformerEncoder(layer, num_layers=2)
causal = nn.Transformer.generate_square_subsequent_mask(5)

# Each position predicts the next sentence's vector, trained by regression instead of a softmax over tokens
predicted = lcm(concepts[:, :-1], mask=causal, is_causal=True)
loss = nn.functional.mse_loss(predicted, concepts[:, 1:])
loss.backward()
print(f"Next-concept predictions: {tuple(predicted.shape)}")

It prints Next-concept predictions: (1, 5, 1024): five predictions, each a full 1,024-number sentence vector, where a token-level model would output five probability distributions over its vocabulary. Meta's training data averaged about 19 tokens per sentence, so an LCM processes sequences roughly 19 times shorter than an LLM does for the same text.

Phrase-level vectors of this kind also drive open-vocabulary detection. Ultralytics YOLOE encodes each class prompt passed to set_classes(), such as "double-decker bus", into a single MobileCLIP text embedding and matches image regions against those vectors, so one embedding stands for the whole phrase.

Base, Diffusion, and Quantized Variants#

A regression loss pulls each prediction toward the average of all plausible next sentences, which can decode to bland or blurred text. Meta therefore also built diffusion LCMs, which start the next vector from noise and denoise it step by step conditioned on the previous sentences, applying diffusion models to embeddings. The One-Tower version uses a single transformer for context and denoising, and the Two-Tower version separates them. Quant-LCM instead discretizes SONAR space with residual vector quantization and predicts the resulting codes much as an LLM predicts tokens.

Meta ran its comparisons with 1.6-billion-parameter models trained on about 1.3 trillion tokens, then scaled the Two-Tower diffusion LCM to 7 billion parameters trained on 142.4 billion sentences, or 2.7 trillion tokens, and released the training code. On zero-shot text summarization with the XLSum dataset across 42 languages, the 7B LCM scored a Rouge-L of 23.5 in English against 20.7 for Llama-3.1-8B-Instruct.

Limits#

Compressing a sentence into 1,024 numbers can blur exact names, numbers, and code, and the frozen decoder can only reproduce what the embedding preserves. Treating one sentence as one concept is a practical choice, since a sentence can hold several ideas and an idea can span several sentences. The fixed SONAR space also means the LCM cannot represent anything SONAR's encoder never learned to encode, and three of XLSum's 45 languages were left out of the evaluation for that reason.

LCMs vs. Related Terms#

• Large language models: Predict a distribution over tokens. An LCM performs language modeling in a sentence embedding space and relies on a separate encoder and decoder for text.
• Joint embedding predictive architecture (JEPA): Another Meta approach that predicts representations in place of raw data, mainly for images and video.
• Latent reasoning: Reasoning carried out in a model's hidden states without writing every step as text.
• Concept bottleneck models: Classifiers that route predictions through human-named attributes such as "has wings", for explainable AI (XAI).
• Latent consistency models: Share the LCM abbreviation but are distilled image latent diffusion models that generate in a few steps.