Language Modeling

Language modeling is the task of assigning a probability to a sequence of words or tokens. By the chain rule of probability, P(w1, w2, ..., wn) = P(w1) x P(w2 | w1) x ... x P(wn | w1, ..., wn-1), so a model that predicts each next token from the ones before it defines the probability of any text, and sampling from those predictions generates new text. That next-token objective trains every GPT model and most large language models (LLMs), and smaller language models ranked candidate outputs in speech recognition and machine translation systems for decades before LLMs and still suggest the next word on phone keyboards.

How a Language Model Generates Text#

A trained model reads the prompt, scores every token in its vocabulary, and turns the scores into next-token probabilities with a softmax. A decoding rule then picks one token, the model appends it, and the loop repeats until it emits an end token or reaches a length limit, which is how text generation works:

Sampling with a temperature or nucleus sampling trades some likelihood for variety. Because each token depends on the ones already written, a reply arrives one token at a time. The Ultralytics LLM interface passes stream=True through to the provider, so a YOLO26 pipeline that asks a model to describe a detected scene can print each text delta as it arrives.

From N-Grams to Neural Models#

Claude Shannon built word and letter approximations of English from n-gram statistics in his 1948 paper on information theory. An n-gram model estimates P(next word | previous n-1 words) by counting how often word sequences occur in a corpus, and IBM speech recognition systems of the 1970s and 1980s relied on them. A bigram model, with n = 2, shows the idea in a few lines of Python:

from collections import Counter

sentences = ["the cat sat on the mat .", "the dog sat on the rug .", "the cat saw the dog ."]
corpus = " ".join(sentences).split()

# Count word pairs to estimate P(next word | previous word)
pairs = Counter(zip(corpus, corpus[1:]))
firsts = Counter(corpus[:-1])

# The model's next-word distribution after "the"
print({nxt: round(n / firsts["the"], 2) for (prev, nxt), n in pairs.items() if prev == "the"})

# Chain rule: a sentence's probability is the product of its next-word probabilities
sentence = ["the", "dog", "sat", "on", "the", "mat"]
p = 1.0
for prev, nxt in zip(sentence, sentence[1:]):
    p *= pairs[(prev, nxt)] / firsts[prev]
print(f"P(dog sat on the mat | the) = {p:.4f}")

After "the", the model predicts "cat" and "dog" with probability 0.33 each and "mat" and "rug" with 0.17 each. It gives "dog sat on the mat" after "the" a probability of 0.0278 although that sentence never appears in the corpus, because every word pair in it does. Any pair the corpus lacks gets probability zero, so practical n-gram models need smoothing methods such as Kneser-Ney, and they still cannot use context beyond n-1 words.

Yoshua Bengio and colleagues' 2003 neural probabilistic language model replaced counts with learned embeddings and a neural network, so similar words share statistical strength. Recurrent neural networks (RNNs) and LSTMs extended the usable context, and the transformer, with self-attention over the whole context window, became the standard architecture after 2017. GPT-3 showed in 2020 that a 175-billion-parameter language model could learn new tasks from examples in its prompt.

Causal and Masked Language Modeling#

Causal, or autoregressive, language modeling predicts each token from the tokens to its left and is the objective behind generative models. Masked language modeling hides some tokens and predicts them from context on both sides. BERT trains this way on 15% of its input tokens, and the two-sided context makes it strong at understanding tasks such as classification and named entity recognition while leaving it unable to write text left to right. Diffusion language models extend masking into a generator that fills many positions in parallel over several steps.

Training and Evaluation#

Training is self-supervised learning: raw text supplies its own labels, and the model minimizes the cross-entropy loss function between its predicted distribution and each actual next token. The exponential of that average loss is perplexity, the standard intrinsic metric, and scaling laws describe how the loss falls as models, data, and compute grow. A pretrained base model only continues text, and instruction tuning and reinforcement learning from human feedback (RLHF) turn it into an assistant that follows requests.

Language Modeling vs. Related Terms#

• Language model and LLM: A language model is any model trained on this task, from a bigram table to a transformer. An LLM is a language model with billions of parameters trained on trillions of tokens.
• Natural language processing (NLP): The whole field of computing with human language, of which language modeling is one task.
• Visual autoregressive modeling (VAR): Applies the same next-step prediction to images, generating token maps from coarse to fine resolution scales instead of words.