GPT (Generative Pre-trained Transformer)

GPT (Generative Pre-trained Transformer) is a decoder-only transformer language model that is first pretrained on large amounts of unlabeled text to predict the next token and then adapted to tasks. OpenAI introduced the approach in June 2018 with Alec Radford and colleagues' paper Improving Language Understanding by Generative Pre-Training, and the name spells out the recipe: a generative model of text, pretrained before any task-specific training, built on the transformer. The GPT series grew from 117 million parameters in GPT-1 to 175 billion in GPT-3, and ChatGPT, launched on November 30, 2022, made it the most widely known family of large language models (LLMs). "GPT-style" also describes any decoder-only LLM trained the same way, including Meta's Llama and Alibaba's Qwen models.

How a GPT Model Works#

Text is first split into tokens; GPT-2 introduced the byte-level tokenization with a 50,257-token vocabulary that GPT-3 kept. Each token becomes an embedding plus a position signal, and a stack of identical decoder blocks refines those vectors. In every block, masked self-attention lets each position read only itself and the positions before it, and a feed-forward layer transforms each position on its own:

Because of the causal mask, one forward pass over a 1,024-token training sequence yields 1,024 next-token predictions at once, which makes pretraining parallel even though generation is sequential. At inference, the model scores the next token, appends one, and repeats, reusing earlier computation through a KV cache. OpenAI's 124-million-parameter GPT-2 shows the first step:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tokenizer("The capital of France is", return_tensors="pt")

# One forward pass scores every vocabulary token as the next token
with torch.no_grad():
    logits = model(**inputs).logits[0, -1]
top = logits.softmax(dim=-1).topk(5)
for p, i in zip(top.values, top.indices):
    print(f"{tokenizer.decode(i)!r}: {p:.3f}")

# Generation repeats that step, appending one token at a time
output = model.generate(**inputs, max_new_tokens=8, do_sample=False, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0]))

GPT-2 gives " the" a probability of 0.085 and " Paris" only 0.032, and greedy decoding writes "The capital of France is the capital of the French Republic, and". A pretrained GPT continues text the way its training data would, and instruction tuning followed by reinforcement learning from human feedback (RLHF) turns it into an assistant that answers the question.

From GPT-1 to GPT-6#

GPT-1 pretrained a 12-layer decoder on more than 7,000 unpublished books and then fine-tuned it per task, improving the state of the art on 9 of the 12 tasks studied. GPT-2, announced in February 2019, scaled to 1.5 billion parameters trained on 40 GB of web text and performed tasks zero-shot; OpenAI released its weights in stages through November 2019 over concerns about misuse. GPT-3, in May 2020, showed few-shot learning from examples placed in the prompt, a result that scaling laws had anticipated.

InstructGPT and then ChatGPT added human feedback training, and GPT-4 accepted images as input in March 2023. OpenAI's o-series reasoning models, starting with o1 in September 2024, learned to write a long chain of thought before answering, and GPT-5, released on August 7, 2025, put a fast model and a reasoning model behind a router. Two days earlier, OpenAI had released gpt-oss-120b and gpt-oss-20b under the Apache 2.0 license, its first open-weight language models since GPT-2. The GPT-6 family arrived in September 2026: GPT-6 Astra on September 3, GPT-6 Sol and GPT-6 Luna on September 22, and GPT-6.1 Sol on September 29.

GPT in Computer Vision#

The same next-token recipe applies to images. OpenAI's Image GPT trained GPT-2-scale models to predict pixels one at a time in 2020, visual autoregressive modeling (VAR) predicts images scale by scale, and many vision-language models (VLMs) attach an image encoder to a GPT-style decoder that writes the answer. Text encoders borrow the design too: OpenAI's CLIP encodes text with a 63-million-parameter, 12-layer transformer that uses GPT-2's architecture and masked self-attention. Tencent AI Lab's YOLO-World, which runs in the Ultralytics package, calls that CLIP text encoder when set_classes(["person", "bus"]) is called, turning each class name into an embedding the detector matches against image regions, so the list of classes can change without retraining. OpenAI's own GPT models also read images, and the VLM Leaderboard measures how accurately GPT-6.1 Sol and other vision language models detect objects in real-world datasets.

GPT vs. Related Terms#

BERT uses the transformer encoder without a causal mask, so every token sees the whole input; it suits classification and retrieval and does not generate text. An LLM is any large language model, and GPT names both OpenAI's model series and the decoder-only design most LLMs share. A chatbot such as ChatGPT is an application built on GPT models, with a system prompt, conversation history, and tools wrapped around the model.