GPT-3

GPT-3 (Generative Pre-trained Transformer 3) is a 175-billion-parameter large language model (LLM) that OpenAI described in May 2020 in the paper Language Models are Few-Shot Learners. It was more than 100 times larger than its predecessor GPT-2, and its headline result was in-context learning: given a task description and a few examples in the prompt, the same frozen model could translate, answer questions, or do arithmetic without any fine-tuning. OpenAI never released the weights and offered GPT-3 only through its API, launched in June 2020. Instruction-tuned descendants of GPT-3 became the GPT-3.5 models behind ChatGPT, which made the GPT family a household name in late 2022.

Architecture and Training#

GPT-3 is a decoder-only transformer with the same design as GPT-2, scaled up. The largest model has 96 layers, a width of 12,288, 96 attention heads of 128 dimensions each, and a context window of 2,048 tokens, and its layers alternate between dense attention and locally banded sparse attention. It reuses GPT-2's byte-level BPE tokenization with a 50,257-token vocabulary. The paper trained eight sizes, from 125 million to 175 billion parameters, each on 300 billion tokens drawn from filtered Common Crawl (60% of the training mix), WebText2 (22%), two book corpora (8% each), and English Wikipedia (3%), using V100 GPUs on a cluster provided by Microsoft. Those hyperparameters account for the model's size and training cost:

# GPT-3 175B: layers, width, and context from the paper's Table 2.1, plus GPT-2's vocabulary
n_layers, d_model, vocab_size, context = 96, 12288, 50257, 2048

# Each decoder block holds 4 * d^2 attention weights and 8 * d^2 feed-forward weights (4x hidden width)
blocks = n_layers * 12 * d_model**2
embeddings = (vocab_size + context) * d_model  # token and learned position embeddings
params = blocks + embeddings
print(f"Parameters: {params / 1e9:.1f} billion ({blocks / params:.1%} in the decoder blocks)")

# Training compute: about 6 FLOPs per parameter per token, over 300 billion training tokens
flops = 6 * params * 300e9
print(f"Training compute: {flops:.2e} FLOPs = {flops / (1e15 * 86400):,.0f} petaflop/s-days")

It prints 174.6 billion parameters, 99.6% of them in the decoder blocks, and 3.14e+23 FLOPs, or 3,637 petaflop/s-days, matching the 174,600 million parameters and 3.14e+23 FLOPs listed in the paper's appendix.

Few-Shot and In-Context Learning#

The paper evaluated three prompt settings. Zero-shot gives only an instruction, one-shot adds a single worked example, and few-shot adds as many examples as fit in the context window, typically 10 to 100. Performance rose with both model size and the number of examples: on the TriviaQA question answering benchmark, the 175-billion-parameter model scored 64.3% zero-shot, 68.0% one-shot, and 71.2% few-shot, the last beating fine-tuned systems in the same closed-book setting. Few-shot prompting made prompt engineering a standard way to use LLMs, and the results supported the scaling laws view that larger models learn more from context. The same prompt-instead-of-retraining idea reaches object detection in YOLOE-26, whose visual prompts use boxes drawn around examples in a reference image to define a class that the model then finds in new images without any weight update.

From GPT-3 to ChatGPT#

Microsoft received an exclusive license to GPT-3's underlying model in September 2020, and OpenAI's Codex, a GPT-3 descendant fine-tuned on public code, powered the first version of GitHub Copilot in 2021. Base GPT-3 continued text well but followed instructions poorly, so OpenAI applied instruction tuning and reinforcement learning from human feedback (RLHF). Labelers preferred the outputs of the resulting 1.3-billion-parameter InstructGPT model over those of the 175-billion-parameter GPT-3. That line led to GPT-3.5 and ChatGPT in November 2022, then to the multimodal GPT-4 in March 2023. Meta's OPT-175B and BigScience's BLOOM reproduced GPT-3-scale models with open weights in 2022.

GPT-3 vs. Related Models#

• GPT-2: The 1.5-billion-parameter predecessor from 2019, with the same architecture and tokenizer at about one-hundredth the size and a 1,024-token context.
• GPT-4: The 2023 successor, which accepts images as well as text and follows instructions far more reliably after post-training.
• BERT: Google's 2018 encoder-only model, which reads text in both directions and suits classification and extraction tasks.

Limitations#

The GPT-3 paper lists many of the model's weaknesses itself. It repeats itself and loses coherence over long passages and makes basic errors in physical common sense, and like later LLMs it states false facts with confidence, now called hallucinations. The authors measured gender, race, and religion biases in its completions, and they found that some benchmark test sets had leaked into the web training data. Its text generation was also hard to tell from human writing: evaluators identified 200-word news articles written by the 175-billion-parameter model only 52% of the time. OpenAI shut down the original ada, babbage, curie, and davinci API models on January 4, 2024, replacing them with babbage-002 and davinci-002.