BERT (Bidirectional Encoder Representations from Transformers)

BERT (Bidirectional Encoder Representations from Transformers) is a transformer encoder that reads a whole sentence at once and builds each word's representation from the words on both sides of it. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova of Google AI Language introduced it in October 2018 in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, which won the NAACL 2019 best long paper award. A single pretrained BERT, fine-tuned with one extra output layer, set new state-of-the-art results on eleven natural language processing (NLP) tasks, raising the GLUE benchmark score to 80.5%, a 7.7-point absolute improvement, and SQuAD v1.1 question answering to a test F1 of 93.2.

How BERT Is Pretrained#

BERT-Base stacks 12 encoder layers with 768-dimensional hidden states and 12 attention heads, about 110 million parameters, and BERT-Large has 24 layers, 1,024 dimensions, 16 heads, and 340 million parameters. Input text is split by WordPiece tokenization into a 30,000-token vocabulary, a special [CLS] token opens every sequence, and [SEP] separates the two segments of a sentence pair. Google pretrained both sizes on BooksCorpus (800 million words) and English Wikipedia (2,500 million words) with two self-supervised objectives.

Masked language modeling (MLM) selects 15% of token positions, replaces 80% of them with [MASK], 10% with a random token, and leaves 10% unchanged, and the model predicts the original tokens. BERT's self-attention has no causal mask, so every prediction draws on context from both sides, which a left-to-right language model cannot do. Next sentence prediction (NSP) pairs each segment with its true successor half of the time and with a random sentence otherwise, and a classifier on the [CLS] output learns to tell the two apart.

A pretrained checkpoint keeps its masked-word head, so the original model can fill in a blank directly:

from transformers import pipeline

# BERT predicts a masked word from the words on both sides of it
fill = pipeline("fill-mask", model="bert-base-uncased")

for text in ["She went to the [MASK] to buy bread.", "She went to the [MASK] to see a doctor."]:
    predictions = fill(text, top_k=3)
    print(text, [(p["token_str"], round(p["score"], 2)) for p in predictions])

The two sentences differ only after the blank, and BERT's top prediction changes from "market" (0.82) to "hospital" (0.78). A model reading left to right sees "She went to the" in both sentences and has nothing to separate them.

Fine-Tuning and Applications#

Fine-tuning adds a small task layer and trains all weights on labeled examples, an approach that made transfer learning the default in NLP. The [CLS] vector feeds sentence-level classifiers such as sentiment analysis, every token vector feeds tagging tasks such as named entity recognition (NER), and start and end scores over the tokens extract answer spans.

Google began using BERT for 1 in 10 English-language searches in the US in October 2019, citing the query "2019 brazil traveler to usa need a visa," where BERT recognized that "to" describes a Brazilian traveling to the US. By October 2020, it ran on almost every English query. BERT-family encoders also produce the embeddings behind semantic search: Sentence-BERT fine-tunes BERT to map each sentence to one vector for vector search, ColBERT keeps one vector per token, and cross-encoder rerankers score query-passage pairs in retrieval-augmented generation (RAG) pipelines.

In vision-language detection, Grounding DINO encodes text prompts with a BERT-base encoder before matching them to image regions. Ultralytics Platform hosts Grounding DINO for class-prompted smart annotation, where it turns a dataset's class names into editable boxes for training a YOLO26 detector. Masked prediction carried over to images as well: the vision transformer (ViT) borrowed BERT's [CLS] token for classification, and masked autoencoders (MAE) pretrain image encoders by hiding 75% of image patches and reconstructing them.

BERT vs. GPT and Later Encoders#

GPT models are built from the transformer decoder with a causal mask, so they predict each next token from the tokens before it and suit text generation. BERT outputs a vector for every input token, which suits classification, extraction, and retrieval, and it has no built-in way to write text left to right. A large language model (LLM) with billions of parameters handles many of the same understanding tasks through prompting, at many times the cost per query of a 110-million-parameter encoder.

Later encoders refined the recipe. Meta's RoBERTa dropped NSP and trained longer with larger batches on about 160 GB of text, showing that BERT had been significantly undertrained, and Hugging Face's DistilBERT used knowledge distillation to cut the parameter count by 40% and run 60% faster while keeping 97% of BERT's language understanding performance. ModernBERT, released in December 2024 by Answer.AI and LightOn, was trained on 2 trillion tokens with a native context length of 8,192 tokens, 16 times BERT's 512-token limit.