Optical Character Recognition (OCR)

Optical character recognition (OCR) is the conversion of text in images, such as scanned pages, photos, screenshots, and video frames, into machine-readable characters that software can search, edit, and analyze. Early systems matched each character against stored templates of known fonts. Modern OCR uses deep learning to find text anywhere in an image and read whole words and lines at once, which handles varied fonts, low resolution, perspective, and cluttered backgrounds. OCR covers both document text, such as invoices and books, and scene text, such as signs, product labels, and license plates.

How OCR Works#

Most OCR systems run two models in sequence:

1. Text detection: A detector finds the regions of the image that contain text and outputs a box or polygon for each word or line. Scene text detectors such as EAST, CRAFT, and DBNet predict text regions pixel by pixel, which handles curved and rotated text.
2. Text recognition: Each region is cropped, straightened, and resized to a fixed height, and a recognizer reads its characters. The CRNN design passes the crop through a CNN, reads the resulting feature columns left to right with a bidirectional LSTM, and decodes them with connectionist temporal classification (CTC), which aligns per-column predictions to a character string without labeling where each character starts. Transformer recognizers such as Microsoft's TrOCR generate the characters with a text decoder instead.

Preprocessing such as binarization and deskewing still helps on poor scans, and post-processing checks the output against dictionaries, expected formats, or a language model. EasyOCR packages both stages, a CRAFT detector and a CRNN recognizer, for more than 80 languages:

import easyocr

# Load EasyOCR's CRAFT text detector and English CRNN text recognizer
reader = easyocr.Reader(["en"])

# Detect every text region in the photo, then read the characters inside it
results = reader.readtext("https://ultralytics.com/images/bus.jpg")

# Each result holds a 4-corner box, the recognized string, and a confidence score
for box, text, confidence in results:
    if confidence > 0.5:
        print(f"{text}: {confidence:.2f}")

It prints cero: 0.95 and emisiones: 0.99, the words painted on the side of the bus. The detector also finds the bus's dot-matrix destination sign and the small print on its windows, but the recognizer returns garbled strings for them with confidences below 0.5, so the threshold drops them.

Documents add a layout step before recognition. A layout model splits the page into blocks, each block type gets its own treatment, and the results are reassembled in reading order, so a two-column article reads one column at a time:

Accuracy is measured with the character error rate (CER), the number of character insertions, deletions, and substitutions needed to turn the output into the correct text, divided by the length of the correct text, and with the word error rate (WER), computed the same way over words.

OCR Engines and Models#

The open-source Tesseract engine began at Hewlett-Packard in the 1980s, was released as open source in 2005, and was developed with Google's sponsorship from 2006 to 2018; version 4 added a recognition engine based on an LSTM network. Cloud services such as Google Cloud Vision and Amazon Textract add layout, table, and form extraction. Vision language models increasingly read documents end to end: olmOCR from the Allen Institute for AI converts PDF pages into clean text with a fine-tuned vision language model, and OCR-free document models answer questions about a page without producing a separate transcript first.

OCR vs. Related Terms#

• Handwriting recognition: Also called handwritten text recognition (HTR) or intelligent character recognition (ICR), it reads handwriting, where letter shapes vary far more than in print and characters often join.
• Object detection: A detector can find a stop sign as an object, while OCR reads the letters S-T-O-P printed on it. OCR's own detection stage locates text without reading it.
• Image classification: Assigns one label to a whole image, such as "invoice", while OCR extracts the character sequence inside it.
• Intelligent document processing: Builds on OCR with layout analysis and named entity recognition (NER) to pull specific fields, such as an invoice's date, vendor, and total, into business systems.

Real-World Applications#

• License plate recognition: Tolling, parking, and traffic enforcement systems detect each plate, then read it. A YOLO26 model trained on plate boxes handles the detection stage, and its oriented bounding box variant fits rotated boxes to tilted plates so each crop can be straightened before recognition.
• Document digitization: Banks, insurers, and governments turn forms, receipts, and archives into searchable text, and libraries digitize printed books at scale.
• Accessibility and translation: Screen readers and camera translation apps read signs, menus, and labels aloud or overlay a translation on the live image.
• Manufacturing and logistics: Cameras read lot codes, expiry dates, and serial numbers on production lines. Shipping container codes follow ISO 6346, whose 11 characters end in a check digit computed from the first 10, which lets a reader reject most misread codes automatically.