License YOLO
Back to Ultralytics Glossary

Image Captioning

Image captioning generates a natural-language sentence describing an image's content. How captioning models work, how captions are scored, and uses.

Image captioning is the task of generating a natural-language description of an image, such as "A red double-decker bus driving down a city street." A captioning model has to recognize the important objects, their attributes, and their relationships, and then express them as fluent text, which places the task where computer vision meets natural language processing (NLP). Today most captions come from a vision-language model (VLM) that treats captioning as one instruction among many, and the captions themselves have become training data for the text-to-image and text-to-video generators that run the task in reverse.

How Captioning Models Work#

A captioning model pairs a vision encoder, which turns pixels into features, with a language decoder that writes the caption one token at a time. Oriol Vinyals and colleagues at Google set this template in 2014 with Show and Tell, a convolutional neural network (CNN) encoder feeding a long short-term memory (LSTM) decoder. Kelvin Xu and colleagues' Show, Attend and Tell (2015) added an attention mechanism so the decoder could look at different image regions for each word, and Peter Anderson and colleagues' bottom-up and top-down attention (CVPR 2018) attended over regions proposed by an object detection model, so each word could draw on a specific detected object.

Transformer encoders and decoders replaced CNNs and LSTMs, and pretraining on web-scale image and alt-text pairs replaced small curated datasets. Web alt text is noisy, so Salesforce's BLIP (2022) bootstrapped it: a captioner wrote synthetic captions for web images, a filter removed captions that did not match their images, and the cleaned set trained the next model. Modern VLMs such as Florence-2 take captioning as a prompt and return a short caption, a detailed caption, or a paragraph on request, and dense captioning, introduced by Justin Johnson and colleagues' DenseCap in 2016, writes a caption for each detected region. Florence-2 also returns boxes for objects named in a prompt, and the VLM Leaderboard compares how accurately it and other vision language models detect objects in real-world datasets.

Datasets and Evaluation#

The standard benchmark, COCO Captions (2015), gives each image five independent human captions, because one image can be described correctly in many ways. Automatic metrics compare a generated caption with those references by overlapping n-grams: BLEU from machine translation, METEOR, and CIDEr, which Ramakrishna Vedantam and colleagues designed for captioning in 2015 to measure consensus with the references while down-weighting n-grams that appear in the captions of many images. Overlap metrics reward wording more than truth, as this NLTK BLEU score shows:

from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu

# Human reference captions for one street photo, in the style of COCO Captions
references = [
    "a red double decker bus driving down a city street",
    "a red bus drives down the street past people on the sidewalk",
    "people walk along a street as a double decker bus passes",
]
candidates = {
    "correct paraphrase": "a large red coach travels along an urban road",
    "wrong color": "a blue double decker bus driving down a city street",
}

refs = [r.split() for r in references]
smooth = SmoothingFunction().method1
for name, caption in candidates.items():
    score = sentence_bleu(refs, caption.split(), smoothing_function=smooth)
    print(f"{name}: BLEU-4 = {score:.2f}")

The correct paraphrase scores 0.03 and the caption with the wrong color scores 0.78, because it matches nine of the first reference's ten words. SPICE (2016) addresses this by comparing scene graphs of objects, attributes, and relations, and CLIPScore (2021) skips references altogether, scoring a caption by the cosine similarity between its CLIP text embedding and the image embedding. Captioners also hallucinate objects that typically appear in similar scenes but are absent from the image, and they inherit dataset bias, such as guessing gender from context.

  • Image classification: Assigns labels such as "bus" to the whole image, while a caption also states what the bus is doing and where.
  • Visual question answering (VQA): Answers a specific question about an image instead of describing it in general.
  • Optical character recognition (OCR): Transcribes the text visible in an image, which a caption usually summarizes or skips.
  • Alt text: Describes an image's purpose on a particular page, so a general caption is a draft that an author adapts, shortens, or replaces with an empty alt for decorative images.

Applications#

Accessibility was an early large-scale use: Facebook began adding automatically generated alt text to photos for screen reader users in April 2016, and screen readers now describe unlabeled images on request. Media libraries and retailers caption large image collections so that text queries such as "red backpack beside a tent" find files nobody tagged, alongside semantic search over image embeddings. Generative models depend on captions too: OpenAI trained DALL-E 3 on highly descriptive synthetic captions written by its own captioner and reported much better prompt following, and the Sora video model reused the same re-captioning technique. In Ultralytics Platform Agents, the YOLO → VLM Monitor template writes a scene description only when a YOLO26 nano model detects at least one person, so the vision-language model, and its per-call cost, runs only on images that contain a person.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning