Multimodal AI

Multimodal AI refers to artificial intelligence (AI) systems that take in, combine, or produce more than one type of data, or modality: text, images, video, audio, and sensor readings such as lidar and radar. A multimodal assistant can look at a photo of a dashboard warning light, read a typed question about it, and answer in text or speech, and a self-driving car merges camera, lidar, and radar data into one picture of the road. The term names these systems and products, which are built with multi-modal learning, the training methods, such as contrastive alignment and fusion, that teach a model how modalities relate.

How Multimodal Systems Are Built#

Each modality first passes through an encoder suited to it, such as a vision transformer (ViT) for images or a spectrogram encoder for audio, which turns raw input into embeddings. A fusion stage then combines them, and a decoder or task head produces the output. Most current assistants route everything through a large language model (LLM): LLaVA (2023) connected a CLIP image encoder to the Vicuna language model through a learned projection that turns image features into token-like vectors, and DeepMind's Flamingo (2022) inserted new cross-attention layers into a frozen language model so that its text tokens could attend to image features.

Natively multimodal models train on mixed data from the start instead of attaching encoders to a finished language model. Google introduced Gemini as natively multimodal in December 2023, and OpenAI's GPT-4o (May 2024) accepts any combination of text, audio, image, and video and responds to spoken input in 320 milliseconds on average.

Earlier single-stream models such as ViLT (2021) fed image patches and question tokens into one transformer and picked each answer from a fixed vocabulary of 3,129 answers. ViLT is small enough to answer questions about an image on a CPU:

import requests
import torch
from PIL import Image
from transformers import ViltForQuestionAnswering, ViltProcessor

# ViLT embeds image patches and question tokens into one transformer input sequence
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
image = Image.open(requests.get("https://ultralytics.com/images/bus.jpg", stream=True).raw)

for question in ["What color is the bus?", "How many people are there?", "Is it raining?"]:
    inputs = processor(image, question, return_tensors="pt")
    with torch.no_grad():
        logits = model(**inputs).logits

    # Pick the most likely answer from the model's fixed answer vocabulary
    print(f"{question} -> {model.config.id2label[logits.argmax(-1).item()]}")

ViLT answers "blue and white" and "no" correctly and counts 3 people, while four are at least partly visible: two men walking in full view and two people cut off at the left and right edges. Counting small or partly visible objects is a common error for vision-language models.

Applications#

• Visual question answering (VQA): Be My Eyes added Be My AI in 2023, which uses OpenAI's GPT-4 to describe photos and answer follow-up questions for blind and low-vision users.
• Autonomous vehicles: Robotaxis such as Waymo's combine cameras, lidar, and radar, each covering conditions where the others struggle, such as glare, darkness, or fog.
• Healthcare: Clinical models read scans together with the patient's notes and history, the context a radiologist uses when reading an image.
• Text-to-image generation: Models such as Stable Diffusion turn a written description into an image, and text-to-video models extend this to motion.
• Multimodal RAG: Retrieves images, tables, and text passages for an LLM to answer from.

Dataset labeling has become a multimodal task as well. On a detection dataset, the class-prompted smart annotation in Ultralytics Platform sends the dataset's class names as text to hosted models such as Microsoft's Florence-2, IDEA Research's Grounding DINO, and Meta's SAM 3.1, which return editable boxes for training a YOLO26 detector. The VLM Leaderboard compares how accurately these models and generative VLMs label real-world detection datasets.

Multimodal AI vs. Related Terms#

• Multi-modal learning: The methods for training on several modalities, such as contrastive alignment and early, late, or cross-attention fusion. Multimodal AI is the resulting systems and applications.
• Multi-modal model: A single trained model that handles several modalities, the core component of most multimodal AI systems.
• Vision-language model (VLM): The most common kind of multimodal model, limited to images or video and text.
• Sensor fusion: Combines measurements of the same scene from several sensors, often with a Kalman filter, a practice in robotics and vehicles that long predates learned multimodal models.