Multi-Modal Model

A multi-modal model, also written multimodal model, is a single machine learning (ML) model that takes in, and sometimes produces, more than one type of data, or modality: images, text, audio, video, depth maps, or sensor readings. Because it processes the modalities together, it can use each as context for the others, answering a spoken question about a photo, finding the video frames a sentence describes, or detecting pedestrians from camera and radar data at once. A pipeline that chains separate unimodal models, such as a speech recognizer feeding a text-only chatbot, passes information between modalities only through each model's final output, while a multi-modal model learns how they relate.

How Multi-Modal Models Work#

Each modality first passes through an encoder suited to it, such as a Vision Transformer (ViT) for images or a tokenizer and embedding layer for text, which turns it into vectors. The model then fuses those vectors and predicts text, labels, boxes, or actions from the combined representation. Where the fusion happens defines the main design families:

• Early fusion: Inputs are combined before or at the first layers. Co-registered sensors can be stacked as extra image channels, and Ultralytics YOLO26 supports this directly: setting the channels key in a dataset YAML trains a detector on images with more than three channels, such as RGB plus thermal or depth bands saved as one multi-channel TIFF, as the 10-channel COCO8-Multispectral dataset demonstrates.
• Intermediate fusion: Each modality is encoded separately and the features meet in the middle, usually through attention. DeepMind's Flamingo inserts cross-attention layers into a frozen language model so text tokens can attend to image features, and many vision language models (VLMs) project image features into "visual tokens" that a large language model (LLM) reads alongside the prompt.
• Late fusion: Separate models make predictions that are combined at the end, which is robust when one sensor fails but cannot learn interactions between modalities.
• Joint embedding: Encoders map every modality into one shared vector space, trained with contrastive learning so matching pairs land close together. CLIP aligns images and text, and Meta's ImageBind extends one embedding space to six modalities, images, text, audio, depth, thermal, and motion sensor data, using only data paired with images.

The data and the task largely decide which design fits:

ViLT, the Vision-and-Language Transformer, is a compact early-fusion example: it cuts an image into patches, embeds them alongside the words of a question, and runs one transformer over the combined sequence. The following example uses the ViLT checkpoint fine-tuned for visual question answering (VQA) through Hugging Face Transformers:

import requests
import torch
from PIL import Image
from transformers import ViltForQuestionAnswering, ViltProcessor

# ViLT embeds image patches and question tokens into one sequence for a single transformer
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
image = Image.open(requests.get("https://ultralytics.com/images/bus.jpg", stream=True).raw)

# Ask different questions about the same image; each answer is one of the model's 3,129 answer classes
for question in ["What vehicle is this?", "What color is the bus?", "How many people are there?"]:
    inputs = processor(image, question, return_tensors="pt")
    with torch.no_grad():
        logits = model(**inputs).logits
    answer = model.config.id2label[logits.argmax(-1).item()]
    print(f"{question} {answer}")

The model answers bus and blue and white correctly from the same image and answers 3 to the counting question, although four people appear in the photo, two of them cut off by its edges; counting remains a common weakness of vision-language models.

Types of Multi-Modal Models#

• Vision-language models: Read images and text to caption, answer questions, read documents, or locate objects named in a prompt, a task called grounding. VLM benchmark results compare how accurately such models locate objects in real-world detection datasets.
• Natively multimodal assistants: OpenAI's GPT-4o, released in 2024, was trained as a single network end to end across text, vision, and audio, and Google designed its Gemini models to be multimodal from the start, so one model can take a mix of text, images, audio, and video in the same prompt.
• Audio-language models: Combine speech or sound with text for transcription, translation, and spoken conversation.
• Vision-language-action models: Extend a VLM to output robot motor commands, so a robot can follow instructions such as "put the apple in the bowl".
• Sensor-fusion perception: Autonomous vehicles fuse camera, lidar, and radar inputs into one model that detects and tracks objects in 3D.

Multi-Modal Model vs. Related Terms#

• Multimodal AI: The broader field of AI that works with several data types, including systems that chain unimodal models. A multi-modal model is a single model within that field.
• Multi-modal learning: The training methods, such as contrastive alignment and cross-modal pretraining, that produce multi-modal models.
• Sensor fusion: Combining readings from several sensors, often with classical estimators such as Kalman filters rather than a learned model.
• Foundation model: A large model pretrained for many downstream tasks. Many recent foundation models are multi-modal, while others, such as text-only LLMs, handle one modality.

Challenges#

Multi-modal models need large amounts of paired data, such as captioned images or transcribed audio, and they must cope with a modality that is missing or noisy at inference, such as a camera blinded by glare. Image and video inputs are expensive, since every picture adds hundreds or thousands of tokens to an LLM's context. Models can also lean on the easier modality and ignore the harder one: early VQA models often answered from patterns in the question alone, so the VQA v2 benchmark paired each question with two similar images that lead to different answers, which roughly doubled the dataset's image-question pairs.