Multi-Modal Learning

Multi-modal learning is the area of machine learning concerned with training models on data that comes in more than one modality, such as images with captions, video with its soundtrack, or camera frames with lidar scans, so that the model learns how the modalities relate. Its methods answer two questions: how to represent different modalities so they can be compared, and where in the model to combine them. The systems these methods produce, from vision-language assistants to self-driving perception stacks, are described as multimodal AI.

People combine senses without effort. In the McGurk effect, described in 1976, a video of lips saying "ga" dubbed with the sound "ba" is heard as "da." Jiquan Ngiam and colleagues at Stanford brought the idea to deep learning in 2011, training networks on audio and lip video together and showing that features learned from both modalities improved lip reading from video alone.

Core Challenges#

Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency organized the field around five challenges in their 2019 survey:

• Representation: Encoding each modality as vectors, either in one joint space or in separate spaces trained to line up.
• Translation: Mapping one modality to another, as in image captioning or text-to-image generation.
• Alignment: Finding which parts of one modality correspond to which parts of another, such as the image region a phrase describes.
• Fusion: Combining information from several modalities to make a prediction.
• Co-learning: Transferring knowledge from a modality with plenty of data to one with little.

Fusion Strategies#

Early fusion combines inputs or low-level features before any modality-specific processing, such as stacking a depth map with RGB channels into one four-channel image. Late fusion runs a separate model per modality and merges their predictions by averaging or voting, which keeps working when one modality is missing. Intermediate fusion encodes each modality separately and lets them exchange information inside the network, usually with cross-attention, an attention mechanism in which tokens from one modality attend to features of another. Sensor fusion in robots and vehicles uses all three.

Adding a modality can also lower accuracy. Weiyao Wang, Du Tran, and Matt Feiszli showed in 2020 that a video classification network trained jointly on audio, frames, and motion often performs worse than its best single-modality counterpart, because it overfits and its modalities learn at different speeds. Their Gradient-Blending method reweights each modality's loss to compensate.

Contrastive Alignment#

Contrastive learning aligns modalities without manual labels. OpenAI's CLIP (2021) trained an image encoder and a text encoder on 400 million image-caption pairs so that each image's embedding is most similar to its own caption among all the captions in a batch, and the result matched the original ResNet-50's ImageNet accuracy zero-shot, without using any of ImageNet's 1.28 million labeled training images. The loss function is cross-entropy over the similarity matrix, applied in both directions:

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Unit-length embeddings for 4 images and 4 captions; row i of each is a matching pair
image_emb = F.normalize(torch.randn(4, 64), dim=-1)
text_emb = F.normalize(image_emb + 0.5 * torch.randn(4, 64), dim=-1)


def contrastive_loss(images, texts, temperature=0.07):
    """CLIP-style symmetric cross-entropy over the image-text similarity matrix."""
    logits = images @ texts.T / temperature
    targets = torch.arange(len(images))  # the matching caption sits on the diagonal
    return (F.cross_entropy(logits, targets) + F.cross_entropy(logits.T, targets)) / 2


print(f"Aligned pairs:    {contrastive_loss(image_emb, text_emb):.3f}")
print(f"Misaligned pairs: {contrastive_loss(image_emb, text_emb.roll(1, dims=0)):.3f}")

With each caption close to its own image the loss is 0.577, and shifting the captions by one row so that every image is paired with the wrong caption raises it to 2.756. Training lowers this loss over batches of many pairs (CLIP used 32,768), which pulls each image toward its own caption and away from every other caption in the batch, and CLIP learns the temperature during training, starting from 0.07. Google's ALIGN (2021) scaled the same recipe to 1.8 billion noisy image and alt-text pairs, SigLIP (2023) replaced the softmax with an independent sigmoid for each pair, and Meta's ImageBind (2023) aligned six modalities, including audio, depth, thermal, and motion sensor data, by pairing each of them with images.

Open-vocabulary detectors learn the same kind of alignment between text and image regions. The Ultralytics recipe for training YOLOE-26, an open-vocabulary YOLO26 based on Tsinghua University's YOLOE, from scratch combines Objects365 detection labels with GQA and Flickr30k grounding data, about 1.4 million samples; the trainer encodes every class name and phrase once with Apple's frozen MobileCLIP2 text encoder and caches the embeddings, so training updates only the detector, including a small residual layer that refines those cached text vectors before they are matched to region features.

Multi-Modal Learning vs. Related Terms#

• Multimodal AI: The systems and applications built with multi-modal learning, such as visual assistants and driver-assistance perception.
• Multi-modal model: A trained model that handles several modalities, the artifact multi-modal learning produces.
• Vision-language model (VLM): A multi-modal model for images and text, often trained in stages: contrastive pretraining, then connecting the image encoder to a language model, then visual instruction tuning.
• Transfer learning: Reuses a model trained on one task for another, and co-learning applies the idea across modalities.