Visual Instruction Tuning

Visual instruction tuning is the fine-tuning of a vision-language model (VLM) on examples that pair an image and a natural-language instruction with the desired response, so the model learns to follow open-ended requests about images: describe a scene, answer a question, read text, or reason about what is shown. It extends text-only instruction tuning to images and is the step that turns a model which can merely relate pictures and words into a visual assistant. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee of the University of Wisconsin-Madison, Microsoft Research, and Columbia University introduced the term in Visual Instruction Tuning, an oral paper at NeurIPS 2023, together with the LLaVA (Large Language and Vision Assistant) model it produced.

Generating Instruction Data#

Few datasets paired images with instructions and responses, so the LLaVA authors generated them with a text-only model. For each COCO image, they gave GPT-4 a symbolic description of the image, namely its human-written captions and its bounding boxes with object categories and coordinates, plus a few hand-written examples, and asked it to write questions and answers as if it could see the picture. The resulting synthetic data, released as LLaVA-Instruct-150K, has 158,000 samples of three kinds: 58,000 multi-turn conversations, 23,000 detailed descriptions, and 77,000 complex-reasoning questions whose answers require step-by-step thinking about the scene.

Architecture and Training#

LLaVA connects a CLIP ViT-L/14 image encoder to the Vicuna large language model (LLM) through a single linear projection that maps image features into the LLM's word embedding space, so the image enters the LLM as a sequence of visual tokens ahead of the instruction. Training runs in two stages:

1. Feature alignment: On 595,000 image-caption pairs filtered from the CC3M dataset, only the projection is trained, while the vision encoder and LLM stay frozen, so the visual tokens become readable to the LLM.
2. Visual instruction tuning: On the instruction data, the projection and the LLM are both updated while the vision encoder stays frozen. The next-token loss is computed only on the assistant's answers, never on the image tokens or the user's instructions.

The resulting model held conversations about images and instructions it had never seen. On a synthetic multimodal instruction-following benchmark scored by GPT-4, LLaVA reached 85.1% of GPT-4's own score, and after fine-tuning on the ScienceQA benchmark, an ensemble of LLaVA and GPT-4 set a new state of the art at 92.53% accuracy.

LLaVA-1.5 simplified and strengthened the recipe: it replaced the linear projection with a two-layer MLP, raised the CLIP input resolution to 336 pixels, added academic visual question answering datasets with short prompts that specify the answer format, and finished training in about one day on a single node with eight A100 GPUs, using 1.2 million public samples for the 13-billion-parameter model. InstructBLIP from Salesforce took a different route, converting existing labeled datasets into instruction format and conditioning its image features on the instruction.

Applications#

Instruction-tuned VLMs replace many single-task models with one system that answers free-form questions, writes image captions at the level of detail a user asks for, and reads text in photos, combining optical character recognition (OCR) with reasoning about what the text means. Models that read screenshots this way power computer-use agents. The recipe also transfers to specialized domains: LLaVA-Med aligned on biomedical figure-caption pairs from PubMed Central, tuned on GPT-4-generated instructions about them, and trained in under 15 hours on eight A100 GPUs, a path toward medical image analysis assistants that explain findings in words.

Many instruction-tuned VLMs also return object locations when asked. Ultralytics Platform uses this in its class-prompted Smart annotation: on a detection dataset with 1-200 classes, hosted open-source models such as Moondream 3.1, or providers' paid vision models, find the dataset's classes in an image and return editable boxes for review.

Instruction-tuned VLMs also inherit the weaknesses of their data. Generated answers can describe objects that are not in the image, and preference methods such as Direct Preference Optimization (DPO) are applied after instruction tuning to reduce these hallucinations.

Visual Instruction Tuning vs. Related Concepts#

• Text instruction tuning: Fine-tunes a text-only LLM on instruction-response pairs. Visual instruction tuning adds an image to every example and trains the connection between vision encoder and LLM.
• Vision-language pretraining: Contrastive models such as CLIP learn to match images and captions, which gives a VLM its visual features but no ability to follow instructions.
• Visual prompting: Guides a model at inference with visual cues such as points, boxes, or example images, as in promptable segmentation. Visual instruction tuning is a training stage, and its prompts are written in natural language.
• Visual question answering (VQA) fine-tuning: Trains a model on one task and answer format, such as a single VQA benchmark. Visual instruction tuning mixes many task types in one instruction format so the model generalizes to new requests.