Instruction Tuning
Instruction tuning fine-tunes a pretrained language model on instruction-response pairs so it follows requests. Covers FLAN, InstructGPT, data, and VLMs.
Instruction tuning is fine-tuning a pretrained large language model (LLM) on pairs of instructions and desired responses so that it follows requests written in plain language. A pretrained model only continues text: asked by OpenAI to "Explain the moon landing to a 6 year old in a few sentences," GPT-3 continued with similar requests about gravity and relativity, while the instruction-tuned InstructGPT wrote the explanation. Google researchers described the method in the 2021 FLAN paper, which tuned a 137-billion-parameter model on more than 60 natural language processing (NLP) datasets rewritten as instructions; the tuned model beat the zero-shot performance of the 175-billion-parameter GPT-3 on 20 of 25 tasks it had never seen. In chat-model pipelines, the same step is usually called supervised fine-tuning (SFT), and the two terms are used as synonyms.
How Instruction Tuning Works#
Each example is formatted with a template that marks where the instruction ends and the response begins, such as the chat templates of current models or the "### Instruction" and "### Response" headers used by Stanford's Alpaca. Training uses the same next-token loss function as pretraining, but only the response tokens count, so the model learns to produce answers instead of to predict the instructions themselves. One training example on OpenAI's GPT-2 shows the setup:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)
# One instruction-response pair in the Alpaca prompt format
prompt_ids = tokenizer("### Instruction:\nName three animals that can fly.\n\n### Response:\n").input_ids
response_ids = tokenizer("Bats, eagles, and bees." + tokenizer.eos_token).input_ids
input_ids = torch.tensor([prompt_ids + response_ids])
# Label -100 drops the instruction from the loss, so the model learns only to respond
labels = torch.tensor([[-100] * len(prompt_ids) + response_ids])
for step in range(3):
loss = model(input_ids=input_ids, labels=labels).loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
print(f"step {step}: loss {loss.item():.2f}")Only the 10 response tokens count toward the loss, and three updates on this single example cut it from 4.45 to 1.49. An instruction-tuning run applies the same update across thousands to millions of different instruction-response pairs, so the model learns the general habit of answering instead of one answer.
Instruction tuning is usually the first stage of post-training, and the data can come from several sources:
Instruction Datasets#
Google's Flan collection grew to 1,836 tasks in 2022, and instruction tuning on it improved the 540-billion-parameter PaLM by 9.4% on average. OpenAI's InstructGPT combined labeler-written demonstrations with reinforcement learning from human feedback (RLHF), and people preferred the outputs of its 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3. Stanford's Alpaca, released in March 2023, fine-tuned Meta's LLaMA 7B for less than $600 on 52,000 examples that an OpenAI model generated with the Self-Instruct method, an early example of synthetic data as a source of instruction data. Meta's LIMA showed how far a small dataset goes: it tuned a 65-billion-parameter LLaMA on only 1,000 carefully chosen examples, and its authors concluded that a model learns almost all of its knowledge during pretraining while instruction tuning teaches the format and style of answers. Methods such as direct preference optimization (DPO) then refine the tuned model with preference data.
Visual Instruction Tuning#
Vision-language models (VLMs) learn to follow instructions about images the same way. LLaVA, published in April 2023, gave text-only GPT-4 the captions and bounding boxes of COCO images and had it write 158,000 conversations, detailed descriptions, and complex reasoning examples about them, then tuned a model that connects a CLIP image encoder to a language model on that data, a method known as visual instruction tuning. Instruction-tuned VLMs answer free-form requests about a photo, and the LLM block in Ultralytics Platform Agents relies on that ability: it sends the image, the upstream YOLO26 results, and a written prompt to the selected vision-language model.
Instruction Tuning vs. Related Terms#
Prompt tuning learns a few soft prompt vectors and leaves the model frozen, and prompt engineering changes only the input text, while instruction tuning updates the model's weights. Domain fine-tuning adapts a model to a field such as medicine or law without necessarily teaching it to follow requests. Instruction tuning can also run with parameter-efficient fine-tuning (PEFT): QLoRA trains LoRA adapters on a 4-bit quantized model, and its authors instruction-tuned a 65-billion-parameter model on a single 48 GB GPU.










