Direct Preference Optimization

Direct Preference Optimization (DPO) is a method for aligning a large language model (LLM) with human preferences by training it directly on pairs of responses, one preferred and one rejected, using a single classification-style loss function. It reaches the goal of reinforcement learning from human feedback (RLHF) without a separate reward model and without a reinforcement learning loop that samples from the model during training. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn at Stanford introduced it in 2023 in Direct Preference Optimization: Your Language Model is Secretly a Reward Model, presented at NeurIPS 2023. Its key idea is that the language model being trained defines its own implicit reward, so the reward model and the policy are fit in one step.

How DPO Works#

Each training example is a prompt with a chosen response and a rejected response, as labeled by people or by another model. Training starts from a model that has already gone through supervised fine-tuning, usually instruction tuning, and keeps a frozen copy of it as the reference. For every response, DPO compares the log-probability under the model being trained with the log-probability under the reference. The implicit reward of a response is β times that log-ratio, and the loss is the negative log-sigmoid of the reward of the chosen response minus the reward of the rejected one. Minimizing it raises the chosen response's probability relative to the rejected one, with larger updates on pairs the model still ranks the wrong way.

The coefficient β plays the role of the KL penalty in RLHF. Rafailov et al. showed that the KL-constrained reward maximization used in RLHF has a closed-form optimal policy, the reference probabilities reweighted by the exponential of reward divided by β, and rearranged that formula to express the reward through the policy itself. The smaller β is, the further training can move the model from the reference.

This PyTorch example runs ten DPO steps on GPT-2 from Hugging Face Transformers with two customer-service preference pairs, then checks a third pair the model never trained on:

import copy

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
policy = AutoModelForCausalLM.from_pretrained("gpt2")
reference = copy.deepcopy(policy).requires_grad_(False)  # frozen copy of the starting model
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-5)

# Preference pairs: (prompt, chosen reply, rejected reply)
train = [
    ("My order arrived broken.", " Sorry about that, we will send a replacement today.", " That is not our problem."),
    ("I was charged twice.", " Sorry for the trouble, we have refunded the extra charge.", " Check your bank again."),
]
held_out = ("My package is a week late.", " Sorry for the delay, we will track it down for you.", " Be more patient.")


def log_prob(model, prompt, reply):
    """Sum of the log-probabilities the model gives the reply tokens after the prompt."""
    ids = tokenizer(f"Customer: {prompt}\nAgent:{reply}", return_tensors="pt").input_ids
    token_logps = model(ids).logits[0, :-1].log_softmax(-1).gather(1, ids[0, 1:, None])
    return token_logps[-len(tokenizer(reply).input_ids) :].sum()


def margin(prompt, chosen, rejected):
    """Policy-to-reference log-ratio of the chosen reply minus that of the rejected reply."""
    chosen_ratio = log_prob(policy, prompt, chosen) - log_prob(reference, prompt, chosen)
    return chosen_ratio - (log_prob(policy, prompt, rejected) - log_prob(reference, prompt, rejected))


for step in range(10):
    loss = sum(-F.logsigmoid(0.1 * margin(*pair)) for pair in train) / len(train)  # DPO loss, beta = 0.1
    if step in {0, 9}:
        print(f"Step {step}: training loss {loss.item():.3f}")
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

with torch.no_grad():
    print(f"Held-out margin: {margin(*held_out).item():+.1f}")

It prints a training loss of 0.693 at step 0, which is ln 2, because the policy still equals the reference and both responses have zero implicit reward. By step 9 the loss is 0.047, and the held-out margin is +7.8: on a complaint it never saw, the model raised the log-probability of the apologetic reply relative to the reference by 7.8 more than that of the dismissive reply.

DPO vs. RLHF#

Classic RLHF trains a reward model on the preference pairs and then optimizes the language model against it with Proximal Policy Optimization (PPO), which needs fresh samples from the model at every step and keeps a policy, a value model, a reward model, and a reference model in memory. DPO needs only the policy and the reference and trains on a fixed dataset like ordinary supervised learning, so it is simpler to tune and much cheaper to run. In the original experiments, DPO beat PPO at controlling the sentiment of generated movie reviews for a given KL budget. On TL;DR Reddit summarization, GPT-4 judged DPO summaries better than the human-written references 61% of the time at sampling temperature 0, against 57% for PPO at its best temperature. On Anthropic's helpful and harmless dialogue data, DPO was the only computationally efficient method tested that improved on the preferred responses in the dataset.

The trade-off is that DPO learns only from the pairs it is given. Online RL methods such as Group Relative Policy Optimization (GRPO) score new samples from the current model during training, which is why they dominate reinforcement learning with verifiable rewards (RLVR) for math and code, where a checker can grade every fresh answer. The preference labels can also come from an AI model guided by written principles, as in Constitutional AI, and DPO trains on such labels in the same way.

Where DPO Is Used#

DPO became a standard post-training step for open-weight chatbots. Hugging Face's Zephyr-7B-β applied it to AI-ranked UltraFeedback data, and Meta's Llama 3 models were aligned with several rounds of supervised fine-tuning followed by DPO. The TRL library provides a ready-made DPO trainer. The Ultralytics LLM interface calls models through any OpenAI-compatible endpoint, including local servers, so a DPO-aligned open model can write the text side of an Ultralytics YOLO26 vision pipeline.

The method also carries over to images. Diffusion-DPO rewrites the loss for diffusion models and fine-tuned the Stable Diffusion XL base model on 851,000 crowdsourced pairwise preferences from the Pick-a-Pic dataset, improving prompt alignment and visual appeal in text-to-image generation. For vision-language models (VLMs), RLHF-V collected segment-level human corrections of hallucinated image descriptions and trained with a dense variant of DPO, cutting the base model's hallucination rate by 34.8% with only 1,400 annotated samples.

DPO Variants#

• IPO (Identity Preference Optimization): Replaces the log-sigmoid with a squared loss toward a fixed margin, which stops the implicit rewards from growing without bound and overfitting when the same response always wins.
• KTO (Kahneman-Tversky Optimization): Learns from single responses labeled desirable or undesirable, so it needs no paired comparisons.
• ORPO (Odds Ratio Preference Optimization): Drops the reference model and adds an odds-ratio penalty to the supervised fine-tuning loss, merging instruction tuning and preference alignment into one stage.
• SimPO (Simple Preference Optimization): Uses the length-normalized average log-probability of a response as its reward, removes the reference model, and requires a target margin between chosen and rejected rewards.