Reward Modeling
Reward modeling trains a network to score AI outputs the way people would, from preference comparisons. Covers the Bradley-Terry loss, types, and risks.
Reward modeling is the practice of training a neural network, called a reward model, to score an AI system's outputs the way people would judge them. In standard reinforcement learning, the environment supplies a reward, such as a game score. Many goals have no such score: a helpful answer, a well-composed image, or a robot that hands over a cup gently are easy to judge and hard to define in code. A reward model learns that judgment from human feedback, usually comparisons between two outputs, and then scores new outputs automatically, far faster and more consistently than people could. Reward models are the component that makes reinforcement learning from human feedback (RLHF) work, and they are also used to rerank candidate responses, filter training data, and evaluate models.
How a Reward Model Is Trained#
Training data consists of comparisons. People see a prompt with two or more responses from a model and pick the better one; in OpenAI's InstructGPT work, labelers ranked 4 to 9 responses per prompt, and every pair from a ranking became one training comparison. Ranking is used instead of absolute scores because people agree far more on which of two answers is better than on whether one answer deserves a 6 or a 7 out of 10.
The reward model is usually a pretrained large language model (LLM), often the model being aligned, with its output layer replaced by a linear head that maps the final embedding to one number. Training uses the Bradley-Terry model of pairwise comparisons, which says the probability that people prefer response A over response B is the sigmoid of the difference between their rewards. The loss is the negative log of that probability for the response people actually chose. This PyTorch example trains a linear reward head on 2,000 simulated comparisons:
import torch
import torch.nn.functional as F
torch.manual_seed(0)
# 2,000 preference pairs: embeddings of two responses to the same prompt
# Simulated raters prefer the response with higher hidden quality, with some noise
hidden_quality = torch.randn(64)
a, b = torch.randn(2000, 64), torch.randn(2000, 64)
a_wins = (a @ hidden_quality + torch.randn(2000) * 2 > b @ hidden_quality).unsqueeze(1)
chosen, rejected = torch.where(a_wins, a, b), torch.where(a_wins, b, a)
# Reward model: a linear head mapping a response embedding to one scalar score
reward_model = torch.nn.Linear(64, 1)
optimizer = torch.optim.Adam(reward_model.parameters(), lr=0.01)
train, test = slice(0, 1500), slice(1500, 2000)
for step in range(300):
# Bradley-Terry loss: maximize the probability that the chosen response scores higher
margin = reward_model(chosen[train]) - reward_model(rejected[train])
loss = -F.logsigmoid(margin).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
accuracy = (reward_model(chosen[test]) > reward_model(rejected[test])).float().mean()
print(f"Held-out pairs ranked like the raters: {accuracy:.1%}")It prints 94.8%. The noisy simulated raters themselves contradict the noise-free quality ranking on about 5% of those held-out pairs, so the reward model has learned the preference about as well as its labels allow, even though it never saw a single absolute score. Pairwise accuracy on held-out comparisons is also how real reward models are compared, for example on the RewardBench benchmark. Because only differences between rewards enter the loss, the scale of the scores is arbitrary, and pipelines often normalize them before RL.
Types of Reward Models#
- Outcome reward models: Score a complete response or episode, the standard choice in RLHF.
- Process reward models: Score each intermediate step of a solution. OpenAI's PRM800K dataset contains 800,000 step-level human labels on math solutions.
- Generative reward models: An LLM acting as a judge writes a critique or verdict, which lets a single model grade many kinds of tasks and explain its scores.
- Implicit reward models: Direct Preference Optimization (DPO) trains the policy on preference pairs directly, and the policy's log-probability ratio to its reference model acts as the reward.
- Image reward models: ImageReward was trained on 137,000 expert comparisons of generated images and is used to fine-tune text-to-image models toward prompt-faithful, appealing results.
Robot learning uses the same idea with vision models. An image classifier trained on camera frames that people labeled as successful or failed attempts, such as a cup placed upright or knocked over, outputs a success probability that serves as the reward for a learning policy. An Ultralytics YOLO26 model trained on such frames with the classify task fills that role, scoring every frame of an attempt in milliseconds.
Keeping the Reward Model Accurate#
A reward model is accurate only on outputs that resemble its training data. As the policy improves, it produces responses the reward model has never seen, so Meta's Llama 2 team collected fresh comparisons on the latest model's outputs and retrained the reward model before each new round of RL:
Optimizing too hard against any learned reward leads to reward hacking. OpenAI's study of reward model overoptimization used a large "gold" reward model in place of human raters and found that, as a policy was optimized against a smaller proxy reward model, the gold score first rose and then fell while the proxy score kept climbing, with the effect shrinking smoothly as the proxy model grew. Reward models also pick up rater biases, such as preferring longer or more agreeable answers, which RL then amplifies into verbosity and sycophancy. A KL penalty toward the original model, ensembles of reward models, and regular retraining on fresh comparisons all limit the damage.
Reward Modeling vs. Related Terms#
- RLHF: The full pipeline of supervised fine-tuning, reward modeling from human comparisons, and RL optimization. Reward modeling is its second stage.
- Reward function: A reward written by hand, such as +1 for reaching a goal. A reward model is learned from data because no one can write the function down.
- Verifiable rewards: A program checks the answer against a known solution, such as a unit test or a reference math result, which is harder to exploit than a learned model but works only where answers can be checked.
- Inverse reinforcement learning: Infers a reward function from expert demonstrations of the task, while reward modeling for RLHF learns from people's comparisons of the system's own outputs.










