Reinforcement Learning from Human Feedback (RLHF)

Reinforcement learning from human feedback (RLHF) is a training method that improves an AI model using people's judgments of its outputs instead of a hand-written objective. Human raters compare pairs or sets of model responses, a reward model learns to predict which response they would prefer, and reinforcement learning then fine-tunes the original model to produce responses that the reward model scores highly. RLHF exists because qualities such as helpfulness, honesty, and tone are easy for people to recognize and nearly impossible to write as a loss function. In 2017, Christiano et al. used it to teach agents novel behaviors in Atari games and robot simulators from about an hour of human comparisons, and it became the standard final stage for large language models (LLMs) and other generative AI systems after OpenAI's InstructGPT and ChatGPT in 2022.

How RLHF Works#

The classic recipe, described in the InstructGPT paper, has three stages:

1. Supervised fine-tuning (SFT): A pretrained base model is fine-tuned on demonstrations written by people, such as good answers to sample prompts, which teaches it the expected format and tone.
2. Reward model training: The SFT model generates several responses per prompt, and labelers rank them. InstructGPT's roughly 40 contractors ranked between 4 and 9 responses per prompt. A copy of the model with a single scalar output is trained to give preferred responses higher scores than rejected ones.
3. Reinforcement learning: The SFT model becomes a policy and is optimized with Proximal Policy Optimization (PPO) to maximize the reward model's score, minus a penalty proportional to its KL divergence from the SFT model. The penalty keeps the policy close to fluent text the reward model has seen and stops it from drifting toward strange outputs that the reward model happens to overrate.

The KL coefficient, usually written β, decides how far the policy may move. The KL-regularized objective has an exact optimum, the reference probabilities reweighted by the exponential of reward divided by β, which this NumPy example computes for four candidate responses:

import numpy as np

# Four candidate responses: probability under the fine-tuned reference model and reward-model score
reference = np.array([0.50, 0.30, 0.15, 0.05])
reward = np.array([0.0, 1.0, 2.0, 4.0])

for beta in [10.0, 1.0, 0.1]:
    # Optimal policy for "maximize expected reward - beta * KL(policy || reference)"
    policy = reference * np.exp(reward / beta)
    policy /= policy.sum()
    kl = (policy * np.log(policy / reference)).sum()
    print(f"beta={beta:>4}: policy={policy.round(2)}, expected reward={policy @ reward:.2f}, KL={kl:.2f}")

It prints a policy of [0.46 0.3 0.17 0.07] at β = 10, nearly the reference, and [0. 0. 0. 1.] at β = 0.1, where all probability moves to the response the reference model gave only 5%, for a KL of 3.00. A small β buys the highest reward-model score, and if the reward model overrates that rare response, the policy has learned to exploit an error instead of pleasing people. Direct Preference Optimization (DPO) uses this same closed-form optimum to train on preference pairs directly, without a separate reward model or an RL loop.

Where RLHF Is Used#

• Chat assistants: InstructGPT's labelers preferred outputs from its 1.3-billion-parameter RLHF model over those of the 175-billion-parameter GPT-3. Meta's Llama 2 chat models were tuned against two separate reward models, one for helpfulness and one for safety.
• Summarization: OpenAI's 2020 summarization models trained with human feedback produced summaries that raters preferred over the human-written reference summaries in their dataset.
• Image generation: Reward models trained on human ratings of generated images are used to fine-tune text-to-image models toward prompts followed faithfully and images people find appealing.
• Robotics and control: In tasks where success is easy to judge and hard to define, such as a natural-looking gait or a safe trajectory through a crowd, people compare short video clips of the robot's attempts instead of writing a reward function.

Vision-language models from major providers receive the same preference tuning as text chatbots. In Ultralytics Platform Agents, an Ultralytics YOLO26 detector decides which images need a closer look, and one of these models from the Platform model list, such as Google Gemini, follows the workflow's prompt to describe them.

Limitations#

Human preference data labeling is slow and costly, and raters disagree with each other, so the reward model learns an average of their tastes along with their blind spots. Raters also reward confident, agreeable answers: an Anthropic study of five assistants found that human preference data favors responses that match the user's stated views, one source of AI sycophancy. Optimizing hard against any learned reward eventually produces reward hacking, in which the reward model's score keeps rising while true quality falls.

RLHF vs. Related Methods#

• Standard reinforcement learning: The environment supplies a fixed reward, such as a game score. RLHF learns the reward from human judgments because no fixed reward captures the goal.
• Reward modeling: The reward model is one component of RLHF, the network that turns human rankings into a score the RL algorithm can optimize.
• Constitutional AI: An AI model, guided by written principles, produces the preference labels instead of people, a variant called reinforcement learning from AI feedback (RLAIF).
• Reinforcement learning with verifiable rewards (RLVR): A program checks each answer, such as a math result or a unit test, and supplies the reward. RLVR, usually with Group Relative Policy Optimization (GRPO), trains reasoning models, and modern post-training pipelines use both: verifiable rewards where a checker exists and RLHF-style reward models for open-ended prompts.