License YOLO
Back to Ultralytics Glossary

Reinforcement Learning with Verifiable Rewards (RLVR)

Reinforcement learning with verifiable rewards (RLVR) trains models on automatic checks, such as passing unit tests or matching a known math answer.

Reinforcement learning with verifiable rewards (RLVR) is a way to train large language models (LLMs) and other generative models with reinforcement learning, where the reward comes from a program that checks each output against a known correct result. A math answer either matches the reference or it does not, and generated code either passes its unit tests or fails them. These checks are cheap and objective, so RLVR can run for many thousands of training steps, and it is a core technique behind reasoning models that work through a long chain of thought before answering. AI2 named the method in its Tülu 3 report in November 2024, and DeepSeek-R1 brought it wide attention in January 2025, when its R1-Zero model learned strong step-by-step reasoning from rule-based accuracy and format rewards alone, with no supervised fine-tuning first.

How RLVR Works#

Training starts from a pretrained, usually instruction-tuned model and a set of prompts whose answers a program can check. For each prompt the model samples several responses, a verifier scores each one, and a policy-gradient algorithm such as Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO) raises the probability of the responses that scored better than expected, which in GRPO means above the group's average, and lowers the rest. Clipping, and in many recipes a KL penalty, keeps the updated model close to where it started. The model is never shown a worked solution; it learns only which of its own attempts succeeded, so every reasoning pattern it strengthens is one it produced itself.

The verifier and the group comparison fit in a few lines. This sketch scores four sampled answers to one math prompt by extracting the final answer from a \boxed{} tag, the answer format DeepSeek-R1 required for math, and then computes GRPO's group-relative advantages:

import re

import numpy as np

reference = "42"  # known answer to the training prompt
completions = [  # a group of sampled responses to the same prompt
    "6 * 7 = 42, so the answer is \\boxed{42}",
    "Adding the terms gives \\boxed{41}",
    "The product is 42.",  # right number, but no final answer in the required format
    "Check: 7 * 6 = 42. \\boxed{42}",
]

# Verifier: extract the final boxed answer and compare it with the reference
rewards = []
for text in completions:
    match = re.search(r"\\boxed\{([^}]*)\}", text)
    rewards.append(1.0 if match and match.group(1).strip() == reference else 0.0)

# GRPO-style advantages: each reward relative to the group mean, scaled by the group std
rewards = np.array(rewards)
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-6)
print("Rewards:", rewards)
print("Advantages:", advantages.round(2))

It prints rewards of [1. 0. 0. 1.] and advantages of [ 1. -1. -1. 1.]. The two correct responses are reinforced, and the third is pushed down even though it contains 42, because it never states a final answer in the required format.

What Makes a Reward Verifiable#

A verifiable reward needs a ground truth and a program that compares the output with it. Common verifiers include:

  • Math answers: The final answer is parsed and compared with the reference after normalizing equivalent forms such as 1/2 and 0.5, which libraries like Math-Verify handle.
  • Code: Generated programs run in a sandbox against unit tests, and the reward is whether, or what fraction of, the tests pass.
  • Instruction following: Constraints a program can test, such as a word limit, required keywords, or valid JSON. Tülu 3 applied RLVR to this kind of precise instruction following alongside math.
  • Format: A small reward for placing reasoning and the final answer in the expected tags, which DeepSeek-R1 combined with its accuracy reward.
  • Vision tasks: Intersection over Union with labeled boxes scores grounding answers, and object detectors check generated images.

GenEval uses an object detector to check whether a generated image contains the objects, counts, colors, and positions its prompt asked for, and Flow-GRPO used those rule-based scores as its reward to raise Stable Diffusion 3.5 Medium's GenEval accuracy from 63% to 95%. An Ultralytics YOLO26 detector can act as the same kind of checker, turning a prompt such as "four people at a bus stop" into a counting reward:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")

# Prompt: "four people at a bus stop"; the image stands in for a generated sample
results = model("https://ultralytics.com/images/bus.jpg", classes=[0])  # class 0 = person
n_found, n_requested = len(results[0].boxes), 4

# Flow-GRPO's counting reward: full credit for the exact count, less for each missing or extra object
reward = 1 - abs(n_found - n_requested) / n_requested
print(f"People found: {n_found}, reward: {reward:.2f}")

RLVR vs. RLHF and Process Rewards#

RLVR sits alongside other ways of turning feedback into a training signal:

  • Reinforcement learning from human feedback (RLHF): A reward model learned from human preference comparisons scores the outputs. It covers subjective qualities such as tone and helpfulness, but a learned scorer has blind spots that a policy can exploit. DeepSeek-R1 avoided neural reward models for reasoning for that reason, and its final training stage paired rule-based rewards on reasoning prompts with learned reward models on general ones.
  • Process reward models (PRMs): These score each intermediate reasoning step. RLVR usually rewards only the final outcome, which is cheaper to obtain and harder to game.
  • Direct Preference Optimization (DPO): DPO learns offline from fixed pairs of preferred and rejected responses, with no sampling or verifier during training.

Training pipelines often choose the reward source prompt by prompt, falling back on AI feedback in the style of Constitutional AI when neither a checker nor human labels exist:

Strengths and Limits#

Because the reward needs no human in the loop, RLVR scales with compute and a supply of checkable problems, and it produces behaviors no one wrote down: DeepSeek-R1-Zero's responses grew steadily longer during training as the model learned to re-check and revise its own work. Libraries such as Hugging Face's TRL GRPO trainer accept a custom reward function, so any verifier like the ones above plugs straight in.

The method only reaches tasks with checkable answers, and a weak verifier invites reward hacking: a coding model rewarded for passing tests can learn to special-case or edit the tests instead of fixing the code. Researchers also disagree about what RLVR adds. A 2025 study of RLVR-trained models found that they beat their base models with one attempt per problem, while the base models solved more problems once allowed many attempts, which suggests RLVR mostly concentrates probability on solutions the base model could already produce. A follow-up paper counted a sample as correct only when both its answer and its reasoning chain were correct, and by that CoT-Pass@K measure an RLVR-trained Qwen2.5-32B stayed ahead of its base model at every K up to 1,024 on AIME 2024 and AIME 2025.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning