Reinforcement Learning

Reinforcement learning (RL) is a branch of machine learning (ML) in which an agent learns to make a sequence of decisions by trying actions in an environment and receiving rewards or penalties, with the goal of collecting as much reward as possible over time. Nobody tells the agent which action is correct. It discovers which actions pay off, including actions whose payoff arrives many steps later, such as a chess move that wins material ten moves on. That makes RL the standard approach for sequential control problems: playing games, steering robots, and training large language models (LLMs) toward answers that people prefer or that a program can verify. Richard Sutton and Andrew Barto, whose textbook defined the modern field, received the 2024 ACM A.M. Turing Award for their foundational work on reinforcement learning.

How Reinforcement Learning Works#

RL problems are formalized as a Markov decision process (MDP). At every time step the agent observes the current state, chooses an action from its action space, and receives a reward together with the next state. Its objective is the return, the sum of future rewards in which a discount factor between 0 and 1 makes distant rewards count less than immediate ones. The reward is the only description of the task the agent ever gets, so a careless one invites reward hacking, where the agent scores well without doing what the designer intended.

• Agent: The learner and decision-maker, such as a robot controller or a game-playing program.
• Environment: Everything the agent interacts with, from a physics simulator to a real warehouse floor.
• State: The information the agent acts on, such as joint angles, a board position, or features extracted from a camera frame.
• Action: A choice available to the agent, either discrete (left or right) or continuous (a steering angle).
• Reward: A number the environment returns after each action, positive for progress and negative for mistakes or wasted time.
• Policy: The agent's rule for picking an action in each state, which is what training improves.
• Value function: The expected return from a state, or from taking a particular action in a state, which lets the agent judge actions by their long-term consequences.

Because the agent only learns about actions it tries, it must balance exploration, trying actions whose value is uncertain, against exploitation, choosing the action that currently looks best. A common rule is epsilon-greedy: act randomly with a small probability and greedily otherwise. This NumPy example uses Q-learning, a classic algorithm that learns the value of every state-action pair, to teach an agent to walk down a six-cell corridor to a reward at the far end:

import numpy as np

rng = np.random.default_rng(0)
# Corridor of 6 cells: start in cell 0, reward +1 for reaching cell 5; actions 0 = left, 1 = right
n_states, goal = 6, 5
Q = np.zeros((n_states, 2))  # estimated return for each (state, action)
alpha, gamma, epsilon = 0.5, 0.9, 0.2  # learning rate, discount factor, exploration rate

for episode in range(200):
    state = 0
    while state != goal:
        # Explore with probability epsilon, otherwise exploit the best-known action
        action = rng.integers(2) if rng.random() < epsilon else Q[state].argmax()
        next_state = max(0, state - 1) if action == 0 else state + 1
        reward = 1.0 if next_state == goal else 0.0
        # Q-learning update: move toward the reward plus the discounted best value of the next state
        Q[state, action] += alpha * (reward + gamma * Q[next_state].max() - Q[state, action])
        state = next_state

print("Greedy policy:", ["left" if a == 0 else "right" for a in Q[:goal].argmax(axis=1)])
print("Value of moving right:", Q[:goal, 1].round(2))

It prints a policy of right in every cell and values of [0.66 0.73 0.81 0.9 1. ]. The reward was only ever given in the last cell, yet the update propagated it backward, and each step farther from the goal is worth 0.9 times the next, exactly the discount factor at work.

Families of RL Algorithms#

Tabular methods like the one above store a value for every state, which stops working once states are camera images or continuous joint angles. Deep reinforcement learning replaces the table with a neural network that generalizes across similar states, and the main algorithm families differ in what that network learns:

Model-free agents learn purely from real or simulated experience, while model-based agents learn a world model that predicts how the environment responds and then plan with it, as in model predictive control or Monte Carlo tree search. Proximal Policy Optimization (PPO), which limits how far each update can move the policy, became the default policy-gradient method for robotics and for language model training, and Group Relative Policy Optimization (GRPO) adapted it to LLM reasoning by dropping the learned critic. OpenAI's Spinning Up walks through these algorithms with reference implementations.

Real-World Applications#

• Games: DeepMind's AlphaGo combined deep neural networks with tree search and RL self-play to defeat Go champion Lee Sedol 4 games to 1 in March 2016.
• Robotics: Quadruped and humanoid locomotion controllers are trained with RL across thousands of parallel simulated robots, often with randomized friction, masses, and terrain so the policy transfers to real hardware.
• Language model alignment: Chat assistants are fine-tuned with RL against a reward model trained on human preference comparisons, and reasoning models are trained with RL against automatic checkers that verify math answers or run unit tests.
• Recommendation systems: Treating a sequence of recommendations as decisions lets a system optimize long-term engagement instead of the next click.

Simulation is where most robot policies are trained, because an agent may need millions of attempts and real hardware breaks; frameworks such as NVIDIA Isaac Lab run thousands of environments on one GPU, and domain randomization narrows the gap between simulated and real physics. Camera-based RL systems usually act on the output of a perception model instead of raw pixels: Ultralytics YOLO26 running on a robot through the ROS integration publishes the object classes in each camera frame, and with a depth camera each object's distance, as lightweight messages that a policy can take as its state.

Reinforcement Learning vs. Related Paradigms#

• Supervised learning: Each training example comes with the correct answer. An RL agent gets only a scalar reward, often delayed, and its own actions decide which data it sees next.
• Unsupervised learning: The model finds structure, such as clusters, in unlabeled data, with no actions to take and no reward to maximize.
• Markov decision process: The MDP is the problem statement, and RL is the family of methods that solve it from experience without being given the transition probabilities.
• Imitation learning: The agent copies demonstrations from an expert, most simply through behavioral cloning, and cannot easily surpass its teacher, while RL can discover strategies no demonstrator showed. The two are often combined: AlphaGo first learned from 30 million positions from human games and then improved through self-play, and its successor AlphaGo Zero skipped the human data entirely and beat that version of AlphaGo 100 games to 0.