Deep Reinforcement Learning

Deep reinforcement learning (DRL) combines reinforcement learning (RL), where an agent learns by trial and error to maximize reward, with deep neural networks that represent what the agent learns. Classic RL stores a value for every state in a table, which is impossible when a state is a camera image or a robot's continuous joint angles. A neural network instead maps raw, high-dimensional observations such as pixels, lidar scans, or text to action values or action probabilities, and it generalizes to states it has never seen. DeepMind's DQN made the approach famous in 2015 by learning to play 49 Atari 2600 games from screen pixels and the game score alone, with one network architecture and one set of hyperparameters, at a level comparable to a professional human games tester.

How Deep RL Works#

The agent still follows the RL loop of a Markov decision process: observe a state, act, receive a reward, and update. What changes is that a network with millions of parameters now holds one or more of the following:

• Value function: The network predicts the expected return of each action, and the agent picks the highest. DQN's network took the last four game frames, downscaled to 84×84 pixels, so it could see motion.
• Policy: The network outputs a probability for each action, or the mean and spread of a continuous action such as a steering angle, and policy-gradient methods adjust it directly toward actions that earned high return.
• Actor and critic: An actor network chooses actions while a critic network estimates values to tell the actor how much better than expected each action was, which reduces the noise in its updates.
• World model: Model-based agents such as Dreamer train a network to predict future observations and rewards, then learn a policy inside that learned simulator.

Training a network on data the agent generates for itself is unstable, because consecutive frames are highly correlated and the targets move as the network changes. DQN solved this with two ideas that later algorithms kept: an experience replay buffer that trains on random minibatches of past transitions, and a separate target network that is updated only periodically:

Policy Gradients in Code#

Policy-gradient methods skip value tables entirely: they sample actions from the network, then raise the probability of the actions that were followed by high return. This PyTorch example trains a small policy network with REINFORCE, the original policy-gradient algorithm, to balance a pole on a moving cart in the Gymnasium CartPole environment:

import gymnasium as gym
import torch
from torch import nn

torch.manual_seed(0)
env = gym.make("CartPole-v1")
policy = nn.Sequential(nn.Linear(4, 64), nn.ReLU(), nn.Linear(64, 2))  # cart and pole state -> logits for left/right
optimizer = torch.optim.Adam(policy.parameters(), lr=0.02)
lengths = []

for episode in range(500):
    obs, _ = env.reset(seed=episode)
    log_probs, rewards, done = [], [], False
    while not done:  # play one episode with the current policy, earning +1 per step the pole stays up
        dist = torch.distributions.Categorical(logits=policy(torch.as_tensor(obs)))
        action = dist.sample()
        obs, reward, terminated, truncated, _ = env.step(action.item())
        log_probs.append(dist.log_prob(action))
        rewards.append(reward)
        done = terminated or truncated
    # REINFORCE: raise the log-probability of actions followed by above-average discounted return
    returns = torch.tensor([sum(0.99**k * r for k, r in enumerate(rewards[t:])) for t in range(len(rewards))])
    loss = -(torch.stack(log_probs) * (returns - returns.mean()) / (returns.std() + 1e-8)).sum()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    lengths.append(len(rewards))

print(f"Average episode length: first 20 = {sum(lengths[:20]) / 20:.0f}, last 20 = {sum(lengths[-20:]) / 20:.0f}")

It prints first 20 = 14, last 20 = 500: the untrained network drops the pole within about 14 steps, and after 500 episodes it balances for the full 500-step limit of every episode. With other random seeds the same script can stall or collapse partway through training, the high variance that critics and the clipped updates of Proximal Policy Optimization (PPO) were designed to reduce. Libraries such as Stable-Baselines3 provide tested implementations of PPO, DQN, and soft actor-critic (SAC).

Landmark Applications#

• Games: After Atari, deep RL agents defeated world champions at Go with AlphaGo and reached Grandmaster level in StarCraft II with AlphaStar, combining learning from human games with RL self-play.
• Drone racing: The University of Zurich's Swift drone, trained with deep RL in simulation and fine-tuned with real flight data, won several head-to-head races against each of three champion pilots in 2023 and set the fastest recorded race time.
• Fusion energy: DeepMind and EPFL's Swiss Plasma Center used a deep RL policy to control the magnetic coils of a tokamak, shaping the plasma into several configurations, including two separate plasma "droplets" at once.
• Robotics: Legged robots learn to walk over rough terrain from policies trained across thousands of simulated robots, with friction and mass randomized so the policy survives the move to real hardware.
• Language models: RL fine-tuning of transformer policies with preference-based or automatically checked rewards produced today's chat assistants and reasoning models.

Perception as the Observation#

DQN's convolutional neural network had to learn to see through reward alone, from 84×84×4 = 28,224 pixel values per step. Many robotic and driving systems instead feed the policy the output of a pretrained computer vision model, which makes the observation smaller and lets the RL algorithm spend its samples on decision-making. Ultralytics YOLO26 in tracking mode turns each camera frame into a class, a box, and a persistent ID for every object, an object-centric state from which a policy can also read each object's motion across frames.

Challenges#

Deep RL is expensive in data: DQN trained on 50 million frames per game, roughly 38 days of play, which is why most robot policies are trained in simulation and then transferred to hardware. Simulated physics never matches reality exactly, so engineers randomize friction, masses, lighting, and textures with domain randomization and add synthetic data for rare conditions. Results also swing with random seeds, hyperparameters, and implementation details, a reproducibility problem described in Deep Reinforcement Learning That Matters. A strong optimizer also finds loopholes in any imperfect reward, the failure known as reward hacking, as when a boat-racing agent learned to circle endlessly collecting points instead of finishing the race.

Deep RL vs. Related Terms#

• Reinforcement learning: The broader field. Tabular RL works for small problems, and deep RL is RL with neural networks as function approximators.
• Deep learning: Ordinary deep learning trains on a fixed, labeled dataset. In deep RL the agent's own actions generate the data, and the training signal is a delayed reward instead of a label.
• Reinforcement learning from human feedback (RLHF): A deep RL method in which a reward model learned from human preference comparisons replaces a hand-written reward, used to align language models.
• Model predictive control: MPC solves a fresh optimization over an explicit model at every control step, while a trained deep RL policy chooses each action in a single forward pass of its network.