Recurrent Neural Network (RNN)

A recurrent neural network (RNN) is a neural network that processes a sequence one element at a time while carrying a hidden state from each step to the next, so its output at any step can depend on everything it has read so far. The same weights are applied at every step, which lets one network handle sequences of any length, from a sentence to hours of sensor readings. RNNs were the main architecture for natural language processing (NLP), speech recognition, and time series analysis until transformers replaced them for most language tasks in the late 2010s. Their gated variants are still common in streaming and on-device models, and recurrence has returned in state space models (SSMs) such as Mamba.

How an RNN Works#

A simple RNN, also called an Elman network after Jeffrey Elman's 1990 paper, updates its hidden state as hₜ = tanh(Wxₜ + Uhₜ₋₁ + b) and can emit an output yₜ = Vhₜ + c at each step. The hidden state is a fixed-size vector that summarizes the sequence so far, and tanh keeps its values between −1 and 1 as it is fed back. Drawn with one copy of the network per step, the loop unrolls into a deep feedforward network whose layers share the same weights:

The snippet below runs PyTorch's nn.RNN over a sequence and repeats the same recurrence in a plain loop:

import torch
from torch import nn

torch.manual_seed(0)
rnn = nn.RNN(input_size=3, hidden_size=4, batch_first=True)
x = torch.randn(1, 6, 3)  # one sequence: 6 time steps, 3 features each

# PyTorch runs the whole sequence; output holds the hidden state after every step
output, h_n = rnn(x)

# The same recurrence by hand, reusing one set of weights at every step
h = torch.zeros(1, 4)
for t in range(x.shape[1]):
    h = torch.tanh(x[:, t] @ rnn.weight_ih_l0.T + rnn.bias_ih_l0 + h @ rnn.weight_hh_l0.T + rnn.bias_hh_l0)

print(f"Output shape: {tuple(output.shape)}, final state shape: {tuple(h_n.shape)}")
print(f"Hand-written loop matches nn.RNN: {torch.allclose(h, h_n[0], atol=1e-6)}")

It prints Output shape: (1, 6, 4), final state shape: (1, 1, 4) and Hand-written loop matches nn.RNN: True. The output holds one 4-dimensional hidden state per time step, and the loop reproduces the final state with just one weight matrix for the input and one for the previous state.

Sequence Tasks#

The same recurrence fits tasks with different input and output lengths:

• Many to one: Read a whole sequence and classify it from the final state, as in sentiment analysis or labeling a window of sensor data.
• Many to many: Emit an output at every step, as in tagging each word of a sentence or each frame of audio.
• Sequence to sequence: An encoder RNN reads the input and a decoder RNN generates an output of a different length, the design of early neural machine translation and other sequence-to-sequence models.
• One to many: Generate a sequence from a single input, such as a caption from an image's feature vector in early image captioning models.

Bidirectional RNNs add a second RNN that reads the sequence backward and combine both states at each step, so every output sees past and future context. Stacked RNNs feed one layer's hidden states into the next as its input sequence.

Training With Backpropagation Through Time#

RNNs are trained with backpropagation through time (BPTT): the network is unrolled over the sequence, the loss is summed over steps, and backpropagation runs through the unrolled graph. The gradient that reaches step t from step T passes through T − t copies of the recurrent layer and is multiplied by U and by tanh's derivative each time. When those factors are mostly smaller than 1, the gradient shrinks geometrically, the vanishing gradient problem that Yoshua Bengio, Patrice Simard, and Paolo Frasconi analyzed in 1994. When they are larger than 1, it grows geometrically, the exploding gradient problem. The snippet below measures how much each input of a 50-step sequence affects the final hidden state:

import torch
from torch import nn

torch.manual_seed(0)
rnn = nn.RNN(input_size=8, hidden_size=32, batch_first=True)
x = torch.randn(1, 50, 8, requires_grad=True)  # one 50-step sequence

# Backpropagate from the final hidden state to the input at every step
output, h_n = rnn(x)
h_n.sum().backward()
grad_norms = x.grad[0].norm(dim=1)

for distance in [0, 10, 20, 30, 40]:
    print(f"Input {distance:>2} steps before the end: gradient {grad_norms[-1 - distance]:.1e}")

It prints gradients of 1.6e+00, 3.4e-03, 6.2e-06, 1.2e-08, and 1.1e-11 for inputs 0, 10, 20, 30, and 40 steps before the end. Every ten steps cost about three orders of magnitude, so training can barely adjust the weights to use an input a few dozen steps back. Gradient clipping, analyzed for RNNs by Razvan Pascanu and colleagues in 2013, caps exploding gradients, and truncated BPTT limits backpropagation to a fixed window to bound memory. Vanishing gradients need a change of architecture, which is what the gates of the LSTM and GRU provide.

RNNs vs. Related Architectures#

• LSTM and GRU: Gated RNNs with an additive state update that lets information and gradients pass through many steps. In practice, "RNN" often refers to one of these as well as to the simple RNN.
• Transformer: Attends to every earlier position directly instead of compressing history into one state, and trains on all positions in parallel. Its attention cost grows with sequence length, and an RNN spends the same compute and memory on every new step.
• Convolutional neural network (CNN): Mixes nearby positions with fixed-size filters, across space in images or across time with 1D convolutions, and its receptive field is set by its depth.
• State space models: Linear recurrences such as Mamba train in parallel with a scan and run step by step like an RNN at inference. Liquid neural networks are continuous-time RNNs whose neurons follow differential equations.

Applications#

• Keyboard prediction: Gboard's next-word model was a coupled input-forget gate LSTM, a lighter LSTM variant, trained with federated learning on users' phones, as Google researchers described in 2018.
• Demand forecasting: Amazon's DeepAR trains one autoregressive RNN across many related time series and outputs a probability distribution for each future value.
• Streaming audio: Wake-word detectors and noise suppressors process audio frame by frame with small recurrent layers, which keeps memory and latency constant on phones and earbuds.
• Object tracking: A tracker carries a state for each object from frame to frame, the role an RNN's hidden state plays. Every tracker built into YOLO26 track mode, from ByteTrack and BoT-SORT to the default TrackTrack, updates that state with a Kalman filter, a linear recurrence with hand-set constant-velocity equations in place of learned weights.