Sequence-to-Sequence Models

Sequence-to-sequence (seq2seq) models turn one sequence into another whose length can differ, such as an English sentence into its French translation, a long article into a summary, or an audio clip into a transcript. Most use an encoder-decoder design: the encoder reads the whole input into internal representations, and the decoder generates the output one token at a time, conditioned on those representations and on the tokens it has already produced. The approach took its modern form in 2014, when Sutskever, Vinyals, and Le at Google published Sequence to Sequence Learning with Neural Networks and Cho et al. described an RNN encoder-decoder for machine translation. Since 2017, most seq2seq models have been built from transformers.

How Seq2Seq Models Work#

In the 2014 models, an encoder long short-term memory (LSTM) network read the source sentence word by word and compressed it into one fixed-length vector, and a decoder LSTM unrolled that vector into the translation. Sutskever et al. used four-layer LSTMs, found that reversing the order of the source words made training easier, and reached a BLEU score of 34.8 on the WMT'14 English-to-French test set. Squeezing every sentence into one vector hurt long inputs, so Bahdanau, Cho, and Bengio added an attention mechanism in Neural Machine Translation by Jointly Learning to Align and Translate, letting the decoder look back at every encoder state at each step. The transformer kept this encoder-decoder layout, replaced the recurrence with self-attention, and connected the two halves with cross-attention. At inference time, the decoder runs in a loop:

Because the loop stops when the decoder emits an end token, the output length is learned per example. Decoding usually keeps several candidate outputs with beam search, the standard choice for translation and speech recognition.

Training With Teacher Forcing#

During training, the decoder does not need to loop. With teacher forcing, it receives the correct target sequence shifted right by one position and predicts every next token in a single parallel pass, with a causal mask so each position sees only earlier targets. The loss is the cross-entropy between those predictions and the true next tokens. PyTorch's nn.Transformer implements the original encoder-decoder:

import torch
from torch import nn

torch.manual_seed(0)
src_vocab, tgt_vocab, d_model = 1000, 800, 64
embed_src, embed_tgt = nn.Embedding(src_vocab, d_model), nn.Embedding(tgt_vocab, d_model)
model = nn.Transformer(d_model, nhead=4, num_encoder_layers=2, num_decoder_layers=2, batch_first=True)
to_logits = nn.Linear(d_model, tgt_vocab)

source = torch.randint(0, src_vocab, (1, 9))  # a 9-token source sentence
target = torch.randint(0, tgt_vocab, (1, 7))  # start token + a 6-token translation

# Teacher forcing: the decoder reads the correct previous tokens and predicts every next token in parallel
decoder_input, labels = target[:, :-1], target[:, 1:]
causal_mask = nn.Transformer.generate_square_subsequent_mask(decoder_input.size(1))
logits = to_logits(model(embed_src(source), embed_tgt(decoder_input), tgt_mask=causal_mask))
loss = nn.functional.cross_entropy(logits.flatten(0, 1), labels.flatten())

print(f"Source tokens: {source.shape[1]} -> decoder logits: {tuple(logits.shape)}")
print(f"Training loss: {loss.item():.2f}")

It prints Source tokens: 9 -> decoder logits: (1, 6, 800), one prediction over the 800-token target vocabulary for each of the 6 target positions, computed from a 9-token source in one pass. The untrained model's loss of 6.73 sits near ln(800) ≈ 6.68, the loss of a uniform guess. Because training always feeds correct previous tokens while inference feeds the model's own guesses, errors can compound at inference time, an effect called exposure bias.

Seq2Seq vs. Related Architectures#

• Encoder-only models such as BERT: These models read the input in both directions and output one representation per token, which suits classification and tagging.
• Decoder-only large language models (LLMs): Models such as GPT place the input and output in one sequence and handle seq2seq tasks by continuing the prompt, which is how most text generation runs today.
• Recurrent neural networks (RNNs) and GRUs: The recurrent layers inside the original 2014 encoders and decoders. Cho et al. introduced the gated recurrent unit in the same paper as their encoder-decoder.

Applications#

Google moved Google Translate to its GNMT system in 2016, an 8-layer LSTM encoder and decoder with attention, and translation remains the defining seq2seq task. Google's T5 framed every natural language processing (NLP) task as text-to-text, including text summarization and question answering. OpenAI's Whisper is an encoder-decoder transformer for speech recognition trained on 680,000 hours of audio. Vision models use the same pattern for image captioning and optical character recognition (OCR), and Microsoft's Florence-2 handles captioning, grounding, and detection as text outputs from one seq2seq model, and its Large version runs on Ultralytics Platform for auto-annotation. Google's Pix2Seq casts object detection as seq2seq by writing each object as five discrete tokens, four quantized coordinates and a class. RT-DETR, Baidu's real-time detector that runs in the Ultralytics package, keeps the encoder and cross-attending decoder but has its decoder refine 300 object queries in parallel, so every box comes out in one pass.