Machine Translation

Machine translation (MT) is the automatic translation of text or speech from one language into another. It is one of the oldest tasks in natural language processing (NLP) and one of the most widely used: Google Translate, DeepL, and Microsoft Translator translate web pages, documents, chat messages, product listings, and spoken conversations at a scale no team of human translators could match. Today's systems use neural networks, and large language models (LLMs) translate as one of their general abilities.

From Rules to Statistics to Neural Networks#

The Georgetown-IBM experiment of 1954 translated more than sixty Russian sentences into English with a 250-word vocabulary and six grammar rules, and its organizers predicted that MT would be solved within a few years. In 1966 the ALPAC report judged machine translation slower and less accurate than human translation, and US funding for MT research was sharply cut. Rule-based systems such as SYSTRAN, founded in 1968, encoded dictionaries and grammar rules for each language pair by hand.

Statistical MT learned translation from data instead. IBM researchers led by Peter Brown published word alignment models in 1993 that learned which French words correspond to which English words from the bilingual record of the Canadian Parliament, and phrase-based MT (2003) translated multi-word chunks, the approach Google Translate used when it launched in 2006. Neural MT replaced the statistical pipeline with one network trained end to end: Ilya Sutskever and colleagues' sequence-to-sequence model (2014) used long short-term memory (LSTM) networks to encode a sentence into one vector and decode it in another language, and Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio added an attention mechanism that lets the decoder look back at every source word. Google's neural system reduced translation errors by an average of 60% against its phrase-based production system in a 2016 human side-by-side evaluation of simple sentences, and the transformer, introduced the next year for translation, reached 28.4 BLEU on English-to-German and 41.8 on English-to-French on the WMT 2014 benchmarks.

How Neural Machine Translation Works#

Source text is split into subword tokens by tokenization schemes such as byte-pair encoding, so rare words and names break into known pieces. The encoder turns the whole source sentence into contextual vectors, and the decoder writes the translation one token at a time, attending both to the tokens it has already written and, through cross-attention, to the encoder's vectors:

Because the encoder reads the whole sentence before the decoder writes anything, the translation of each word depends on its context:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# T5 is an encoder-decoder transformer; a text prefix selects the translation task
tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

sentences = ["She went to the bank to open an account.", "They had a picnic on the bank of the river."]
inputs = tokenizer([f"translate English to German: {s}" for s in sentences], return_tensors="pt", padding=True)

# The decoder writes each German sentence one token at a time with beam search
outputs = model.generate(**inputs, num_beams=4, max_new_tokens=40)
for source, translation in zip(sentences, tokenizer.batch_decode(outputs, skip_special_tokens=True)):
    print(f"{source} -> {translation}")

Google's 60-million-parameter T5-small, trained with English-German translation as one of its tasks, renders "bank" as "Bank" (the financial institution) in "Sie ging an die Bank, um ein Konto zu eröffnen." and as "Ufer" (riverbank) in "Sie hatten ein Picknick am Ufer des Flusses." It also moves "eröffnen" (open) to the end of its clause, as German word order requires.

Evaluating Translation#

BLEU, introduced by Kishore Papineni and colleagues at IBM in 2002, scores a translation by its overlapping word sequences with one or more human reference translations, with a penalty for output that is too short. It is cheap and reproducible, and it gives low scores to correct translations that word things differently from the references. Learned metrics such as COMET (2020) score a translation with a neural network trained on human quality judgments, and the 2022 WMT metrics shared task, titled "Stop Using BLEU", found that neural metrics agree with expert human ratings more closely than BLEU does.

Current Systems and Challenges#

Meta's NLLB-200 (2022) translates between 200 languages, many with little digital text, and improved BLEU by 44% relative to the previous state of the art on its Flores-200 benchmark. Low-resource languages lack parallel training data, so systems generate synthetic pairs with back-translation, a data augmentation method that translates monolingual target-language text into the source language. Neural systems can also hallucinate fluent output unrelated to the source, especially on noisy or low-resource input. Gender is a known source of bias: a gender-neutral Turkish pronoun can come out as "he" for a doctor and "she" for a nurse, which led Google Translate to show both feminine and masculine translations for some queries in 2018. MT also combines with other modalities: phone camera apps run optical character recognition (OCR) on signs and menus before translating them, and speech translation chains speech-to-text, MT, and text-to-speech or does all three in one model. The Ultralytics LLM interface accepts an image argument alongside the prompt, so a Python pipeline can send a photo of a sign to a vision-language model and ask for its text in English in one call.

Machine Translation vs. Related Terms#

• NLP: The field of computing with human language, of which MT is one task.
• LLM translation: General models such as GPT and Gemini translate when prompted and can use a whole document or glossary as context. Dedicated MT models are trained only on translation, and NLLB-200 ships distilled versions with 600 million and 1.3 billion parameters that cover all of its 200 languages.
• Localization: Adapting a product for a market, including dates, currencies, images, and cultural references. MT translates the text, and localization teams review and adapt it.