Text-to-Speech

Text-to-speech (TTS), also called speech synthesis, converts written text into spoken audio. A TTS system decides how each word is pronounced, how long each sound lasts, and how pitch and emphasis move through a sentence, and then generates a waveform. Older systems either spliced together recorded fragments of speech or generated audio from statistical models and sounded audibly synthetic. Deep learning changed that: DeepMind's WaveNet generated raw audio one sample at a time in 2016, and in 2017 Google's Tacotron 2 reached a mean opinion score of 4.53 out of 5, against 4.58 for professional recordings. Current systems speak many languages in many voices, clone a voice from a few seconds of audio, and follow instructions about tone, powering screen readers, voice assistants, audiobooks, and conversational AI. TTS is the inverse of speech-to-text.

How Text-to-Speech Works#

Every system starts with a text front end. Text normalization expands numbers, dates, currencies, and abbreviations into words, so "Dr. Lee paid $3.50" becomes "Doctor Lee paid three dollars and fifty cents," and grapheme-to-phoneme conversion picks pronunciations, including for words such as "read" whose sound depends on tense. Many models also predict prosody, the duration, pitch, and energy of each sound. Two main families then produce the audio:

• Acoustic model plus vocoder: An acoustic model predicts a mel spectrogram, a compact picture of how energy is spread across frequencies over time, and a neural vocoder converts it into a waveform. Tacotron 2 predicted spectrograms autoregressively, one frame after another. FastSpeech predicts them in parallel from per-phoneme durations and generated mel spectrograms 270 times faster than an autoregressive transformer. The HiFi-GAN vocoder synthesizes 22.05 kHz audio 167.9 times faster than real time on one V100 GPU. Voicebox and F5-TTS also predict mel spectrograms non-autoregressively, using flow matching, and pass them to a vocoder.
• Audio-token language models: Microsoft's VALL-E treats speech as discrete tokens from a neural audio codec and generates them with a transformer language model, much as an LLM generates text. Prompted with a 3-second recording of an unseen speaker, it continues in that voice.

End-to-end models such as VITS merge the acoustic model and vocoder into one network trained in a single stage, and multimodal models such as GPT-4o generate speech directly as part of their response, without a separate TTS step.

Text-to-Speech vs. Related Terms#

• Speech-to-text: The reverse direction, transcribing spoken audio into text. Voice assistants chain the two: speech-to-text transcribes the user, a large language model writes the reply, and TTS speaks it.
• Voice cloning: TTS that targets a particular person's voice, from a short reference clip or from fine-tuning on recordings of that speaker. Used without consent, it produces audio deepfakes.
• Voice conversion: Takes recorded speech as input and changes the speaker's identity while keeping the words and delivery. TTS starts from text.

Real-World Applications#

• Accessibility: Screen readers speak interface elements and documents aloud for blind and low-vision users, and people who have lost their voice type messages that a synthesized voice speaks for them.
• Voice assistants and agents: Customer service agents, in-car assistants, and smart speakers speak generated replies, often streaming audio for the first phrase while the rest of the reply is still being written.
• Audiobooks, dubbing, and e-learning: Publishers and course creators narrate long texts in consistent voices and many languages.
• Navigation: Turn-by-turn directions keep drivers' eyes on the road.

Paired with computer vision, TTS describes the physical world aloud, for example on assistive devices that tell a blind user what is in front of them. This code runs Ultralytics YOLO26 object detection on a street photo and speaks the result with gTTS, a Python library for Google Translate's text-to-speech service:

from collections import Counter

from gtts import gTTS
from ultralytics import YOLO

# Detect objects in a street photo with YOLO26
model = YOLO("yolo26n.pt")
results = model("https://ultralytics.com/images/bus.jpg")
result = results[0]

# Count detections per class and turn them into a sentence
counts = Counter(result.names[int(c)] for c in result.boxes.cls)
sentence = "I can see " + ", ".join(f"{n} {name}" for name, n in counts.items())

# Synthesize the sentence and save it as an MP3 file
gTTS(text=sentence, lang="en").save("detections.mp3")

The saved detections.mp3 speaks a sentence that lists each detected class with its count. gTTS sends the text to Google's servers, so the script needs an internet connection.

Measuring Speech Quality#

• Mean opinion score (MOS): Listeners rate naturalness from 1 to 5, and recordings of real speakers set the ceiling.
• Intelligibility: A speech recognition model transcribes the synthesized audio, and the word error rate against the input text shows how often words come out wrong, skipped, or repeated. Voicebox reached a 1.9% word error rate in zero-shot TTS, against 5.9% for VALL-E.
• Speaker similarity: For cloned voices, the cosine similarity between speaker embeddings of the generated and reference audio measures how closely the voice matches.
• Real-time factor (RTF): Synthesis time divided by audio duration. F5-TTS reports an RTF of 0.15, generating 10 seconds of speech in 1.5 seconds.