Speech-to-Text

Speech-to-text (STT), also called automatic speech recognition (ASR) or simply speech recognition, converts spoken audio into written text. An STT model turns a waveform into a time-frequency representation, maps it to characters or word pieces with a neural network, and outputs a transcript, often with punctuation, timestamps, and labels for who spoke. It is the input layer of voice assistants, live captions, meeting notes, and call analytics, and its transcripts feed natural language processing (NLP) for search, summarization, and intent detection. Self-supervised pretraining on unlabeled audio and weakly supervised training at scale, such as OpenAI's Whisper trained on 680,000 hours of multilingual audio from the internet, made modern systems accurate across accents, background noise, and dozens of languages.

How Speech-to-Text Works#

Audio is first resampled, to 16 kHz for Whisper, and converted into a log-mel spectrogram, a grid of energy per frequency band over time. Whisper computes 80 mel bands on 25-millisecond windows every 10 milliseconds, and its large-v3 model uses 128 bands. An acoustic encoder, built from convolutional and transformer layers, turns these frames into features. The Conformer encoder interleaves convolution, which captures local sound patterns, with self-attention, which relates distant parts of an utterance, and reached word error rates of 2.1% and 4.3% on the LibriSpeech test-clean and test-other sets without an external language model. Three decoder families turn encoder features into text:

• Connectionist temporal classification (CTC): Predicts a character, word piece, or blank for every audio frame, then collapses repeats and removes blanks. All frames are decoded in parallel, which makes CTC fast.
• Transducers: RNN-T and its successors emit tokens as audio frames arrive, so they suit streaming recognition on phones and in live captions.
• Attention encoder-decoders: A text decoder generates tokens one at a time while attending to the encoded audio. Whisper decodes 30-second windows this way and uses special tokens to also identify the language, predict timestamps, or translate into English.

Streaming systems show partial transcripts that change as more audio arrives and finalize a segment when the speaker pauses:

After decoding, inverse text normalization rewrites spoken forms such as "twenty five dollars" as "$25," and speaker diarization splits the transcript by who spoke when.

Training Data and Models#

• Self-supervised pretraining: Meta's wav2vec 2.0 learns speech representations by masking parts of unlabeled audio. Pretrained on 53,000 hours of unlabeled speech and fine-tuned on only 10 minutes of transcribed audio, it reached word error rates of 4.8% and 8.2% on LibriSpeech's clean and other test sets.
• Weak supervision at scale: Whisper trains on audio paired with the imperfect transcripts found online. Its models range from 39 million parameters for tiny to 1.55 billion for large, and the turbo model, an optimized large-v3 with 809 million parameters, runs about 8 times faster than large.
• Hallucination: Attention decoders behave like language models, so silence or noise can lead them to invent fluent text. The Careless Whisper study found that about 1% of Whisper transcriptions contained entire phrases or sentences that were not in the audio, more often for speakers with long pauses.

Speech-to-Text vs. Related Terms#

• Speech recognition and ASR: Synonyms for speech-to-text.
• Speaker recognition and diarization: Determine who is speaking rather than what was said. Diarization labels a recording's segments by speaker, so a meeting transcript can attribute each line.
• Natural language understanding (NLU): Interprets the transcript's meaning, such as the intent and entities in "book a table for two at seven."
• Text-to-speech (TTS): The reverse direction, generating speech from text. Voice assistants chain STT, a large language model, and TTS.

Real-World Applications#

• Captions and accessibility: Live captions on phones, video platforms, and meeting software make speech accessible to deaf and hard-of-hearing people and searchable for everyone.
• Clinical documentation: Ambient scribes transcribe doctor-patient conversations and draft clinical notes for the electronic health record.
• Contact centers: Transcribed calls feed search, compliance checks, and sentiment analysis.
• Voice control of vision systems: A transcribed command such as "find the red backpack" becomes a text prompt for YOLOE-26, the open-vocabulary version of Ultralytics YOLO26, which detects and segments categories named at inference time even if they were never in its training classes.

Measuring Accuracy#

Word error rate (WER) counts the substitutions, deletions, and insertions needed to turn the transcript into a reference transcript, divided by the number of reference words. This code computes it with the word-level edit distance:

import numpy as np

reference = ["turn", "on", "the", "kitchen", "lights"]
hypothesis = ["turn", "of", "the", "kitchen", "light", "please"]

# Edit distance over words: d[i, j] is the cheapest way to turn the first i reference words into the first j hypothesis words
d = np.zeros((len(reference) + 1, len(hypothesis) + 1), dtype=int)
d[:, 0] = np.arange(len(reference) + 1)  # deletions
d[0, :] = np.arange(len(hypothesis) + 1)  # insertions
for i in range(1, len(reference) + 1):
    for j in range(1, len(hypothesis) + 1):
        substitution = d[i - 1, j - 1] + (reference[i - 1] != hypothesis[j - 1])
        d[i, j] = min(substitution, d[i - 1, j] + 1, d[i, j - 1] + 1)

# Word error rate: (substitutions + deletions + insertions) / words in the reference
print(f"{d[-1, -1]} errors / {len(reference)} words = WER {d[-1, -1] / len(reference):.0%}")

It prints 3 errors / 5 words = WER 60%: "of" for "on" and "light" for "lights" are substitutions, and "please" is an insertion. Because insertions count, WER can exceed 100%, and benchmarks such as the Open ASR Leaderboard normalize case, punctuation, and spelling variants before scoring so that "Lights." and "lights" match. Languages written without spaces between words, such as Chinese and Japanese, are scored with character error rate instead.