Speech Recognition

Speech recognition, or automatic speech recognition (ASR), converts spoken audio into text with neural models. It is a synonym of speech-to-text.

Speech recognition, also called automatic speech recognition (ASR) or speech-to-text (STT), is the conversion of spoken language into written text by a computer. A recognizer turns an audio waveform into a sequence of acoustic features, maps those features to characters or word pieces with a neural network, and outputs a transcript. It is the input layer of virtual assistants, live captions, dictation, and call analytics, and its transcripts feed natural language processing (NLP) systems, such as named entity recognition, that search, summarize, or act on what was said. Models trained on very large speech collections, such as OpenAI's Whisper, trained on 680,000 hours of multilingual audio, now transcribe many accents, noisy recordings, and dozens of languages without per-speaker training.

From Templates to Neural Networks#

Bell Labs' Audrey system recognized spoken digits from a single speaker in 1952, and IBM's Shoebox understood 16 English words, including the digits 0 to 9, at the 1962 Seattle World's Fair. From the 1980s, statistical recognizers built on hidden Markov models (HMMs) dominated: Gaussian mixture models scored how well each short audio frame matched each speech sound, the HMM tracked how sounds follow one another, a pronunciation dictionary mapped sounds to words, and an n-gram language model ranked word sequences. Around 2012, research groups at the University of Toronto, Microsoft, Google, and IBM showed that deep neural networks scored the frames more accurately than Gaussian mixture models across several benchmarks, and the same push for better acoustic models produced general tools such as the leaky ReLU activation. End-to-end models followed, replacing the separate components with one network trained directly from audio to text.

How Speech Recognition Works#

Audio is resampled, to 16 kHz for Whisper, and converted into a log-mel spectrogram, a grid of energy per frequency band over time that is computed every 10 milliseconds. An encoder built from convolutional and transformer layers turns these frames into features, and a decoder turns the features into text. Three decoder families are common: connectionist temporal classification (CTC), transducers that emit text while audio is still arriving, and attention encoder-decoders such as Whisper, which generate tokens one at a time. CTC, introduced by Alex Graves and colleagues in 2006, lets the network predict one symbol or a special blank for every audio frame, so training needs no frame-level alignment between audio and text. Decoding then merges repeated symbols and removes blanks:

from itertools import groupby

# Most likely symbol for each audio frame from a CTC acoustic model ("_" is the blank)
frames = "hh_ee_ll_ll_oo__"
no_blanks = "hheelllloo"

for path in (frames, no_blanks):
    # CTC decoding: merge repeated symbols, then remove blanks
    merged = [symbol for symbol, _ in groupby(path)]
    text = "".join(symbol for symbol in merged if symbol != "_")
    print(f"{path:>16} -> {text}")

It prints hh_ee_ll_ll_oo__ -> hello and hheelllloo -> helo. The blank between the two runs of l is what keeps the double letter, and the frame path without blanks collapses into a misspelling. Accuracy is measured with word error rate (WER), the number of substituted, deleted, and inserted words divided by the number of words in a reference transcript, so lower is better and insertions can push it above 100%.