Voice Cloning

Voice cloning is the use of generative AI to produce new speech in the voice of a specific person. Given recordings of a speaker, a cloning system learns what makes that voice recognizable, including its pitch, timbre, accent, and rhythm, and then speaks any new text or translated line with it. Sercan Arık and colleagues at Baidu framed the two approaches still in use in their 2018 paper "Neural Voice Cloning with a Few Samples": adapting a multi-speaker model to the new voice, or encoding the voice into a vector the model conditions on. Modern systems produce a convincing clone from a few seconds of audio, which makes the technology valuable for accessibility and media production and equally useful to fraudsters.

How a Voice Clone Is Built#

Most cloning systems extend text-to-speech (TTS) with a representation of speaker identity. A speaker encoder, trained on thousands of voices, compresses a reference recording into a fixed-length vector called a speaker embedding. Google's 2018 SV2TTS system trained its encoder on speaker verification, deciding whether two recordings come from one person, and needed only a few seconds of an unseen speaker's audio. The TTS model receives the text together with the embedding and predicts an acoustic representation such as a mel spectrogram, and a neural vocoder converts that into an audio waveform.

Systems clone voices in two main ways:

• Zero-shot cloning: Feeds a short reference clip, often 3 to 30 seconds long, to a model trained on many speakers. Nothing is retrained, so a new voice is available instantly, though strong accents and emotional delivery transfer less reliably.
• Fine-tuned cloning: Adapts the model's weights on minutes to hours of clean recordings of one speaker. It needs more data and compute and produces a closer, more stable match, which is why professional narration voices are usually built this way.

Newer systems treat speech as language. A neural audio codec compresses audio into discrete tokens with residual vector quantization, and a transformer language model generates those tokens. Microsoft's VALL-E (January 2023), trained on 60,000 hours of English speech, continues a three-second prompt in the same voice, emotion, and room acoustics; Microsoft did not release it publicly because of the risk of misuse. OpenAI took the same course with Voice Engine, which it previewed in March 2024 as cloning a voice from a 15-second sample. In every approach, noise, clipping, and room echo in the reference come through in the clone.

Cloning vs. Text-to-Speech and Voice Conversion#

Standard TTS speaks in a fixed set of designed or licensed voices, and cloning adds the ability to target one particular person. Voice conversion is a close relative that takes recorded speech as input and swaps the speaker identity while keeping the words, timing, and emotion of the original performance. Speech-to-text runs in the opposite direction, transcribing audio into words. Automated dubbing chains all three: transcribe the original dialog, translate it, then speak the translation in the actor's cloned voice.

A cloned voice used to make someone appear to say something they never said is an audio deepfake.

Voice Banking, Dubbing, and Game Dialog#

• Voice banking with Apple's Personal Voice: People diagnosed with ALS or facing throat surgery record a short set of phrases while they still can, and the iPhone builds a synthetic version of their voice on the device to speak the messages they type.
• Dubbing and narration: Studios and audiobook publishers use authorized clones to localize content into other languages in the original speaker's voice and to record corrected lines without bringing a narrator back to the booth.
• Games and virtual assistants: Developers generate large volumes of character dialog in a consistent voice, with voice actors licensing their clones under contract.

Fraud, Consent, and Detection#

Criminals clone a voice from social media clips, then call a relative or an employee with an urgent request for money, and the reliable defense against these fake emergency scams is to hang up and call the person back on a known number. In January 2024, robocalls with a cloned voice of President Biden told New Hampshire voters to skip the state's primary, and in February 2024 the US Federal Communications Commission ruled that AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, making such robocalls illegal without prior consent. Responsible voice platforms require recorded consent from the voice owner, restrict cloning of public figures, and embed inaudible AI watermarks in generated audio so it can be identified later. C2PA Content Credentials add a signed record of how a media file was created and edited.

Detection combines audio and visual evidence. Audio classifiers look for spectral artifacts that vocoders leave behind, and the ASVspoof challenge series, running since 2015, benchmarks them against new cloning and conversion attacks. Video checks compare lip movement against the soundtrack to catch a cloned voice dubbed over real footage, since a mouth that closes for "b," "m," and "p" sounds must match the audio at the same moment. Those checks begin by finding and following the speaker's face in every frame, a job that a face detector trained with Ultralytics YOLO26 performs in real time in tracking mode, which assigns each face a persistent ID so the lip-sync model receives one continuous sequence of mouth crops per speaker.