arXiv · 2024 · Preprint
Peng et al. · → Paper · Demo: ✓ · Code: ✓
VoiceCraft introduces a token infilling neural codec language model that unifies speech editing and zero-shot TTS via a novel token rearrangement procedure, enabling autoregressive generation conditioned on bidirectional context from in-the-wild recordings.
Problem
Autoregressive codec language models (NCLMs) are naturally suited to speech continuation but cannot straightforwardly handle in-sequence infilling: generating tokens that must fit between existing speech segments requires conditioning on future context, which standard left-to-right decoding cannot provide. Prior speech editing models addressed this through masked reconstruction objectives or modular pipelines, but these approaches struggled with naturalness at boundaries and with the diversity of real-world recording conditions found in podcasts, YouTube videos, and audiobooks. No open model had simultaneously achieved strong speech editing and zero-shot TTS results on in-the-wild, multi-domain data with human evaluation confirming near-indistinguishability from real recordings.
Method
VoiceCraft reframes both speech editing (infilling) and zero-shot TTS (continuation) as left-to-right language modeling over rearranged neural codec token sequences. The core mechanism is a two-step token rearrangement procedure applied before autoregressive modeling:
In the first step (causal masking), the spans to be generated are relocated from their original position in the codec matrix to the end of the sequence. The unmasked spans remain in place, with placeholder mask tokens inserted where the edits should occur. This transformation makes the generation task causal: the model generates the target tokens after having seen all unmasked context, including tokens that are temporally future in the original utterance.
In the second step (delayed stacking), tokens from different RVQ codebooks within each span are staggered by their codebook index. This delay pattern, adapted from MusicGen, ensures that when predicting codebook k at time t, the model can condition on the prediction for codebook k-1 at the same timestep, improving multi-codebook coherence without resorting to a fully flattened sequence.
The backbone is a decoder-only Transformer with 16 layers, hidden/FFN dimensions of 2048/8192, and 12 attention heads, totalling 830M parameters. EnCodec (50Hz framerate, 4 RVQ codebooks with vocabulary size 2048) provides the speech token representation. Text transcripts are phonemized using IPA. At each time step, K parallel MLP heads project the transformer’s final hidden state to logits over each codebook, with training loss weighted toward lower-indexed codebooks (weights 5, 1, 0.5, 0.1) to favour intelligibility. The model is trained on 9k hours of GigaSpeech (audiobooks, podcasts, YouTube) using the ScaledAdam optimizer.
For speech editing inference, word-level forced alignment identifies the codec spans to mask, a small margin parameter extends the mask on both sides to smooth coarticulation, and the model runs 10 inference passes with varying margins, discarding the 4 longest outputs to mitigate a repeating-loop pathology. For zero-shot TTS, the voice prompt and its transcript are concatenated with the target transcript, and the model generates via continuation; 5 samples are drawn and the shortest is selected.
Key Results
On speech editing evaluated on the REALEDIT dataset (310 in-the-wild examples from audiobooks, YouTube, and Spotify podcasts), VoiceCraft achieves a naturalness MOS of 4.03 versus 3.81 for the scaled FluentSpeech baseline. In side-by-side comparison, human listeners prefer VoiceCraft over FluentSpeech 56.1% of the time (with 19.7% ties). Notably, listeners judge VoiceCraft’s edited speech to be equally or more natural than the original unedited recording 56.4% of the time, with the original preferred 43.6% of the time (Table 5, §5.3).
On zero-shot TTS (250 prompts from LibriTTS and YouTube), VoiceCraft achieves naturalness MOS 4.17 versus 3.96 for XTTS v2 and 3.86 for VALL-E, with speaker similarity MOS of 4.34 versus 4.07 for VALL-E (Table 6, §5.4). WER is 4.5% (versus ground truth 3.8%), indicating near-ground-truth intelligibility.
VALL-E results use an open-source reimplementation (not the original Microsoft model) trained on GigaSpeech for comparability. XTTS v2 is a commercial model trained on unknown proprietary data. Comparisons with FluentSpeech use a scaled-up version retrained on GigaSpeech to match training conditions.
Ablation over model sizes (120M, 430M, 830M) shows consistent gains from scaling across WER, MCD, F0, and energy distance metrics. Codebook reweighting improves intelligibility (WER, MCD) at a small cost to prosody metrics (F0, Energy) (Table 3, §5.2).
A recurrent observation across both tasks: FluentSpeech achieves lower WER than the ground truth (3.5% vs 3.8% for TTS; 4.5% vs 5.4% for editing), yet receives substantially lower intelligibility MOS from human listeners. This suggests that ASR-based WER can be a misleading proxy for perceptual intelligibility when systems produce dull or robotic speech that ASR models handle well.
Novelty Assessment
The primary contribution is the token rearrangement procedure (causal masking combined with delayed stacking), which is a genuine architectural innovation. The insight that span infilling can be recast as left-to-right continuation by relocating masked spans to the sequence end is clean and general, drawing inspiration from CM3 (causal masked multimodal models) and MusicGen’s delay pattern, but the application to speech codec sequences for unified editing and TTS is new.
The unified framing of speech editing and zero-shot TTS as the same operation (infilling versus end-of-sequence insertion) is elegant and avoids task-specific engineering. The open release of code, model weights, and the REALEDIT evaluation dataset adds practical value.
The REALEDIT dataset is a genuine evaluation contribution: 310 manually crafted editing examples from three domains with diverse accents, recording conditions, and background sounds, covering insertion, deletion, and substitution at multiple span lengths. It is a more challenging and realistic benchmark than the audiobook-only datasets previously standard for speech editing evaluation.
The reliance on a multi-sample-and-discard strategy for handling generation artifacts (looping silence) is a practical workaround but not an elegant solution. Model size and training data scale are more moderate compared to contemporaneous commercial systems.
Field Significance
Tip
High — VoiceCraft demonstrates that a unified autoregressive codec LM can achieve near-ground-truth naturalness on in-the-wild speech editing and competitive zero-shot TTS, with the core token rearrangement technique offering a general solution to the infilling-via-continuation problem for sequence-level codec generation. The REALEDIT benchmark provides the field with a more realistic and challenging evaluation standard for speech editing than audiobook-only alternatives, and the full open release (code, weights, dataset) enables reproducible follow-up.
Claims
- Autoregressive codec language models can perform speech infilling with naturalness approaching that of the original unedited recording when masked spans are relocated to the end of the sequence, enabling bidirectional context conditioning within a causal framework. (§3.1, §5.3, Table 5)
- Zero-shot TTS and speech editing can be unified as a single autoregressive infilling operation without task-specific architectural components, at no cost to performance on either task. (§3.4, §5.4, Table 6)
- WER measured by ASR systems is an unreliable proxy for perceptual intelligibility when evaluating speech synthesis quality: systems can achieve lower WER than ground truth recordings while receiving substantially lower intelligibility ratings from human listeners. (§5.3, §5.4)
- Evaluation of speech synthesis exclusively on audiobook data underestimates the performance gap between systems when applied to in-the-wild recordings with diverse accents, noise, and speaking styles. (§5.3, §5.4)
- Scaling autoregressive codec LM parameters consistently improves objective metrics across intelligibility and acoustic fidelity measures, with larger gaps between larger model sizes suggesting further gains from continued scaling. (§5.2, Table 3)
Limitations and Open Questions
Warning
The inference-time artifact mitigation (generating 10 candidates and discarding the 4 longest) adds significant latency and compute cost. The strategy is acknowledged as inelegant, and the underlying cause (repetitive loop generation) is unresolved. This limits practical deployment in real-time or low-compute settings.
Additional limitations include:
- The model is trained and evaluated in English only; generalisation to other languages is untested.
- Zero-shot speaker similarity, while competitive, still falls short of ground truth (MOS 4.34 vs 4.44), particularly for in-the-wild YouTube recordings where acoustic conditions are challenging.
- REALEDIT covers only English and is limited to 310 examples, restricting statistical power for fine-grained analysis by edit type and domain.
- The model does not incorporate explicit watermarking or deepfake detection, which the authors identify as an AI safety gap given the model’s voice cloning capability.
Wiki Connections
Concepts: autoregressive-codec-tts | zero-shot-tts | neural-codec | evaluation-metrics | subjective-evaluation
Related papers: 2301.02111 (VALL-E, the primary NCLM baseline) | 2210.13438 (EnCodec, the codec used) | 2010.05646 (HiFi-GAN) | 2402.01912 | 2303.03926 | 2305.09636