ICLR · 2025 · Conference
Zhang et al. (Meta AI) · → Paper · Demo: ✓ · Code: ?
Vevo achieves controllable zero-shot voice imitation by decomposing speech into three independently manipulable attributes (content, style, timbre) through a fully self-supervised VQ-VAE tokenization scheme applied to HuBERT features, enabling accent conversion, emotion conversion, voice conversion, and TTS from a single framework trained without any style-annotated data.
Problem
Existing voice imitation approaches suffer from two entangled problems: they rely heavily on annotated data (parallel corpora, style labels, text transcriptions) to learn speaking style, and they fail to cleanly separate timbre from style in latent representations. When content representations from self-supervised models (like HuBERT) are used directly, timbre leaks into the representation, making independent control of who speaks and how they speak difficult. Prior zero-shot TTS systems that used in-context learning could clone all attributes of a reference prompt but could not target specific attributes (timbre only, style only) independently.
Method
Vevo is a two-stage framework that addresses both problems through a self-supervised tokenization strategy and a modular generation pipeline. At its core, the approach treats the VQ-VAE codebook vocabulary size as a controllable information bottleneck applied to HuBERT-Large 18th-layer features: a large vocabulary (4,096 entries) produces content-style tokens that preserve style and content but strip out timbre; a small vocabulary (32 entries) produces content tokens that strip out both timbre and most style, retaining primarily linguistic content. The key insight, borrowed conceptually from AutoVC’s bottleneck idea but implemented self-supervisedly via VQ-VAE, is that as vocabulary size decreases, the model is forced to discard increasingly high-frequency attribute information in order (timbre first, then style, then content).
The first stage, content-style modeling, uses a 463M-parameter autoregressive transformer (LLaMA-style, 12 layers) to generate content-style tokens from the input content tokens, prompted by a style reference. A duration reduction step collapses consecutive duplicate content tokens before feeding them to the model, which further suppresses unit-level duration patterns that encode style, and also shortens inference sequence length. The global style encoder (WavLM-based TDNN) extracts a style embedding from the reference for use in generation. Two inference modes are supported: reference-style-enhanced continuation (which prepends the style reference’s content-style tokens to the input for maximum fidelity) and reference-global-guided continuation (which uses only the global style embedding, reducing inference input length to 42%).
The second stage, acoustic modeling, uses a 334M-parameter flow-matching transformer to generate mel spectrograms from content-style tokens, conditioned on a timbre reference mel spectrogram via a span-masking strategy inspired by Voicebox. The model learns to fill in the masked source speech given the unmasked timbre reference. At inference, the entire source spectrogram is masked, and the timbre reference is appended. A BigVGAN vocoder (112M, fine-tuned on 60K hours) produces the final waveform.
By adjusting which tokens and references are provided at inference time, the same pre-trained models support four task variants: Vevo-Timbre (timbre-only conversion using content-style tokens from source), Vevo-Style (style-only imitation, content from source, timbre from source), Vevo-Voice (voice conversion, both style and timbre from reference), and Vevo-TTS (text-to-speech synthesis using text-derived content tokens). All training uses only audiobook speech data (60K hours, same corpus as Voicebox) with no style-specific fine-tuning.
Key Results
On zero-shot timbre imitation (AB+CV test sets), Vevo-Timbre achieves WER 2.97%, SPK-SIM 0.420, and N-MOS 3.35, outperforming HierSpeech++, LM-VC, UniAudio, and FACodec on most metrics including speaker similarity (SS-MOS 3.36) and intelligibility. Vevo-Voice additionally leads on style imitation metrics across voice imitation baselines.
For zero-shot style imitation (Table 4), Vevo-Style outperforms accent conversion baselines (ASR-AC, VoiceShop, Conv-Speak) and emotion conversion baseline (Emovox) in N-CMOS and style accuracy metrics (A-ACC, A-CMOS, E-ACC, E-CMOS) despite using no parallel corpus, no style labels, and no textual transcriptions.
For zero-shot TTS (Table 5, ACCENT+EMOTION samples), Vevo-TTS achieves emotion similarity MOS of 4.03 and accent similarity MOS of 4.12, surpassing CosyVoice (trained on 171K hours of in-the-wild data) and MaskGCT (trained on 100K hours) on these style dimensions, while being trained only on 60K hours of audiobooks. WER is slightly higher than non-AR systems (12.07% vs. Voicebox 9.41% on this subset), reflecting a known weakness of AR generation.
Ablations (Table 6) confirm that duration reduction improves style imitation and significantly reduces inference input length. Reducing content tokenizer vocabulary from 32 to 16 causes a large WER increase (Table 8), marking the linguistic content boundary.
Novelty Assessment
The core novelty is the use of VQ-VAE codebook vocabulary size as a principled, self-supervised information bottleneck for progressive speech attribute disentanglement, avoiding the need for ASR supervision, speaker labels, or parallel corpora. This is a genuinely new mechanism: prior work used ASR fine-tuning, adversarial training, or mutual information minimization to achieve disentanglement, each requiring additional annotation or training signal. The VQ-VAE bottleneck approach is conceptually clean and empirically well-characterized through the vocabulary size ablation study (Table 2), which clearly tracks the sequence in which timbre, style, and content are stripped.
The two-stage hybrid architecture (AR transformer for style modeling, flow-matching transformer for acoustic modeling) is not itself novel; it follows a pattern established by CosyVoice and related systems. The novelty lies in what conditions each stage: self-supervised, annotation-free tokens rather than ASR-derived semantic tokens. The result is a unified framework covering timbre imitation, style imitation, and voice synthesis without any task-specific fine-tuning, which is a meaningful capability advance over existing systems.
The evaluation is thorough but not without limitations: the style imitation comparison uses demo website samples from baselines rather than a shared, controlled test set, making direct numerical comparison approximate. Training is English-only and on audiobook data, limiting generalizability to expressive or conversational speech.
Field Significance
Tip
High — Vevo demonstrates that the VQ-VAE vocabulary size can serve as a self-supervised information bottleneck for progressive speech disentanglement, eliminating the need for style annotations in controllable voice imitation. The result that a single model trained on unlabelled audiobooks surpasses supervised accent and emotion conversion systems in a zero-shot setting provides strong evidence that annotation-free disentanglement can unlock style transfer capabilities at zero label cost. The unified multi-task architecture covering timbre imitation, style imitation, and TTS from a shared backbone can serve as a reference design for controllable voice generation.
Claims
- The VQ-VAE codebook vocabulary size can function as a self-supervised information bottleneck for progressive disentanglement of timbre, style, and linguistic content in self-supervised speech representations. (§3.1, Table 2)
- Zero-shot style imitation (accent and emotion conversion) without annotation can match or exceed supervised baselines that rely on parallel corpora and style labels. (§4.3, Table 4)
- Hybrid two-stage pipelines combining autoregressive style modeling with flow-matching acoustic generation can decouple style and timbre control more effectively than single-stage approaches that use in-context learning to mimic all speech attributes jointly. (§3.4, Tables 3–5)
- Autoregressive models in zero-shot TTS consistently trade intelligibility (higher WER) for stronger style imitation compared to non-autoregressive alternatives trained on the same data. (§4.4, Tables 5, 9)
- Duration reduction on content tokens improves style transfer fidelity by removing unit-level duration patterns that encode source speaking style. (§4.5, Table 6)
Limitations and Open Questions
Warning
Style imitation evaluations (Table 4) use demo website samples from baseline systems as the test set, meaning evaluation conditions (recording environment, speaker demographics, utterance content) differ between Vevo and baselines. These comparisons are suggestive but not controlled, and the reported improvements should be treated as approximate.
Training is restricted to English audiobook speech (clean, single-domain), and no multilingual or expressive speech experiments are reported. The content-style token vocabulary size (K_s = 4096) and content token vocabulary size (K_c = 32) are empirically selected; the authors note these may not be globally optimal. The AR content-style model has 463M parameters and requires sequential decoding, introducing latency that could be problematic for streaming applications. The self-supervised disentanglement quality depends on HuBERT-Large features, requiring a large pre-trained SSL model as a prerequisite. Style controllability through a single reference utterance may be brittle for rare or highly expressive speaking styles not represented in the audiobook training distribution.
Wiki Connections
- zero-shot-tts — zero-shot timbre, style, and voice imitation from short reference clips without speaker-specific training
- voice-conversion — Vevo-Timbre and Vevo-Voice modes perform timbre and full voice conversion respectively
- disentanglement — VQ-VAE vocabulary size as self-supervised bottleneck for progressive content/style/timbre separation
- self-supervised-speech — HuBERT-Large 18th-layer features are the input to the VQ-VAE tokenization pipeline
- flow-matching — Stage 2 acoustic model uses a flow-matching transformer conditioned on content-style tokens and timbre reference
- autoregressive-codec-tts — Stage 1 content-style model is an AR LLaMA-style transformer generating content-style tokens
- 2301.02111 (VALL-E) — AR codec TTS baseline; Vevo contrasts in-context learning with explicit disentanglement
- 2407.05407 (CosyVoice) — supervised semantic token TTS compared in zero-shot TTS evaluation
- 2402.08093 (BASE TTS) — large-scale TTS compared in zero-shot timbre imitation
- 2406.02430 (Seed-TTS) — compared baseline in zero-shot TTS evaluation