arXiv · 2025 · Preprint
Peng et al. (Microsoft Research) · → Paper · Demo: ✓ · Code: ✓
VibeVoice is a long-form multi-speaker TTS system that combines an ultra-low frame rate acoustic VAE tokenizer (7.5 Hz) with a next-token diffusion LLM framework, enabling continuous generation of up to 90 minutes of conversational audio across up to four speakers within a 64K-token context window.
Problem
Existing TTS systems excel at producing high-fidelity short utterances from single speakers, but scaling to long-form, multi-speaker conversational content — podcasts, multi-participant audiobooks — exposes fundamental limitations. Concatenating individually synthesised utterances produces unnatural turn-taking and fails to capture content-aware prosodic dynamics across speakers. The few systems that address this domain either remain proprietary or struggle with generation stability and context length. A key bottleneck is computational: existing neural codecs operate at 50–600 tokens per second, making a 90-minute context prohibitively long for an LLM backbone.
Method
VibeVoice (paper name: VIBEVOICE) resolves the context-length bottleneck through two innovations that work in tandem: an ultra-compressed acoustic tokenizer and a next-token diffusion generation framework.
The acoustic tokenizer is a causal VAE with a hierarchical encoder-decoder architecture. The encoder uses seven stages of modified Transformer blocks where self-attention is replaced with 1D depth-wise causal convolutions, followed by six downsampling layers that yield a 3200× compression rate — 7.5 Hz from 24 kHz audio, or roughly one acoustic token per two BPE text tokens. The VAE follows the σ-VAE variant from LatentLM, with a pre-defined rather than learnable variance schedule to ensure robust variance for autoregressive modelling. Each encoder/decoder has approximately 340M parameters and is trained with DAC-style discriminator and perceptual losses. A separate semantic tokenizer mirrors the encoder architecture but is trained via an ASR proxy task, producing content-aligned representations; this component is discarded after pre-training and used only to provide hybrid acoustic-semantic context tokens during generation.
For generation, VibeVoice uses a pre-trained LLM (Qwen2.5, instantiated at 1.5B or 7B parameters) as the sequence model. Input is constructed by concatenating voice prompt acoustic features and text scripts for each role, interleaved with speaker identity tokens. The LLM processes this hybrid context and produces hidden states that condition a lightweight token-level diffusion head (4 transformer layers) adapted from the MAR image generation framework. During inference, the diffusion head iteratively denoises a Gaussian noise vector toward the acoustic VAE features using DPM-Solver++ with a guidance scale of 1.3 and 10 denoising steps, then the acoustic decoder reconstructs waveform. Training uses curriculum learning, progressively expanding the LLM context from 4K to 64K tokens; only the LLM and diffusion head are trained — the tokenizers are frozen.
Key Results
On a custom podcast test set of 8 long conversational transcripts (approximately 1 hour total), VibeVoice-7B achieves average MOS scores of 3.71 (Realism), 3.81 (Richness), and 3.75 (Preference), outperforming ElevenLabs v3 alpha (3.40 average), Gemini 2.5 Pro Preview TTS (3.66 average), Sesame CSM (2.89 average), and Higgs Audio V2 (2.99 average). On WER, VibeVoice-1.5B achieves 1.11% (Whisper) — the lowest among all systems evaluated, including Gemini. Speaker similarity (WavLM-large) for VibeVoice-7B is 0.692, competitive with Sesame CSM (0.685) and Higgs Audio V2 (0.543), though Gemini’s SIM is not reported.
On the SEED short-utterance benchmark (CommonVoice English/Chinese), VibeVoice-1.5B achieves CER of 1.16% and SIM of 0.744 for Chinese, and WER of 3.04% and SIM of 0.689 for English, despite being primarily trained for long-form content. These figures are competitive with CosyVoice 2 and Spark-TTS on Chinese, though the 7.5 Hz frame rate trades some English WER performance (3.04% vs. MaskGCT’s 2.62%) for dramatic reduction in decoding steps.
The acoustic tokenizer comparison (Table 3) shows that at 7.5 Hz — 40× fewer tokens than WavTokenizer at 40 Hz — the proposed tokenizer achieves PESQ 3.068 and UTMOS 4.181 on LibriTTS test-clean, beating all discrete codec baselines including DAC (UTMOS 3.433) and WavTokenizer 75 Hz (UTMOS 4.049).
Note
The subjective evaluation covers only 8 test conversations and 24 annotators. Gemini 2.5 Pro is evaluated without speaker prompt control (default voices), making direct SIM comparison impossible. VibeVoice’s training data composition is not disclosed, limiting reproducibility assessment.
Novelty Assessment
The key architectural contribution is the combination of an ultra-low frame rate continuous VAE tokenizer with a token-level diffusion head conditioned on LLM hidden states. The 7.5 Hz frame rate is a genuine advance over prior low-rate codecs (WavTokenizer at 40–75 Hz, EnCodec at 75–600 Hz) and is what makes 64K-token context windows for hour-long audio tractable. The next-token diffusion approach is not invented here — it follows the LatentLM framework from concurrent work — but applying it to speech with a purpose-designed tokenizer is the paper’s core contribution.
The hybrid acoustic-semantic tokenizer design (separate specialised codecs trained with different objectives, frozen during generation training) follows an established pattern from SpeechTokenizer and related work, but the causal streaming architecture and extreme compression rate are new. The removal of the VQ bottleneck in favour of continuous VAE features bypasses the codebook collapse problem that has limited single-codebook discrete tokenisers.
Scaling from 1.5B to 7B LLM shows consistent gains across all subjective dimensions and speaker similarity, providing moderate evidence that LLM scale generalises to this generation paradigm. The paper is a technical report with undisclosed training data, which limits the claims’ generalisability.
Field Significance
Tip
High — VibeVoice demonstrates that extreme codec compression (7.5 Hz, continuous VAE) combined with LLM-scale sequence modelling enables a qualitatively new capability: hour-long, multi-speaker conversational TTS that outperforms both open-source and commercial systems on subjective preference. The 7.5 Hz tokenizer is a meaningful technical contribution that may influence how the field approaches long-context speech generation; the next-token diffusion architecture illustrates that continuous-space token prediction via diffusion heads can rival discrete autoregressive systems without requiring a codebook.
Claims
- Extreme acoustic codec compression (single-codebook VAE at 7.5 Hz) can achieve superior perceptual quality over multi-codebook discrete codecs operating at much higher frame rates. (§3.3, Table 3)
- Long-form multi-speaker TTS benefits from separate acoustic and semantic tokenizers trained with task-specific objectives rather than a single unified codec. (§2.1)
- Scaling the LLM backbone in a next-token diffusion speech system yields consistent gains in perceptual quality, speaker similarity, and expressiveness. (§3.1, Table 1)
- Token-level diffusion conditioned on LLM hidden states enables streaming speech generation without the codebook constraints of discrete autoregressive systems. (§2.2)
- TTS systems optimised for long-form conversational content retain competitive performance on short-utterance benchmarks without dedicated fine-tuning. (§3.2, Table 2)
Limitations and Open Questions
Warning
Training data is not disclosed. The paper is from Microsoft Research but does not specify the data composition, size, or any cleaning procedures, making it impossible to assess whether the reported gains are attributable to architecture or data advantage.
The model is limited to English and Chinese; other languages produce unpredictable outputs. The system does not model overlapping speech — a significant gap for realistic conversational audio. The subjective evaluation used only 8 long-form test conversations, which is a narrow sample; standard benchmark evaluations (SEED) are short-utterance only and do not capture the long-form quality the paper targets. Speaker similarity at 7.5 Hz remains below the best short-utterance systems (e.g., Seed-TTS at 0.762 SIM for English), suggesting the compressed representation sacrifices some speaker identity fidelity. The maximum of 4 speakers is a hard constraint imposed by the context design.
Wiki Connections
Core concepts this paper advances: autoregressive-codec-tts, neural-codec, diffusion-tts, zero-shot-tts, spoken-language-model, disentanglement
In-corpus papers cited: 2301.02111 (VALL-E, neural codec LM baseline), 2406.02430 (Seed-TTS, evaluation benchmark and comparison), 2412.10117 (CosyVoice 2, streaming TTS baseline), 2502.03930 (DiTAR, diffusion transformer autoregressive baseline)