arXiv · 2025 · Preprint
Xueyao Zhang et al. (The Chinese University of Hong Kong, Shenzhen / ByteDance Seed / Amphion Technology) · → Paper · Demo: ✓ · Code: ✓
Vevo2 unifies controllable speech and singing voice generation in a single AR+flow-matching framework via novel chromagram-based prosody tokenization and multi-objective GRPO post-training, achieving mutual quality gains across both domains.
Problem
Existing TTS and singing voice synthesis (SVS) systems are trained independently, missing the mutual benefit that speech abundance (for generalization) and singing expressiveness (for prosody richness) could provide to each other. Singing voice generation additionally depends on expert-annotated music notation (MIDI alignments), which limits scalability. There is no prior unified system that simultaneously supports TTS, VC, SVS, SVC, speech editing, singing lyric editing, humming-to-singing, and instrument-to-singing from a single model with continuous, fine-grained controllability over text, prosody, style, and timbre.
Method
Vevo2 uses a two-stage architecture shared with classical zero-shot TTS systems:
Stage 1: AR Content-Style Modeling. A 509M-parameter transformer (initialized from Qwen2.5-0.5B) takes text tokens and optionally prosody tokens as input and generates discrete content-style tokens. Two training modes are applied randomly to all data: Implicit Prosody Learning (IPL, text-only input — equivalent to standard zero-shot TTS) and Explicit Prosody Learning (EPL, text + prosody token input). The randomized mixing across speech and singing data, rather than restricting IPL to speech and EPL to singing, is the key unification mechanism.
Stage 2: Flow-Matching Acoustic Modeling. A 363M-parameter FM transformer converts content-style tokens into Mel spectrograms conditioned on a timbre reference. A Vocos-based vocoder (255M params, fine-tuned jointly on speech and singing) converts spectrograms to waveforms.
Two custom VQ-VAE tokenizers are introduced:
- Prosody tokenizer (6.25 Hz, 56.25 bps, codebook 512): encodes chromagram features, which are octave-free (bridging the F0 distribution gap between speech and singing) and notation-free (extractable without MIDI annotation). Used for melody/prosody control.
- Content-style tokenizer (12.5 Hz, 175 bps, codebook 16384): reconstructs both chromagram features and Whisper encoder representations, capturing linguistic content, melody, and style while disentangling timbre via an information bottleneck.
Multi-objective post-training applies GRPO optimization with a composite reward: (1) intelligibility reward (Bradley-Terry model trained on INTP preference corpus) and (2) prosody similarity reward (chromagram cosine similarity between generated and ground-truth singing). Single-objective optimization degrades the complementary objective (intelligibility-only training drops melody accuracy from 65% to 50%); joint optimization achieves both gains.
Inference-time controllability: The system flexibly recombines text, prosody source, style reference, and timbre reference to perform TTS, VC, SVS, SVC, speech/singing editing, duration control (97%+ accuracy via chromagram length scaling), and pitch region control (via F0 shift before prosody token extraction).
Key Results
On the SeedTTS benchmark (regular speech), post-trained Vevo2 achieves WER 3.64 (en) / 2.94 (zh) and SIM-o 0.693 / 0.754, competitive with CosyVoice 2 and MaskGCT despite using a lower frame-rate (12.5 Hz vs 25–50 Hz) content-style tokenizer. On expressive speech (Genshin-Voice), Vevo2 matches baselines with WER 11.48 and SIM 0.689. On singing voice, Vevo2 substantially outperforms zero-shot TTS baselines in N-CMOS (gap > 1.5) and SS-CMOS (gap > 0.9), validating the benefit of joint training. For zero-shot SVS, Vevo2 achieves WER 24.77 (en) / 9.83 (zh) with Melody-MOS > 2.0 (indicating at least rough melody following), outperforming TCSinger in intelligibility while remaining competitive in melody tracking. Duration consistency is 97.1–98.4% across speech editing, singing lyric editing, and SVC tasks (Table V). Pitch shift control significantly improves SIM in both VC and SVC. Multi-objective post-training raises text accuracy on instrument-to-singing from 75% to 90%.
Novelty Assessment
The core architectural novelty is the chromagram-based unified prosody tokenizer, which bridges speech and singing prosody spaces without MIDI annotation. This is genuinely new — prior tokenizers use F0 or mel-subband features that fail for singing. The EPL/IPL mixed training strategy is a principled mechanism for joint domain unification. The GRPO multi-objective post-training applied to a unified speech-singing model is also novel (prior post-training work addresses TTS only). The two-stage AR+FM architecture itself is incremental (following Vevo, CosyVoice, Seed-TTS), but the specific tokenizer design and unification objectives are substantial contributions.
Field Significance
Tip
High — Vevo2 introduces the first notation-free prosody tokenizer capable of bridging speech and singing prosody spaces, enabling scalable joint training without MIDI annotations. This tokenizer design and the EPL/IPL unification strategy provide a practical template for future unified voice generation frameworks. The multi-objective GRPO post-training extends preference alignment beyond standard TTS to cover melody following, opening a new alignment axis for expressive voice systems.
Claims
- Joint pre-training on speech and singing voice data improves generation quality in both domains, with singing data contributing to expressive speech prosody richness and speech data compensating for the limited size of singing corpora. (§V-A, Table I)
- Chromagram-based prosody tokenization provides a notation-free, octave-invariant representation that bridges the F0 distribution gap between speech and singing, enabling unified prosody control without expert MIDI annotations. (§III-A)
- Autoregressive models initialized from LLM checkpoints can achieve competitive zero-shot TTS quality at low codec frame rates (12.5 Hz) when paired with strong post-training, partially compensating for the sequence granularity disadvantage relative to higher frame-rate systems. (§V-A, Table VII)
- Single-objective preference alignment in multi-capability speech models degrades complementary objectives; joint multi-objective optimization is necessary to avoid quality regressions on non-targeted capabilities. (§V-F, Figure 5)
- Inference-time pitch shift applied to prosody tokens improves speaker similarity in voice and singing conversion tasks, at the cost of a minor intelligibility degradation due to train-inference distribution mismatch. (§V-E, Table VI)
Limitations and Open Questions
- Training-inference mismatch when applying pitch shift: content-style tokens from pitch-shifted audio are out-of-distribution, causing slight WER degradation. The paper flags future work to include pitch shift augmentation during training.
- Singing voice data used (7K hours source-separated from in-the-wild songs) may have variable quality.
- The 12.5 Hz frame rate of the content-style tokenizer slightly limits VC intelligibility compared to Vevo-FM (50 Hz), as shown in Table VIII.
- Melody-MOS for Vevo2 in SVC is slightly lower than the FM-only Vevo2-FM, suggesting a quality-style tradeoff when adding text input.
- Generalization to tonal languages in singing is not specifically evaluated.
Wiki Connections
- Seed-TTS and CosyVoice provide the two-stage AR+FM architectural foundation that Vevo2 builds upon and extends to the singing domain.
- CosyVoice 2 serves as the primary zero-shot TTS baseline; Vevo2 achieves competitive regular speech quality despite operating at half its content-style tokenizer frame rate.
- VALL-E is the originating paper for the AR codec language model lineage that underpins Vevo2’s AR content-style modeling stage.
- IndexTTS2 is an in-corpus AR-based TTS system situating Vevo2 within the current state of zero-shot speech generation.
- The chromagram prosody tokenizer and EPL/IPL training contribute new evidence to the disentanglement and prosody-control concept spaces.
- The multi-objective GRPO post-training extends the rlhf-speech paradigm to cover melody following in addition to intelligibility alignment.
- The unified zero-shot synthesis capability across speech and singing connects to the zero-shot-tts and autoregressive-codec-tts concept spaces.