COLING · 2025 · Conference
Haoyu Wang et al. (Zhejiang University) · → Paper · Demo: ✓ · Code: ?
ProsodyFlow is a single-speaker end-to-end TTS model that uses WavLM-extracted prosody features and conditional flow matching in the prosody latent space to achieve MOS 4.23 and WER 1.33% on LJSpeech, surpassing StyleTTS 2 while running faster.
Problem
Neural TTS models struggle to generate diverse and natural prosody. Explicit pitch/energy prediction (FastSpeech 2) produces averaged prosody; variational inference methods (VITS) are constrained by their VAE formulation; reference prosody encoders require reference audio at inference. Self-supervised speech models (WavLM, HuBERT) capture rich acoustic representations including prosodic information, but prior work uses them primarily for content or speaker modeling. The challenge is to sample diverse prosody vectors at inference without requiring a reference utterance, while keeping inference fast.
Method
ProsodyFlow extends the StyleTTS 2 framework (two-stage training, end-to-end GAN-based text encoder + decoder) with two additions:
WavLM Prosody Encoder. A frozen WavLM-Base-plus (pretrained on 94k hours of unlabeled speech) encodes input speech during training. Outputs from all 12 transformer layers are averaged over the sequence length dimension, then processed through a self-attention module and convolutional downsampling to produce a fixed-size prosody vector s. This is used in Adaptive Instance Normalization (AdaIN) layers in the decoder, duration predictor, and pitch predictor.
Conditional Flow Matching (CFM) for Prosody. Rather than sampling a prosody vector from a VAE posterior (StyleTTS 2 approach), ProsodyFlow trains a conditional flow matching model to learn the distribution of prosody vectors s conditioned on PLBert text features h_bert. At inference, the CFM generates a diverse prosody vector s’ by flowing from N(0, I) to the prosody distribution conditioned on text. This enables diverse prosody at inference without reference audio. During training, the CFM loss L_CFM trains the vector field v(x, t | c) to match the conditional flow between noise and WavLM-extracted prosody.
Two-stage training (inheriting StyleTTS 2). Stage 1: encoder-decoder trained with L_mel + L_GAN using ground-truth WavLM prosody and ASR-derived alignment. Stage 2: joint training adding L_dur + L_F0 + L_N + L_CFM + L_GAN with PLBert semantic text features. iSTFTNet is used as the decoder to directly generate waveforms from intermediate representations. AdamW optimizer (lr=5×10⁻⁵, β=(0, 0.99), weight decay 10⁻⁴), 150+100 epochs. CFM uses a first-order Euler solver with 8 NFE at inference.
Single-speaker training. 13,100 LJSpeech clips (~24 hours), split 12,500/100/500 for train/val/test.
Key Results
Table 1 on LJSpeech test set (50 randomly selected samples, 20 professional raters):
| Model | MOS (CI) | MCD | WER | RTF |
|---|---|---|---|---|
| Ground Truth | 4.25 (±0.10) | - | 1.27% | - |
| ProsodyFlow (n=8) | 4.23 (±0.08) | 4.63 | 1.33% | 0.0191 |
| StyleTTS 2 (n=8) | 4.18 (±0.09) | 4.93 | 1.71% | 0.0231 |
| VITS | 3.92 (±0.09) | 5.49 | 3.61% | 0.0376 |
| FastSpeech 2 | 3.83 (±0.09) | 5.77 | 5.47% | 0.0143 |
| DiffProsody | 4.05 (±0.10) | 5.27 | 2.25% | 0.0543 |
ProsodyFlow achieves human-level MOS (4.23 vs. 4.25 for ground truth), the highest among all baselines. WER of 1.33% is the lowest (ground truth: 1.27%), confirming excellent intelligibility. RTF of 0.0191 is faster than VITS (0.0376) and DiffProsody (0.0543), though slightly slower than FastSpeech 2 (0.0143).
Ablation study (Table 2, varying NFE): even n=1 (MOS 3.92, WER 2.61%, RTF 0.0114) competes with baseline models, demonstrating flow matching efficiency. Performance saturates at n=8; n=16 yields minimal gain (MOS 4.23, WER 1.29%) but doubles RTF.
Ablation study (Table 3, component removal): removing prosody flow matching (using a reference encoder instead) causes the largest drop (CMOS -0.27); removing WavLM causes -0.18 CMOS; replacing flow matching with diffusion causes only -0.04 CMOS (consistent quality, faster inference).
Novelty Assessment
ProsodyFlow is an incremental contribution: it combines WavLM prosody features (previously used in other TTS works) with conditional flow matching for prosody sampling (adapting CFM from acoustic feature generation to prosody latent space). The base architecture is StyleTTS 2 with additions. The key novelty is applying CFM specifically to the prosody latent space rather than to full acoustic features, making prosody the stochastic variable. The evaluation is limited to a single single-speaker English dataset (LJSpeech), which is the most common TTS benchmark but restricts generalizability claims. The scope and scale of experiments are modest.
Field Significance
Moderate — ProsodyFlow demonstrates that routing conditional flow matching through a self-supervised prosody representation, rather than through full acoustic features, is a viable path toward diverse prosody generation without reference audio. The approach is incremental relative to StyleTTS 2 and the evaluation is single-speaker and single-dataset, but the ablation results provide clean evidence for both the WavLM prosody encoder and the flow matching components individually.
Claims
- Applying conditional flow matching to a self-supervised prosody latent space rather than to full acoustic features enables diverse prosody generation at inference without requiring a reference utterance. (§2.3, §3.3)
- Self-supervised speech representations (WavLM) provide a more effective prosody conditioning signal for TTS than conventional pitch and energy predictors, as shown by ablation. (§3.4, Table 3)
- Flow matching in a prosody latent space achieves comparable quality to diffusion-based prosody modeling with substantially lower computational cost. (§3.4, Table 3)
- A small number of flow matching function evaluations (n=1) is sufficient to match or exceed legacy TTS baselines on MOS and WER, confirming the sample efficiency of flow matching for prosody. (§3.3, Table 2)
Limitations and Open Questions
The model is validated only on single-speaker LJSpeech; extension to multi-speaker and zero-shot settings is explicitly acknowledged as future work. The model architecture size is not reported. WavLM parameters are frozen throughout training, which may limit adaptation to unusual prosody distributions. The comparison set does not include the most recent flow matching TTS systems (Voicebox, Matcha-TTS, E2 TTS) — only legacy baselines (FastSpeech 2, VITS, StyleTTS 2) and one diffusion model (DiffProsody). The absence of speaker diversity means the prosody variability shown may primarily reflect intra-speaker diversity of LJSpeech rather than generalizable prosodic modeling.
Wiki Connections
ProsodyFlow combines flow-matching for prosody latent space sampling with self-supervised-speech representations (WavLM) for acoustic feature extraction. Its prosody modeling connects directly to prosody-control. The StyleTTS 2 base framework places it adjacent to transformer-enc-dec-tts and gan-vocoder (GAN discriminators are used). The CFM for prosody approach is a conceptual extension of how flow matching is used in Matcha-TTS and Voicebox for full acoustic feature generation.