Speech Generation Research Wiki

Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment

7 min read

ACL · 2025 · Conference

Xueyao Zhang et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ✓ · Code: ✓

This paper introduces INTP, a ~250K-pair preference dataset targeting TTS intelligibility failures in hard-to-pronounce text, code-switching, and cross-lingual scenarios, and extends Direct Preference Optimization (DPO) to autoregressive, flow-matching, and masked generative TTS architectures, demonstrating consistent WER reduction and naturalness improvement across five state-of-the-art zero-shot TTS systems.

Problem

Modern zero-shot TTS systems — despite large-scale pre-training — exhibit systematic intelligibility failures in out-of-distribution scenarios: repeated or tongue-twister text (hallucinated insertions and omissions), code-switching (mixed-language utterances), and cross-lingual synthesis (reference speaker speaks language A, target text in language B). Standard pre-training on high-quality monolingual data does not generalize to these conditions, and collecting matched high-quality data for challenging domains is difficult. Prior preference alignment work for TTS had two limitations: (1) datasets built from a single model’s self-comparisons, missing cross-model complementarity, and (2) alignment algorithms designed only for autoregressive architectures, leaving flow-matching and masked generative models without a principled DPO formulation.

Method

The paper makes two inter-dependent contributions: the INTP dataset and architecture-specific DPO extensions.

INTP dataset construction. The authors use three open-weight zero-shot TTS models with distinct architectures as generators: ARS (autoregressive, from the MaskGCT paper), F5-TTS (flow-matching), and MaskGCT (masked generative). Text prompts span five types: regular, repeated, code-switching, pronunciation-perturbed (homophone substitutions that confuse TTS), and punctuation-perturbed (comma-induced prosody changes). Reference speech is stratified-sampled from Emilia-Large (~100K hours EN+ZH). Preference pairs are built three ways:

  • Intra pairs: For each model and prompt, five samples are generated with varying temperature/duration hyperparameters; the pair with greatest WER gap (threshold ≥ 6.0) forms a positive/negative pair.
  • Inter pairs: Cross-model comparisons using the same prompts — the best sample from model A versus the best from model B, leveraging complementary model strengths.
  • Perturbed pairs: DeepSeek-V3 generates phonetically corrupted or punctuation-altered versions of the input text; the model synthesis on the original text is positive, on the perturbed text is negative. This injects human-guided knowledge beyond WER.

Total: ~251K pairs (over 2K hours), with approximately equal intra and inter pairs.

DPO for diverse architectures. Standard DPO (Rafailov et al., 2023) applies to autoregressive models via log-probability ratios. The paper derives analogous objectives for:

  • Flow-matching DPO (DPO-FM): Derives the RL objective under the OT-FM policy, showing that the log-ratio of learned vs. reference velocity fields at each timestep t is equivalent to the MSE difference in predicted velocities. This transforms the DPO loss into a velocity-space objective that can be computed without sampling.
  • Masked Generative Model DPO (DPO-MGM): Defines the policy as the product of log-probabilities over masked tokens and derives the analogous DPO loss operating on masked token distributions.

Target models. ARS, F5-TTS, and MaskGCT are aligned using INTP (they also participated in INTP construction). CosyVoice 2 (Qwen2.5-0.5B LLM backbone) and the authors’ own Ints model (Phi-3.5-mini-instruct 3.8B + DualCodec 12.5Hz tokens) are used to test weak-to-strong generalization — can data built from weaker generators improve stronger models not in the data distribution? Iterative alignment is also evaluated: Ints v1 (INTP-aligned) generates new preference data, which trains Ints v2.

Evaluation. Objective metrics: WER (Whisper-large-v3 for English, Paraformer-zh for Chinese/code-switching), SPK-SIM (WavLM TDNN cosine similarity), UTMOS (automatic naturalness predictor). Subjective: N-CMOS (naturalness comparative MOS, rated -2 to +2 relative to baseline), reading accuracy (binary intelligibility judgment), and A/B testing for speaker similarity.

Key Results

All five models improve on WER after INTP alignment across all four domain types. Key figures from Table 4:

  • ARS: Average WER drops from 24.47 to 15.41 (37% relative reduction); cross-lingual WER 19.76 → 9.57 (52%); N-CMOS +0.64 on articulatory, +0.82 on cross-lingual.
  • F5-TTS: Average WER 17.78 → 9.62 (46% relative); code-switching WER 33.99 → 15.98 (53%).
  • MaskGCT: Average WER 14.04 → 9.73 (31%).
  • CosyVoice 2 (weak-to-strong): Average WER 13.09 → 10.56; N-CMOS improvements consistent with weaker models.
  • Ints (weak-to-strong): Average WER 11.97 → 7.96 (34%); Ints v2 (iterative) reaches 7.10 average WER.
  • Unseen languages (Ints): INTP alignment on EN+ZH data generalizes to Japanese, Korean, German, French, reducing WER substantially in all four (Table 10).

SPK-SIM is essentially unchanged (±0.01) across all systems, and N-CMOS is uniformly positive, confirming that intelligibility gains do not trade off naturalness or speaker fidelity. Ablations (Table 5) show that all three pair types contribute, that inter-pairs and intra-pairs are complementary, and that supervised fine-tuning on positive samples alone provides partial gains, with the full DPO (negative pair contrast) giving larger improvements.

Novelty Assessment

The primary contributions are:

  1. The INTP dataset design, particularly the inter-pair (cross-model comparison) and perturbed-pair (human-guided negative) strategies, which go beyond prior single-model WER-based preference collection.
  2. DPO-FM and DPO-MGM derivations, making preference alignment applicable to non-autoregressive TTS architectures for the first time in a principled way.
  3. Empirical validation of weak-to-strong generalization in TTS alignment — INTP built from ARS/F5-TTS/MaskGCT improves the stronger CosyVoice 2 and Ints.

The paper is primarily a training-recipe and dataset contribution; the underlying architectures are all taken from prior work. The DPO-FM derivation follows a known pattern (Wallace et al. 2024 for diffusion), but applying it to TTS is novel. The scale of INTP (250K pairs, 2K hours) substantially exceeds prior preference TTS datasets.

Field Significance

Tip

High — INTP provides the first large-scale preference dataset explicitly targeting out-of-distribution intelligibility failures in zero-shot TTS, and the DPO-FM and DPO-MGM derivations enable preference alignment for non-autoregressive architectures that previously lacked a principled formulation. The weak-to-strong generalization finding — that data built from weaker generators improves stronger models outside the training distribution — establishes a scalable recipe for iterative TTS alignment that can serve as a template for future work across architectures.

Claims

  • Preference alignment with cross-model preference pairs yields larger intelligibility improvements than single-model self-comparison pairs for TTS post-training. (§5.2, Table 5)
  • DPO objectives can be derived for flow-matching and masked generative TTS architectures by operating in velocity space and masked token distributions, respectively, without requiring explicit reward models. (§4.2, §4.3)
  • Intelligibility gains from preference alignment on English and Chinese data generalize to unseen languages (Japanese, Korean, German, French) without degrading speaker similarity. (§5.1, Table 10)
  • Preference datasets constructed from weaker TTS generators effectively improve stronger models outside the training distribution, demonstrating weak-to-strong generalization in speech alignment. (§5.1, Table 4)
  • Iterative preference alignment yields diminishing returns across rounds, suggesting that base model capability is the primary ceiling on alignment effectiveness. (§5.3, Table 6)

Limitations and Open Questions

  • INTP covers only English and Chinese; while generalization to unseen languages is shown, coverage of low-resource languages remains weak.
  • The computational cost of alignment (8 H100 GPUs per run) limits accessibility for smaller labs.
  • Iterative alignment shows diminishing returns between rounds, suggesting the base model’s capability is the ceiling. Better foundation models will be needed.
  • The paper does not evaluate anti-spoofing robustness; improved intelligibility alignment may inadvertently affect spoofing resilience.
  • All evaluation uses the SeedTTS test sets, which are English and Chinese. No evaluation on truly out-of-distribution natural speech (e.g., radio or phone call recordings) is provided.

Wiki Connections

This paper is a major contribution to rlhf-speech, filling the previously empty concept page with a concrete dataset (INTP), three architecture-specific DPO variants, and empirical results across five systems. It directly advances zero-shot-tts by addressing the intelligibility gap in cross-lingual and code-switching scenarios — an open question explicitly noted in the zero-shot-tts concept page. The multilingual and cross-lingual evaluation links to multilingual-tts, and the use of WER, SPK-SIM (WavLM TDNN), UTMOS, and N-CMOS across four domain types extends evaluation-metrics and subjective-evaluation.

  • The paper builds on F5-TTS, which is one of the three generator models used in INTP construction and is aligned using DPO-FM.
  • It references Emilia as the source dataset for both INTP prompts and the pre-training of Ints.
  • The flow-matching DPO formulation is a TTS application of the technique introduced for diffusion models in Wallace et al. (2024), connecting to flow-matching.

Backlinks

Graph View