arXiv · 2025 · Preprint
Kangxiang Xia et al. (Northwestern Polytechnical University) · → Paper · Demo: ? · Code: ?
MPO introduces a multidimensional preference set construction method paired with a cross-entropy regularization term that simultaneously aligns LM-based TTS with intelligibility, speaker similarity, and prosody criteria while preventing the model collapse typical of vanilla DPO fine-tuning.
Problem
LM-based TTS systems have adopted preference alignment (PA) to improve robustness and naturalness, but prior work (e.g. SpeechAlign, UNO, RIO) treats each evaluation dimension—intelligibility, speaker similarity, prosody—separately or merges them with naive combined rankings. Two concrete failure modes emerge: (1) when optimizing across multiple quality dimensions, independently ranking candidates and selecting extreme pairs can produce conflicting preferred/dispreferred sets that are hard to contrast; and (2) vanilla DPO is prone to degenerate solutions where the winning response probability collapses toward zero—effectively catastrophic forgetting of speech synthesis capability—because the global DPO minimum only requires the losing response probability to go to zero, not that the winning response remains plausible.
Method
MPO operates as a post-training alignment stage on top of a LLaMA-style autoregressive LM that predicts speech tokens conditioned on input text tokens. The base LM (24 transformer layers, 16 attention heads, embedding dimension 1024) is trained from scratch on roughly 160,000 hours of speech data combining WenetSpeech4TTS, LibriHeavy, and a web-crawled internal set, then supervised fine-tuned (SFT) on a 2,000-hour high-quality TTS dataset. Speech tokenization uses a single-codebook neural codec with a codebook size of 8,192.
The alignment stage introduces two technical contributions:
Multidimensional preference set. Rather than one preferred and one dispreferred response per input, MPO constructs a preference set: for each input, 10 candidate utterances are generated with elevated temperature to encourage diversity, then each is scored on three axes—CER (via Paraformer ASR), speaker cosine similarity (via WavLM-large fine-tuned for speaker verification), and log F0 RMSE with dynamic time warping (following the ESPnet evaluation script). The preferred set w_set contains the top-scoring sample per axis; the dispreferred set l_set contains the bottom-scoring sample per axis. Hard constraints are applied: preferred candidates must have CER = 0; the margin in speaker similarity and in prosody between preferred and dispreferred must be at least 0.1. If an element appears in both w_set and l_set, it is resolved by substituting the second-best or second-worst candidate for that metric. During training, one sample is drawn at random from w_set and one from l_set to form each preference pair, so the model sees cross-dimensional contrasts rather than per-axis pairs.
Regularized training. The combined training loss is L = λ·L_DPO + L_CE, where L_CE is the standard cross-entropy objective on the preferred sequences. The cross-entropy term acts as a regularizer: it prevents the DPO loss from collapsing the winning-response probability toward zero. The hyperparameter λ is set to 10; the preference optimization runs on a single NVIDIA A6000 GPU with a learning rate of 1×10⁻⁶ for roughly 20,000 steps.
Key Results
All evaluations are on an internal Mandarin test set. Results are reported in Table 2 (objective) and Figure 3 (ABX preference test).
Objective metrics (Table 2):
- MPO achieves CER 3.9, SPK-SIM 0.577, prosody (log F0 RMSE) 0.279 vs. SFT baseline CER 4.72, SPK-SIM 0.548, prosody 0.337.
- A model trained on combined rankings (naive merge of three per-metric ranks) reaches CER 4.3, SPK-SIM 0.564, prosody 0.218—better on prosody but worse on intelligibility and speaker similarity than MPO.
- Single-metric DPO variants confirm that optimizing for one dimension improves that metric but degrades others.
Regularization ablation (Table 1): Without the CE constraint, CER worsens from 4.72 (SFT) to 6.41 (10k steps) and 14.52 (15k steps). MPO with CE constraint reaches CER 4.24 at ~20k steps with no degradation.
Subjective ABX test (Figure 3): MPO is preferred over the SFT baseline in 52.3% of comparisons (22.4% ties); MPO beats combined-ranking in 40.2% vs. 30.7% (29.1% ties); MPO is competitive with ground truth in pairwise preference.
Novelty Assessment
The paper makes two targeted contributions within the already-active TTS preference alignment subfield. The preference set idea—pooling candidates from multiple scorers and drawing random cross-dimensional pairs—is a simple structural change to DPO data preparation that is not complex but demonstrably more effective than naive rank fusion. The CE regularization to prevent DPO collapse in TTS is a practical fix borrowed from NLP alignment literature (analogous to SimPO and KAHNEMAN-TVERSKY alignment work) and applied here to the TTS context. Neither contribution is architecturally novel: the base LM is a standard LLaMA-style AR model. The contribution is primarily a training-recipe improvement: better data curation for preference pairs plus a regularizer that maintains generation capability. The results are shown only on internal Mandarin data without comparison to published systems (e.g. Seed-TTS, CosyVoice), which limits external benchmarking.
Field Significance
Moderate — MPO provides a practical recipe for multidimensional preference optimization in LM-based TTS, demonstrating that pooling candidates across quality dimensions is more effective than naive rank fusion. The CE regularization finding—that vanilla DPO degrades TTS capability without a cross-entropy anchor—is a useful cautionary result for practitioners applying alignment methods in this setting.
Claims
- Optimizing LM-based TTS with preference data across multiple quality dimensions simultaneously is more effective than per-metric or naively combined ranking approaches. (§4.5, Table 2)
- Vanilla DPO applied to LM-based TTS without regularization causes progressive degradation of generation capability, with CER rising sharply after 10k training steps. (§4.4, Table 1)
- Adding a cross-entropy loss on preferred sequences as a regularizer during DPO training prevents model collapse without sacrificing alignment gains. (§3.2, §4.4)
- Multi-dimensional preference alignment in LM-based TTS produces output quality comparable to ground truth in subjective pairwise preference tests. (§4.5, Figure 3)
Limitations and Open Questions
- Evaluation is entirely on an internal Mandarin TTS corpus; there is no open benchmark comparison, making it difficult to assess absolute quality relative to published systems.
- The model size is not disclosed and the base LM architecture details are partially withheld (internal codec), limiting reproducibility.
- The 100-hour preference optimization corpus is Mandarin only; whether the approach generalizes to multilingual or cross-lingual zero-shot scenarios is untested.
- The preference set construction requires running three separate quality scorers (ASR, speaker verification, F0) over 10 generated candidates per input—a non-trivial computational overhead not quantified in the paper.
- The optimal value of λ (set to 10) and the preference margin thresholds (0.1 for SPK-SIM and prosody) are not ablated; sensitivity to these hyperparameters is unknown.
- Comparison against SpeechAlign, UNO, and RIO is discussed only qualitatively; a controlled quantitative comparison against these direct baselines would strengthen the claims.
Wiki Connections
- rlhf-speech — this paper advances preference alignment for LM-based TTS with multidimensional preference set construction and CE-regularized DPO
- autoregressive-codec-tts — the system is built on a LLaMA-style autoregressive speech LM with a single-codebook neural codec
- evaluation-metrics — CER, SPK-SIM, and F0-RMSE are used as the three scoring axes for preference data construction, not just for reporting
- subjective-evaluation — ABX preference tests are used to validate multidimensional alignment against the SFT baseline and ground truth