Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback

arXiv · 2024 · Preprint

Chen Chen et al. (Nanyang Technological University) · → Paper · Demo: ? · Code: ?

UNO (uncertainty-aware optimization) is the first framework to integrate human subjective feedback directly into the TTS training loop without requiring pairwise preference data or a separate reward model, substantially reducing WER and improving naturalness for zero-shot codec language model TTS.

Problem

State-of-the-art autoregressive codec TTS models are trained with supervised cross-entropy objectives on token sequences, but evaluated using MOS, WER, and speaker similarity metrics that reflect human perceptual preferences. This mismatch means the training objective cannot directly minimize the failure modes that listeners penalise most, including disfluencies, omissions, and unnatural prosody. Standard RLHF remedies (DPO, PPO with a reward model) require pairwise preference data based on the same input, but existing TTS models produce insufficiently diverse outputs from a fixed transcript-prompt pair, making pairwise labelling impractical at scale. Prior work (SpeechAlign) partially addressed this by treating ground truth as the preferred sample, but this requires paired ground truth at optimization time and suppresses high-quality model outputs as implicitly negative.

Method

UNO adopts a three-stage sampling-annotating-learning pipeline tailored to the properties of zero-shot codec TTS. In the sampling stage, a fixed target transcript is paired with a batch of distinct speech prompts drawn from an unseen speaker pool, yielding a diverse set of synthesised outputs; varying the prompt (rather than stochastic decoding) provides much richer quality variation across samples. Human annotators, or surrogate annotation models trained on the SOMOS dataset, then assign each sample a binary desirable/undesirable label together with an uncertainty score. Two neural uncertainty estimators are evaluated: EDL (evidential deep learning, placing a Gaussian prior over annotation distributions) and I-CNF (a conditional normalizing flow that learns the full annotation distribution non-parametrically). Both use a frozen WavLM backbone to extract speech embeddings.

The learning step adapts the KTO objective from language model alignment to TTS. Rather than requiring paired win/lose samples, UNO maximises a utility function over positive and negative sample pools. The core insight is that inverse uncertainty (1/u) replaces the fixed beta hyperparameter in the standard KL-constrained RLHF objective: samples with low annotator disagreement receive larger gradient updates, while high-uncertainty samples contribute conservatively. A reference partition function Z_ref (the expected KL divergence between the current policy and the reference model over the training batch) stabilises training without a separate baseline network, analogous to the REINFORCE baseline.

The framework is applied as a fine-tuning pass on a pre-trained VoiceCraft model (330M or 830M parameters), which uses SpeechTokenizer (EnCodec-based, 4 RVQ codebooks, vocabulary of 2048) for discrete acoustic tokenisation. Only 400 self-generated samples (balanced 200 positive / 200 negative) are needed, requiring approximately 10 minutes of fine-tuning on a single A100 GPU. The method extends naturally to emotion TTS by substituting the selection criterion in the sampling stage: using emotional ESD dataset prompts and sorting by valence or arousal score instead of MOS drives the model to generate speech with target affective properties.

Key Results

On LibriSpeech test-clean, UNO-ICNF reduces VoiceCraft’s WER from 8.4% to 2.6% and improves SPK-SIM from 0.84 to 0.91 (Table 1). Human naturalness MOS rises from 3.38 to 4.06 (Table 2), approaching ground truth at 4.55; A/B testing shows listeners strongly prefer UNO outputs over the baseline. The gains come without requiring any ground truth speech at optimization time, unlike SpeechAlign-DPO and SpeechAlign-ODPO which require paired references and still score lower (WER 7.2%, MOS ~3.70 by automatic estimators). PPO with MOSNet as reward model also trails UNO substantially (WER 7.7%). On the 830M model, UNO further improves WER from 2.7% to 2.2% and MOS from 4.30 to 4.41 (Figure 4). An ablation without uncertainty weighting (UNO-null) achieves similar mean MOS but higher output variance, confirming that uncertainty modelling specifically improves generation consistency rather than average quality. Emotional TTS experiments show UNO increases valence from 0.55 to 0.67 and arousal from 0.62 to 0.71 (Table 4), without any explicit emotion labelling at training time.

Novelty Assessment

The core contribution is methodological: adapting KTO-style non-pairwise RLHF to TTS with an uncertainty-aware utility function. The individual components (KTO, evidential deep learning, normalizing flows for annotation simulation) are drawn from NLP and uncertainty estimation literature, but their combination into a TTS-specific pipeline is genuinely new. The observation that varying the speech prompt is a more effective diversity strategy than stochastic decoding for codec LM TTS is practically useful and not widely noted. The paper’s argument that DPO cannot be directly applied to zero-shot TTS due to insufficient output diversity is well-supported and informative for the alignment research agenda. The main limitation is that the framework is demonstrated only on VoiceCraft (an autoregressive codec model); applicability to diffusion or flow-matching TTS is discussed but not demonstrated experimentally. The evaluation uses a single model family on a single benchmark, and the comparison against SpeechAlign is partially unfair by the authors’ own admission (SpeechAlign uses ground truth, which constitutes additional supervision).

Field Significance

Tip

High — UNO establishes that RLHF-style training without pairwise preference data or a reward model is viable for zero-shot TTS, and that addressing the MOS-training objective mismatch yields substantial WER reductions (8.4% to 2.6%) alongside naturalness improvements. The framework provides an alignment recipe that generalises across model scales and extends to emotion control, filling a gap that existing DPO/PPO approaches cannot readily address for TTS due to its limited output diversity from fixed inputs.

Claims

• Standard supervised training objectives for TTS produce a systematic mismatch with human perceptual evaluation metrics such as MOS and WER, and correcting this mismatch through preference-aware fine-tuning yields large performance gains. (§1, §4.2, Table 1)
• Existing RLHF methods requiring pairwise preference data from the same input (DPO) are difficult to apply directly to autoregressive codec TTS because these models lack sufficient output diversity to form meaningful preference pairs from a fixed transcript-prompt combination. (§4.2, Appendix B)
• Uncertainty in human speech quality annotations is not noise to be discarded but an informative signal that, when incorporated into the optimization objective, improves the consistency of generated speech across listeners. (§4.2, §6.3, Table 3)
• RLHF-style alignment for TTS can be achieved with a small number of self-generated samples (hundreds) without access to ground truth speech, making it practical for post-training fine-tuning at low computational cost. (§4.1, §5, Appendix D)
• Alignment objectives designed for naturalness MOS transfer to other perceptual dimensions such as emotion by substituting the selection criterion, demonstrating that preference-based fine-tuning generalises beyond a single quality axis. (§6.4, Table 4)

Limitations and Open Questions

Warning

The comparison with SpeechAlign is acknowledged by the authors to be partially unfair: SpeechAlign-DPO requires ground truth speech as positive samples during optimization, which is additional supervision not available to UNO. Presenting both as baselines without fully separating this distinction may understate SpeechAlign’s performance under matched conditions.

UNO is evaluated exclusively on VoiceCraft, an autoregressive codec language model. Whether the training framework transfers to diffusion-based TTS (e.g., NaturalSpeech 3) or flow-matching models is discussed theoretically but untested. The paper notes that diffusion-based DPO works have been proposed in parallel for image generation, suggesting it is feasible, but speech-specific evidence is absent. Human evaluation uses only 10 listeners for 120 samples, which is relatively small and may not capture the full distribution of listener preferences across accents, domains, or speaking styles. The uncertainty estimators (EDL, I-CNF) are trained on SOMOS, a dataset of 200 TTS systems with over 17 annotations per sample, and their behaviour on out-of-distribution speech styles is unknown. The practical annotation cost of 400 samples per optimization run is modest but still requires human or surrogate labelling infrastructure.

Wiki Connections

• rlhf-speech — UNO introduces uncertainty-aware KTO-style optimization as an alternative to DPO and PPO for TTS alignment, directly advancing the RLHF-for-speech paradigm.
• autoregressive-codec-tts — the method is demonstrated on VoiceCraft, an autoregressive codec language model, and addresses a failure mode specific to autoregressive generation (insufficient output diversity for pairwise preference collection).
• zero-shot-tts — UNO targets zero-shot TTS quality specifically, using diverse speech prompts as the diversity mechanism and optimizing for speaker similarity alongside naturalness.
• subjective-evaluation — the paper directly addresses the mismatch between supervised TTS objectives and subjective human evaluation, and demonstrates that closed-loop integration of human MOS signals substantially improves both subjective and objective metrics.
• evaluation-metrics — the uncertainty-aware annotation simulator models variability across human annotators on the SOMOS dataset, contributing to the understanding of how evaluation uncertainty should be handled in TTS training.
• neural-codec — the method relies on EnCodec-based tokenization (SpeechTokenizer) and the implicit reward function is defined over codec token log-likelihood ratios, making the codec central to the alignment mechanism.
• 2403.16973 (VoiceCraft) — UNO uses VoiceCraft as its base TTS model and fine-tunes it with the proposed alignment framework, demonstrating substantial gains over VoiceCraft’s supervised baseline.
• 2301.02111 (VALL-E) — UNO’s framing of zero-shot TTS as transcript-conditioned speech continuation with a speech prompt prefix builds directly on the VALL-E formulation.
• 2403.03100 (NaturalSpeech 3) — cited as an example of diffusion-based zero-shot TTS whose alignment with UNO is theoretically feasible but experimentally untested.
• 2308.16692 (SpeechTokenizer) — used as the codec backbone for acoustic tokenisation in the VoiceCraft models that UNO fine-tunes.
• 2404.03204 (RALL-E) — cited as a related approach for improving robustness of autoregressive zero-shot TTS, complementary to the alignment approach taken by UNO.
• 2402.08093 (BaseTTS) — cited as an example of large-scale supervised TTS that still does not incorporate human feedback into its training loop, motivating UNO’s contribution.