Abstract

RLHF for speech uses preferences or reward signals to improve a pretrained TTS, spoken-language, or speech-to-speech model after supervised training. Across 49 reviewed papers, post-training often improves the quality named by the reward. It can also damage prosody, diversity, emotion, or another quality that the reward does not measure.

Current State

  • Preference and reward post-training often improve their target dimensions. DPO, KTO, GRPO, policy-gradient methods, and differentiable reward optimization report gains in intelligibility, speaker similarity, naturalness, or semantic coherence. The pattern appears in both discrete and continuous speech generators (2406.00654, 2025.acl-long.598, MPO).
  • Reward hacking is the central cross-method risk. Optimizing WER, likelihood, or another proxy can produce monotone prosody, lower emotional appropriateness, reduced diversity, or evasive dialogue behavior even while the reward improves (2509.18531, Vevo2, Step-Audio).
  • Preference data construction matters as much as the objective. Pairs need meaningful diversity and a reliable ordering; fine-tuning that narrows the generation distribution can leave DPO with too little useful contrast (2025.icnlsp-1.34, 2509.14946).
  • Multi-objective rewards are safer than single-metric rewards, but balancing remains under-specified. Combining intelligibility, identity, emotion, and prosody avoids some one-dimensional failures, yet most papers do not establish principled weights.
  • Speech alignment no longer requires a discrete autoregressive decoder. Preference objectives and policy gradients have been adapted to flow matching, diffusion, autoregressive-diffusion, and CTC decoders, although most techniques remain one-backbone demonstrations (F5R-TTS, 2509.18928, 2509.25416).
  • Alignment cannot create capability absent from the base model. Iterative optimization shows diminishing returns and rarely reaches oracle best-of-N or strong cascaded toplines.
  • Q4 localizes where alignment can and cannot act. Token-level preference signals such as Data-efficient Targeted Token-level Preference Optimization concentrate sparse feedback, while LM-only RLHF cannot repair quality determined by a frozen flow decoder or vocoder. No Verifiable Reward for Prosody shows that a small human-preference stage can recover proxy-induced prosody loss, but repeated rounds can saturate or reverse.

Method Landscape

FamilyPractical ideaMain limitation
DPO/KTO on discrete speech tokensLearn from preferred and rejected generations without online rolloutsHighly sensitive to pair construction and reference regularization
Policy-gradient RLOptimize explicit audio or task rewards with GRPO/PPO/REINFORCEExpensive sampling, reward hacking, and unstable training
Differentiable reward optimizationBackpropagate through discrete tokens from a frozen criticEfficient, but so far concentrated in the CosyVoice lineage
Continuous-generative preference optimizationReformulate DPO or RL for flow/diffusion trajectoriesEach method is validated on a different backbone with little replication
Reward-model infrastructureTrain reusable audio-quality or dialogue reward evaluatorsFew reward models have been validated inside a complete alignment loop

Key Trade-offs

  • Target improvement versus unmeasured quality. A better reward score can conceal worse prosody, diversity, or emotional suitability.
  • Offline stability versus online flexibility. DPO avoids rollouts; policy-gradient methods can optimize arbitrary rewards but are more expensive and unstable.
  • External reward quality versus reward-model cost. Differentiable critics can be efficient but may encode narrow ASR or quality biases.
  • Stronger adaptation versus capability preservation. Long or weakly regularized runs can collapse intelligibility, identity, or held-out quality.
  • Component-level alignment versus final audio quality. Optimizing an LM cannot repair attributes controlled by a frozen downstream flow model or vocoder.
  • Local feedback versus global coherence. Segment-level rewards improve data efficiency but must be balanced against utterance-level naturalness, identity, and dialogue behavior.

Open Questions

  • Is there a general defense against reward hacking, or only task-specific combinations of held-out metrics, constraints, and human preference stages?
  • Do continuous-generation preference methods transfer across flow and diffusion backbones?
  • Can intrinsic rewards from the speech model itself replicate beyond one GFlowNet-based result?
  • How should RLHF for conversational behavior differ from acoustic-quality alignment?
  • Can reward models trained as standalone evaluators remain reliable when used to optimize a generator against them?

Go Deeper

Read RLHF for Speech: In Depth for the evidence on reward hacking, preference data, multi-objective optimization, continuous generators, training stability, and the limits imposed by the base model and downstream pipeline.

Scope

This assessment covers 49 integrated papers through Q4 2025. The area is fast-moving but young. Several strong results come from one backbone or proprietary system. RLHF for spoken dialogue is also much less developed than acoustic TTS alignment. The full record is in wiki/_claims/rlhf-speech.yaml.


This page is generated from wiki/_claims/rlhf-speech.yaml (digest date: 2026-09-26). Complete structured claims and paper-level provenance remain in the source YAML.