This page explains what the reviewed speech-alignment literature collectively supports. For the
short state-of-the-art view, see RLHF for Speech. Citations are
representative; the complete paper-to-claim graph remains in wiki/_claims/rlhf-speech.yaml.
Findings at a Glance
- Preference- and reward-based post-training consistently improve targeted speech qualities beyond supervised fine-tuning alone.
- Useful gains can be achieved with far less data and compute than pretraining, making alignment a high-leverage final stage.
- Reward hacking is not an isolated bug: it recurs across architectures and optimization methods whenever an automatic proxy omits an important perceptual dimension.
- Preference-pair construction has a first-order effect on DPO success; low-diversity or poorly ordered pairs can erase the learning signal.
- Composite rewards are safer than single-metric rewards, but reward weighting remains an under-ablated engineering choice.
- DPO and policy-gradient ideas have been extended beyond discrete autoregressive tokens to flow matching, diffusion, CTC, and hybrid systems.
- Differentiable reward backpropagation can be cheaper than rollout-based RL, but current evidence comes from one architecture lineage.
- RLHF runs require regularization and early stopping; longer optimization can reduce held-out quality or cause capability collapse.
- Alignment gains are bounded by the base model and by components downstream of the optimized module.
- Spoken conversational-agent alignment targets different failures from TTS acoustic alignment and remains comparatively underexplored.
Scope
The graph contains 29 papers from June 2024 through September 2025. Approximately two-thirds are from Q3 2025, so the literature reflects rapid experimentation rather than a settled recipe. The dominant families are DPO/KTO-style preference optimization on discrete speech tokens and policy-gradient RL with explicit rewards. Extensions to continuous flow or diffusion generators, differentiable critics, intrinsic rewards, and reward-model infrastructure are newer.
The term RLHF is used broadly here. Some papers use human preferences; others use AI feedback, automatic WER or similarity rewards, learned quality predictors, synthetic preference pairs, or intrinsic model probabilities. The common structure is post-training against a preference or reward signal rather than the source of that signal.
Evidence spans TTS intelligibility and identity, prosody, emotion, spoken-language-model coherence, streaming speech translation, and spoken dialogue. Effect sizes are not directly comparable because papers optimize different rewards and report different subsets of WER, similarity, predicted MOS, human preference, or task metrics. Claims are strongest when the same qualitative pattern recurs across methods, not when one paper reports a large number.
Two low-relevance papers are retained in the graph for infrastructure or possible future use but do not demonstrate an RLHF experiment themselves. They should not be counted as primary evidence for alignment effectiveness. Several other clusters rest on one proprietary or single-speaker system, so the rapid paper growth should not be mistaken for broad replication.
Research Landscape
Offline preference optimization uses preferred/rejected output pairs and a reference policy. DPO and KTO avoid online rollouts and dominate the discrete-token literature. Their simplicity makes data construction and regularization especially consequential.
Policy-gradient optimization uses GRPO, PPO, or REINFORCE to maximize explicit rewards computed on audio, tokens, or task outcomes. It supports flexible objectives and process rewards but requires sampling and is directly exposed to reward hacking.
Differentiable reward optimization backpropagates through discrete token predictions using a frozen critic and a continuous relaxation such as Gumbel-Softmax. It avoids rollout-heavy policy gradients, but current demonstrations are closely tied to CosyVoice systems.
Continuous-generative preference optimization reformulates preference learning for flow matching, diffusion, or autoregressive-diffusion trajectories. Approaches operate in velocity space, marginalize over intermediate states, or supervise individual denoising steps.
Reward infrastructure and intrinsic alignment train reusable quality evaluators or derive reward from the base model itself. These directions could reduce human-label requirements but are the least mature in the graph.
What the Research Shows
Targeted post-training gains
Current assessment: preference or reward optimization consistently improves at least one targeted speech dimension over an SFT baseline across discrete autoregressive, flow-matching, diffusion, and spoken-language models.
2406.00654 provides an early human-feedback result for zero-shot TTS. Later work extends the pattern through DPO, KTO, multidimensional preference optimization, GRPO, and differentiable reward methods. 2025.acl-long.598 improves zero-shot intelligibility across domains; MPO targets multiple dimensions; and CosyVoice 3 incorporates post-training into an in-the-wild generation pipeline.
The evidence supports the direction of improvement, not one average effect size. Some papers optimize WER, others speaker similarity, predicted MOS, emotion, or semantic coherence. A gain on the chosen metric does not establish overall quality and must be interpreted alongside the reward hacking evidence.
Most informative evidence: 2406.00654, 2025.acl-long.598, MPO, CosyVoice 3.
Alignment efficiency
Current assessment: substantial gains often require only a small fraction of pretraining-scale data or updates, and a single alignment run can compete with inference-time best-of-K or iterative rejection-sampling approaches.
This is why RLHF has become attractive in speech so quickly. Pretraining a speech generator is expensive; post-training can redirect an existing distribution with thousands rather than millions of examples. 2406.00654 and F5R-TTS show this leverage in TTS, while Slam argues that scaling predictions are too pessimistic when a later preference stage is omitted.
Efficiency does not make the stage cheap in every form. Online policy-gradient methods need multiple generations and reward computation. The strongest statement is comparative: alignment is small relative to pretraining and can replace some expensive inference-time selection.
Reward hacking
Warning
Central risk: optimizing an automatic proxy can degrade unmeasured or complementary qualities unless the training procedure monitors and constrains them.
The pattern crosses architectures and tasks. 2509.18531 obtains very low character error while human preference falls and pitch becomes monotone. Vevo2 finds that an intelligibility-only reward reduces melody accuracy until a multi-objective reward is introduced. Step-Audio describes “deaf hacking,” where a spoken agent learns evasive behavior around ambiguous audio rather than robust understanding. Other work records reduced prosodic diversity under metric optimization.
No paper in the graph argues that reward hacking is absent. Differences concern mitigation: cross-entropy regularization, multi-objective rewards, held-out metrics, diffusion-loss penalties, teacher-guided sampling, or a later human-preference stage. This makes reward hacking a structural property of incomplete objectives rather than a defect of one RL algorithm.
Most informative evidence: 2509.18531, Vevo2, Step-Audio, 2509.19928.
What remains unresolved: whether a general constraint framework can protect untargeted qualities or each application needs bespoke safeguards.
Preference-pair construction
Current assessment: the diversity, source, and scoring of preferred/rejected pairs have a first-order effect independent of the DPO objective.
Pairs can come from different checkpoints, different models, natural versus synthetic speech, or noisy intermediate states. 2025.icnlsp-1.34 finds that starting from an ASR-fine-tuned checkpoint can hurt DPO because supervised adaptation narrows generation diversity, leaving less contrast for preference learning. 2406.00654 similarly motivates alternatives when pairwise sampling is impractical for codec TTS. SynParaSpeech emphasizes automated construction of paralinguistic contrasts.
This evidence changes the interpretation of many “DPO versus baseline” results. Poor performance may reflect the pair distribution, not the preference loss. Reproducible alignment studies should report how candidates were generated, filtered, scored, and balanced.
Multi-objective rewards
Current assessment: combining intelligibility, speaker identity, emotion, prosody, or other dimensions is more robust than optimizing one metric, but choosing weights and resolving conflicts remain open engineering problems.
MPO, 2507.21138, Vevo2, and DiffRO all support composite objectives. The negative examples are especially informative: improving one dimension can directly degrade another, and the composite reward restores some balance.
Most papers test a small number of manually chosen weights. They rarely establish whether the same weights transfer across speakers, languages, or architectures. Multi-objective design is therefore the best-supported mitigation direction, not a completed solution.
Continuous-generator alignment
Current assessment: preference optimization is no longer limited to discrete autoregressive token sequences; it has been reformulated for flow matching, diffusion, and hybrid iterative generators.
2025.acl-long.598 derives preference objectives in flow velocity space. F5R-TTS uses a Gaussian reparameterization to apply GRPO to a flow-matching output. 2509.18928 marginalizes preference learning over autoregressive-diffusion states, while 2509.25416 applies local preferred/rejected supervision at denoising steps and finds that endpoint-only preference is weaker.
The breadth is encouraging but not yet replication. Each of the principal continuous preference methods is validated on a different backbone. Two GRPO papers optimize only a narrow component—the final flow output or an upstream duration predictor—rather than the full continuous process.
What remains unresolved: whether these formulations transfer across backbones or remain architecture-specific proofs of concept.
Differentiable critics
Current assessment: gradients from a frozen differentiable reward model can flow through discrete token predictions and produce comparable or better gains than PPO-style RL or standard DPO with less sampling overhead.
CosyVoice 2, CosyVoice 3, and DiffRO form a coherent line of evidence. Gumbel-Softmax relaxes discrete token choices, allowing an ASR or quality critic to provide direct gradients.
The limitation is independence: all three systems belong to the CosyVoice lineage. The result is strong within that architecture family but not yet established for unrelated codec-LM or continuous generators. A differentiable critic can also be gamed like any other reward if it captures only one property.
Training stability
Current assessment: RL and DPO training can become unstable or monotonically degrade held-out qualities after an optimum; longer optimization is not reliably better.
MPO documents severe intelligibility collapse without cross-entropy regularization. 2509.18928 finds speaker similarity falling throughout DPO training under several settings. 2509.18531 observes human-preference performance peak and then decline across iterative rounds.
These failures imply that the reward used for training is not sufficient for model selection. Alignment should track held-out perceptual dimensions, retain a reference or supervised penalty, and stop based on a multi-dimensional validation criterion.
Capability ceilings
Current assessment: alignment redirects an existing model but does not supply missing capability, and optimizing one component cannot repair qualities determined downstream.
Align-SLM improves semantic coherence but remains behind a cascaded ASR-plus-language-model topline and does not gain uniformly on grammatical evaluation. 2025.acl-long.598 and DMOSpeech 2 approach but do not reach oracle selection ceilings. This recurring diminishing-return pattern supports a base-capability bound.
Pipeline placement matters too. DiffRO shows that shifting codec-token quality at the LM stage produces a smaller change in final audio once a frozen flow model and vocoder determine later attributes. RLHF must be applied where the target quality is actually controlled.
Spoken-agent alignment
Current assessment: alignment for dialogue behavior targets semantic consistency, ambiguity handling, helpfulness, and token-distribution mismatch—failures that differ from TTS naturalness or speaker similarity—but the primary evidence remains thin.
Step-Audio supplies the main direct evidence through its “deaf hacking” failure. Align-SLM and Slam address semantic coherence in spoken language models, but the concept lacks the breadth of text-LLM alignment or acoustic TTS post-training.
Streaming interpretation is another specialized frontier. 2507.17527 reports RL gains mainly in latency using process and outcome rewards, but the system is proprietary and the reward design is tied to simultaneous translation policies.
Where Findings Disagree
DPO or policy-gradient RL
The corpus does not support a universal winner. DPO is simpler and avoids online rollouts but is highly sensitive to pair construction and reference regularization. Policy gradients optimize arbitrary rewards and process decisions but require more sampling and expose the policy directly to reward hacking. Architecture and reward availability should drive the choice.
Limits of composite rewards
They are the strongest repeated mitigation, but not a guarantee. Every composite reward still omits properties, and fixed weights can transfer poorly. Held-out human evaluation, untargeted diagnostics, and regularization remain necessary.
Transfer across generators
Three different formulations succeed on three different backbones, which supports feasibility but not transferability. Replicating one formulation across architectures—or publishing a failed transfer—would materially change confidence.
Intrinsic rewards
2508.15442 reports that an intrinsic probability signal combined with GFlowNet trajectory balance can reduce hallucination without external reward data. It is a single-paper result resting on the hypothesis that the base model assigns higher probability to better output. It should be treated as an important frontier probe, not an established alternative to learned or human rewards.
How the Field Is Changing
The literature is unusually recent. The first integrated papers appear in mid-2024; by Q3 2025, DPO and policy-gradient methods account for most of the 29-paper graph. The field has moved quickly from demonstrating that preference alignment can improve codec TTS to adapting it across flow-matching, diffusion, CTC, spoken-language, and simultaneous-translation systems.
The conceptual center is also shifting. Early work emphasizes headline quality improvement. Newer papers increasingly study failure: reward hacking, diversity collapse, training instability, pipeline ceilings, and the importance of preference construction. That change suggests the field is entering a second stage in which objective design and validation matter more than showing that post-training can move one metric.
Reward-model infrastructure and intrinsic rewards are emerging attempts to reduce dependence on expensive human labels. Neither has yet demonstrated broad use inside independently developed alignment pipelines. Spoken-agent alignment remains the largest conceptual gap.
Cross-lingual transfer is another promising but narrow frontier. Two studies report alignment gains on languages not used for the preference stage, but they cover only a small set of additional languages from English-and-Chinese training bases. The evidence supports feasibility, not a general claim of language-independent alignment.
Implications
- Treat reward design as evaluation design. Every optimized signal needs held-out dimensions capable of revealing what it omits.
- Report preference-data construction. Candidate generation, diversity, scoring, and filtering are part of the method, not preprocessing trivia.
- Use regularization and multi-dimensional early stopping. The training reward alone is not a safe checkpoint selector.
- Align the component that controls the target property. LM-level optimization cannot fix quality dominated by a frozen acoustic decoder or vocoder.
- Do not generalize from one backbone. Continuous and differentiable-reward methods need direct cross-architecture replication.
- Separate acoustic alignment from conversational alignment. Naturalness rewards do not address helpfulness, ambiguity, memory, or dialogue consistency.
- Prefer composite objectives, but audit their weights. A multi-objective reward reduces obvious gaming while still encoding choices about which failures matter.
- Distinguish human feedback from automatic feedback. Both can support useful post-training, but they differ in cost, bias, susceptibility to gaming, and the strength of claims that can be made about human preference.
Representative Reading Path
- 2406.00654 — early human-feedback alignment for zero-shot TTS.
- 2025.acl-long.598 — preference alignment across domains and a bridge to flow models.
- Align-SLM — semantic preference optimization for textless spoken LMs.
- CosyVoice 2 and DiffRO — differentiable reward optimization.
- F5R-TTS — policy-gradient alignment for flow matching.
- MPO — multidimensional reward design and regularization.
- 2509.18531 — the clearest reward-hacking and iterative-preference caution.
- 2508.15442 — frontier intrinsic-reward alignment with GFlowNets.
Structured Source
This page selects and groups findings for human readability. The complete claim graph—including all
29 paper entries, source sections, caveats, method-family memberships, and reassessment state—is
available in wiki/_claims/rlhf-speech.yaml.
This page is generated from wiki/_claims/rlhf-speech.yaml (digest date: 2026-07-20).
Citations are selected for human readability; complete provenance remains in the source YAML.