arXiv · 2026 · Preprint
Siddhant Arora et al. (Carnegie Mellon University / Sony Group Corporation) · → Paper · Demo: ? · Code: ✓
Presents the first multi-reward RLAIF/DPO framework for speech-in/speech-out spoken dialogue systems, jointly optimizing semantic coherence, audio naturalness, intelligibility, and emotion consistency, and extends utterance-level preference learning to blockwise duplex models by aggregating per-block log-probabilities into a single DPO objective.
Problem
Reinforcement learning from human or AI feedback (RLHF/RLAIF) for speech-in/speech-out spoken dialogue systems (SDS) is underexplored relative to text-based dialogue, and the handful of prior SDS-focused RLHF efforts each optimize only a single, narrow reward dimension, typically semantic quality or turn-taking timing, even though conversational quality is inherently multi-dimensional (semantic coherence, audio naturalness, speaker/style consistency, emotional alignment, responsiveness). Existing pipelines also assume full-utterance feedback, which is fundamentally incompatible with duplex SDS that generate responses incrementally in fixed-size blocks and must commit to partial utterances before sentence boundaries are known. And while audio-based preference learning has gained traction for standalone TTS systems, it remains largely unexplored as a reward source within a unified spoken-dialogue-level learning framework.
Method
The paper formalizes turn-by-turn SDS (predicting each turn’s response conditioned on prior turns and dialogue history) and duplex SDS (blockwise, time-multiplexed generation as in the SCoT architecture) as two backbone paradigms, then applies Direct Preference Optimization (DPO) as RLAIF post-training on top of each. For turn-by-turn systems, candidate responses are sampled per turn, scored by the reward functions below, and formed into preferred/dispreferred pairs used in a standard DPO loss comparing log-probability margins between the trained policy and a frozen reference (the pre-RLAIF supervised checkpoint). For duplex systems, since preference labels exist only at the utterance level while generation happens blockwise, the paper decomposes each candidate response into fixed-size blocks and computes the utterance’s log-probability as a sum over blockwise conditional log-probabilities (each conditioned on the accumulated user-and-system block history), then substitutes this aggregated quantity into the same DPO objective. This lets utterance-level preference supervision train blockwise duplex generation without needing to define reward signals for individual, semantically incomplete partial-utterance blocks, which the authors note is non-trivial for most conversational and perceptual metrics.
Four independent preference datasets are constructed, one per quality dimension, then jointly sampled during DPO training (dataset-level multi-reward combination, rather than a single preference pair scored on multiple criteria simultaneously). Semantic-quality preferences adapt the Align-SLM LLM-judge framework: Qwen2.5-72B-Instruct scores candidate text responses for coherence, relevance, and dialogue grounding, filtered against an AutoBLEU repetition threshold to exclude degenerate high-overlap candidates, yielding positive/negative pairs; because this signal only judges the intermediate text response in the CoT factorization, the DPO loss for this reward is applied using only the text-policy likelihood term. Audio-quality preferences use UTMOS as an automatic naturalness proxy, selecting the highest- and lowest-scoring synthesized candidates per turn while holding the text response fixed across the pair (isolating acoustic quality from semantic content). Intelligibility preferences use WER between an ASR transcript of synthesized candidates and the model’s own predicted text response, with a margin threshold to avoid ambiguous near-tie comparisons. Emotion-consistency preferences use Emo2Vec similarity between synthesized speech and human reference speech, again holding text content fixed across the preferred/dispreferred pair and requiring a minimum similarity gap. The resulting released dataset totals 165.7K preference pairs (51.1K semantic, 32.0K audio quality, 61.0K intelligibility, 21.6K emotion), built from candidates generated over the Switchboard spontaneous-telephone-conversation corpus.
Key Results
On the turn-by-turn Multi-turn CoT E2E backbone, single-reward RLAIF trained specifically on semantic preferences raises LLM-judge score from 6.18 to 6.33 (statistically significant, p<0.01) and reduces the proportion of low-quality (<5) judged responses from 10.2% to 7.1%, a 28.5% relative reduction, alongside a large drop in repetition (AutoBLEU 68.3→56.5) and lower perplexity (21.2→19.9). Audio-quality single-reward RLAIF raises UTMOS from 2.16 to 3.06; intelligibility single-reward RLAIF reduces WER from 6.1% to 3.3%; each reward selectively improves its own targeted dimension, validating that the constructed preference data isolates the intended quality axis. Jointly combining semantic, audio-quality, and intelligibility preferences (Joint-Reward-v1) achieves the best intelligibility of any RLAIF variant (WER 1.0%) while maintaining strong UTMOS (2.85) and LLM-judge performance, showing joint training does not sacrifice individual dimensions for breadth. Adding emotion-consistency preferences on top (Joint-Reward-v2) further improves LLM-judge score (6.33) and reduces low-quality responses (7.5%), but yields a worse emotion rank (3.00) than single-reward emotion-only training (1.98) or even the non-RLAIF baseline (2.29); the authors attribute this to strong semantic LLM-judge rewards favoring safer, more generic responses that reduce emotional expressiveness, an explicit trade-off between reward objectives. Applying the blockwise log-probability aggregation to a duplex SCoT-Response model yields consistent gains over its non-RLAIF baseline (ROUGE-L 19.8→23.1, perplexity 42.3→25.0, LLM-judge 5.95→6.00), demonstrating the framework generalizes beyond turn-by-turn architectures to blockwise, incrementally-decoded duplex systems.
Novelty Assessment
This is the first systematic multi-reward RLAIF study for speech-in/speech-out SDS spanning semantic, audio-quality, intelligibility, and emotion-consistency rewards jointly, where prior SDS-focused RLHF work (Align-SLM, ORISE, and the deployed-duplex-SDS semantic-preference work the paper cites) each explored only one reward dimension. The blockwise log-probability aggregation for adapting utterance-level DPO to duplex, incrementally-decoded generation is a genuine methodological contribution addressing a real architectural mismatch (partial-utterance reward definition is non-trivial for most perceptual metrics), not merely an engineering convenience. The released 165.7K-pair multi-reward preference dataset is a concrete, reusable artifact supporting reproducible SDS alignment research, distinct from prior single-objective preference datasets in this space. The individual reward-construction techniques (LLM-judge semantic scoring, UTMOS-based audio-quality preference, WER-based intelligibility preference, embedding-similarity emotion preference) are each adapted from established methods rather than newly invented, but their combination and joint application within one unified dialogue-level DPO pipeline for SDS specifically is new.
Field Significance
high — this paper is the first systematic study to jointly optimize semantic, acoustic, intelligibility, and emotional dimensions of spoken dialogue quality within a single RLAIF pipeline, and directly surfaces a genuine trade-off (semantic-reward optimization reducing emotional expressiveness) that single-objective RLHF/RLAIF work for SDS could not have revealed, while also solving a real architectural mismatch between utterance-level preference supervision and blockwise duplex generation via log-probability aggregation.
Claims
- supports: Applying DPO-based preference optimization with a single, targeted reward dimension to a spoken dialogue system selectively and reliably improves that specific quality dimension, without requiring simultaneous optimization of all dimensions.
Evidence: Single-reward RLAIF trained on audio-quality preferences raises UTMOS from 2.16 to 3.06, and single-reward RLAIF trained on intelligibility preferences reduces WER from 6.1% to 3.3%, each moving only its own targeted metric. (§7.1-7.2, Tables 2-3)
- supports: Jointly combining multiple independently-constructed preference datasets within a single DPO training run for a spoken dialogue system yields simultaneous gains across the combined objectives, rather than the trade-offs one might expect from competing reward signals.
Evidence: RLAIF (Joint-Reward-v1), combining semantic, audio-quality, and intelligibility preference data, achieves the best intelligibility of any RLAIF variant (WER 1.0%, vs. 3.3% for the intelligibility-only model) while maintaining strong UTMOS (2.85) and LLM-judge performance. (§7.2, Table 3)
- complicates: Optimizing a spoken dialogue system for stronger semantic-coherence reward signals can come at the cost of emotional expressiveness, since semantic LLM-judge rewards tend to favor safer, more generic responses that reduce speaking-style variation.
Evidence: Adding emotion-consistency preferences on top of joint semantic-audio training (Joint-Reward-v2) yields a worse emotion rank (3.00) than single-reward emotion-only training (1.98) or the non-RLAIF baseline (2.29), attributed to strong semantic rewards favoring safer, more generic responses. (§7.3, Table 4)
- supports: A DPO objective defined over utterance-level preference pairs can align blockwise, incrementally-decoded duplex spoken dialogue systems by aggregating per-block log-probabilities into a single sequence-level likelihood, without requiring reward labels at the partial-utterance level.
Evidence: Applying this blockwise aggregation to a duplex SCoT-Response model yields consistent semantic-quality gains (ROUGE-L 19.8→23.1, perplexity 42.3→25.0, LLM-judge 5.95→6.00) relative to its non-RLAIF baseline, without per-block reward annotation. (§7.4, Table 5)
- complicates: Average-score improvements attributed to RLAIF post-training in spoken dialogue systems are, per the authors’ own account, primarily driven by suppressing degenerate or repetitive generations rather than by a uniform upward shift in response quality.
Evidence: Semantic RLAIF reduces the proportion of low-quality LLM-judge-rated responses (<5) from 10.2% to 7.1% (a 28.5% relative reduction) alongside a large drop in repetition (AutoBLEU 68.3→56.5), which the authors interpret as evidence that gains come from suppressing bad outputs rather than raising the full distribution. (§7.1, Table 2)
Limitations and Open Questions
All preference data in this study is constructed from automatic evaluators (an LLM judge for semantic quality, UTMOS for audio naturalness, ASR-based WER for intelligibility, and an embedding-similarity model for emotion), not human raters; the authors explicitly flag that this may introduce bias or noise relative to genuine human preferences, and identify human-in-the-loop preference collection as important future work rather than something this paper validates.
The multi-reward formulation combines reward signals via dataset-level concatenation (jointly sampling from separately-constructed preference datasets) rather than explicitly modeling trade-offs or interactions between objectives, which the authors note may limit optimal balancing in some conversational contexts, consistent with the observed semantic-vs-emotion trade-off in Joint-Reward-v2. Experiments are conducted entirely on English conversational data (Switchboard, Eval2000); extending the framework to multilingual settings and more diverse conversational domains is left as an open direction.
Wiki Connections
- RLHF Speech — proposes the first multi-reward RLAIF/DPO framework spanning semantic, audio-quality, intelligibility, and emotion-consistency rewards for speech-in/speech-out spoken dialogue systems, extended to blockwise duplex models via log-probability aggregation.
- Speech-to-Speech — targets full speech-in/speech-out dialogue systems in both turn-by-turn Chain-of-Thought and blockwise duplex architectures, evaluated on real human-human conversational data.
- Emotion Synthesis — constructs and evaluates an emotion-consistency preference reward using embedding-similarity between synthesized and human reference speech, revealing a trade-off against semantic-reward optimization.
- Moshi — used as a strong external 7B dual-channel duplex SDS baseline for comparison across semantic, acoustic, and emotion-consistency evaluations.
- Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems — supplies the SCoT duplex architecture and blockwise formulation that this paper’s preference-learning framework is built on top of and extended to.
- From Scores to Preferences — related work on redefining MOS-based benchmarking for speech-quality reward modeling, part of the broader landscape of audio-quality preference learning this paper’s audio-quality reward draws on.
- Align-SLM — supplies the LLM-based semantic evaluation framework this paper directly adapts to construct semantic-quality preference pairs for the SDS setting.
- On the Landscape of Spoken Language Models — cited as a survey providing broader context on the spoken-language-model landscape this preference-learning framework operates within.
- UTMOS — used as the automatic speech-quality estimator supplying the audio-quality preference reward signal.