arXiv · 2026 · Preprint
Lu et al. (Zhejiang University) · → Paper · Demo: ✓ · Code: ✓
SDiaReward trains an end-to-end multi-turn reward model that jointly scores the modality gap (prosody and emotion fidelity) and the colloquialness gap (spoken- vs. written-style naturalness) of full spoken dialogue episodes from a single evaluator.
Problem
Reward modeling and preference learning have become central to alignment in text and vision, but reliable reward modeling for end-to-end spoken dialogue remains underexplored. Moving from text to speech exposes two gaps that existing evaluators cannot handle. The modality gap concerns paralinguistic information (prosody, emotion, channel conditions) that is invisible to text-based evaluators but strongly shapes human preference. The colloquialness gap concerns the mismatch between well-formed, text-optimized responses and the brevity, fragmentation, and discourse markers that characterize natural spoken conversation. General-purpose audio judges exhibit what the authors call “modality blindness,” failing to distinguish natural human speech from synthesized audio when the semantic content is identical, while existing dedicated speech evaluators (e.g., SpeechJudge) target single-turn TTS quality rather than multi-turn conversational dynamics, and evaluators that do incorporate paralinguistic signals (e.g., WavReward) often rely on manually defined acoustic features or rules that generalize poorly to “wild” conversational data.
Method
SDiaReward formulates pairwise preference supervision over full multi-turn spoken dialogue episodes. Given a context of prior turns and a candidate final turn (each turn represented as paired speech audio and transcript), the model outputs a scalar reward that leverages the complete conversational context rather than scoring isolated utterances.
The model is initialized from the Qwen2.5-Omni multimodal LLM backbone (3B and 7B variants), which projects the interleaved speech-text sequence into a joint embedding space. A pooling operator aggregates the final-layer hidden representations across the sequence, and a linear score head maps the pooled representation to a scalar reward. The authors compare last-token, mean, and attention pooling, finding mean pooling most stable across hyperparameters and data mixtures; last-token pooling underperforms, indicating that reward-relevant information is distributed across the context rather than concentrated in one position.
To evaluate along both target dimensions with a single backbone, the reward function is conditioned on a criterion-specific system prompt (inst), reformulating the objective as an instruction-conditioned reward over modality-awareness versus colloquialness rather than training two separate specialized models. Training uses the Bradley-Terry pairwise preference loss to encourage higher scalar rewards for the preferred response of a pair. Because strictly pairwise optimization can allow unbounded score drift across domains (e.g., a model rewarding cleaner studio audio over noisier YouTube audio purely on channel characteristics rather than dialogue quality), a centering regularization term anchors the sum of paired rewards toward zero, stabilizing the reward scale without compromising the discriminative margin between chosen and rejected responses.
Training data (SDiaReward-Dataset, ~13k pairwise samples / 200 hours of paired speech) is built from two complementary preference-pair constructions. Modality-aware pairs juxtapose authentic human speech, stratified into Wild (YouTube conversations), Semi-wild (MELD, acted emotional dialogues), and Scripted (DailyTalk, studio recordings) tiers, against dialogue-consistent synthetic counterparts generated with SoulX-Podcast, a context-aware Dialogue-TTS system chosen specifically to produce high-fidelity “hard negatives” that force the model to discriminate true paralinguistic naturalness rather than exploit gross discontinuity artifacts. Colloquialness pairs contrast LLM-generated written-style dialogues against spoken-style rewrites (fillers, fragmentation, discourse markers) synthesized under identical TTS configuration, isolating stylistic naturalness as the only preference signal. The held-out validation split, source- and metadata-stratified to avoid domination by the high-frequency Wild subset, forms ESDR-Bench, the paper’s episode-level evaluation benchmark.
Key Results
On ESDR-Bench, SDiaReward-7B achieves 96.61% Micro / 94.91% Macro pairwise accuracy on the modality benchmark and 97.2% on colloquialness, for 96.7% Overall Micro / 96.06% Overall Macro accuracy (Table 2). This substantially outperforms zero-shot audio judges: Gemini 2.5 Pro reaches only 72.63% Micro on the modality benchmark despite saturating colloquialness at 98.80%, and open-source audio LLMs (Qwen 3 Omni 30B, Kimi-Audio, Qwen 2.5 Omni) score in the 48-65% range on modality, close to chance for several models. Dedicated speech evaluators SageLM and SpeechJudge hover near chance (50-55%) across modality subsets, and a cascade system (AudioReasoner + Whisper-large-v3 + GPT-4o) reaches 75.20% on colloquialness but only 55.38% Micro on modality, illustrating that discretizing audio to text loses paralinguistic information. SDiaReward-3B trails the 7B model (88.62% Micro / 79.20% Macro on modality), with the largest scale-dependent gap on the Semi-wild subset (55.38% accuracy), suggesting the harder, more heterogeneous Semi-wild data requires larger model capacity.
On out-of-distribution generalization to three unseen TTS engines (OpenAI TTS, CosyVoice 2, FireRedTTS-2), SDiaReward-7B retains 90.9-98.3% modality accuracy, while a dedicated deepfake-detection baseline (Wav2Vec2-large-xlsr-deepfake) falls below chance (38.6%) against CosyVoice 2 (Table 4). A blinded human study on 75 stratified pairs, each rated by three annotators, reports 83.5% (±4.3%) overall weighted agreement with the dataset’s ground-truth preference labels, with agreement rising to 88.3% on high-margin (high-confidence) pairs versus 78.3% on low-margin pairs, and 93.3% agreement on cases where the model itself was wrong.
Ablations (Table 5) show mean pooling with center loss as the best configuration at 7B (96.70% vs. 95.37% without center loss), and confirm that scaling from 3B to 7B, not just pooling choice, drives the largest accuracy gains.
Novelty Assessment
The paper’s contribution is primarily a dataset and benchmark (SDiaReward-Dataset, ESDR-Bench) rather than a new model architecture: the reward model itself combines an existing multimodal LLM backbone (Qwen2.5-Omni), a standard pooling-plus-linear-head design, a Bradley-Terry pairwise loss, and a centering regularization term adapted from prior reward-model literature. The genuinely new elements are the construction pipeline that isolates the modality gap and colloquialness gap as independently controllable preference signals (via TTS-vs-human pairs and written-vs-spoken rewrites under matched acoustic conditions, respectively), the criterion-conditioned single-backbone formulation that evaluates both dimensions without training separate models, and the stratified, metadata-rich ESDR-Bench designed to resist domination by high-frequency data. The OOD analysis and human-agreement study add meaningful empirical rigor beyond a typical benchmark release.
Field Significance
Tip
high — SDiaReward addresses a genuinely underexplored problem (reward modeling for multi-turn spoken dialogue rather than single-turn TTS) with a large, carefully constructed preference dataset, a stratified benchmark designed to probe generalization rather than in-distribution accuracy, and human-verification and OOD-generalization experiments that go beyond typical benchmark releases. It demonstrates that dedicated pairwise supervision can close a modality gap that saturates general-purpose audio LLMs and single-turn speech evaluators near chance, providing a concrete evaluation and potential downstream reward signal for spoken dialogue alignment.
Claims
- supports: Reward models trained on episode-level pairwise preference data can jointly assess paralinguistic (modality) and stylistic (colloquialness) fidelity of spoken dialogue with a single evaluator, without separate specialized models per dimension.
Evidence: A criterion-conditioned reward function
r_θ(C, y, inst)trained on a single Qwen2.5-Omni backbone reaches 96.61% Micro modality accuracy and 97.2% colloquialness accuracy on ESDR-Bench, versus general-purpose audio judges near chance on modality and dedicated evaluators near chance on both. (§4, §5.2, Table 2) - complicates: General-purpose audio-language-model judges and text-mediated cascade pipelines can achieve strong performance on stylistic/textual preference dimensions while remaining close to chance at distinguishing genuine paralinguistic naturalness from synthesis artifacts.
Evidence: Gemini 2.5 Pro reaches 98.80% accuracy on colloquialness but only 72.63% Micro accuracy on the modality benchmark; the AudioReasoner+Whisper+GPT-4o cascade reaches 75.20% on colloquialness but drops to 47.85% on the Semi-wild modality subset, attributed to information loss from discretizing continuous audio into text. (§5.2, Table 2)
- supports: Reward centering regularization mitigates unbounded score drift in pairwise (Bradley-Terry) speech reward model training without sacrificing discriminative accuracy.
Evidence: Without center loss, the mean-chosen reward drifts to µ ≈ 5.03 during training; adding the centering term anchors it to µ ≈ 0.32 while accuracy improves from 95.37% to 96.70% overall. (§4, §5.3, Table 5, Figure 4)
- complicates: A learned speech reward model’s absolute scalar outputs are not domain-invariant, even when its pairwise ranking accuracy is high and stable across domains.
Evidence: Despite 94.91% Macro modality accuracy, the model assigns systematically different absolute reward levels by data source: chosen scores cluster tightly around +0.8 for Wild data but have a negative median (~-0.24) for Scripted data, indicating the model learns a domain-relative rather than a globally calibrated preference scale. (§5.3, Figure 4(c-d))
- supports: Artifact-detection classifiers trained to distinguish synthetic from real speech generalize poorly to unseen, higher-fidelity TTS systems, whereas a preference-trained reward model that also uses full dialogue context degrades more gracefully.
Evidence: A Wav2Vec2-based deepfake-detection baseline scores below chance (38.6% accuracy) against CosyVoice 2 audio, while SDiaReward-7B retains 90.9-98.3% accuracy across three unseen TTS engines (OpenAI TTS, CosyVoice 2, FireRedTTS-2) used only for OOD evaluation. (§5.2, Table 4)
Limitations and Open Questions
Warning
The evaluated systems’ absolute reward scores exhibit domain-dependent offsets (e.g., Scripted-domain chosen scores are negative on average despite correct ranking), meaning SDiaReward’s scalar outputs should be read as within-domain relative preferences rather than a globally comparable absolute quality metric; the authors flag this as an open problem for future domain-invariant reward calibration.
The training and evaluation data prioritize “in-the-wild” recordings (YouTube podcasts/interviews) to capture real-world acoustic variability, and the authors note that broader coverage of high-quality acted speech and additional synthesis engines could further improve robustness. The human verification study, while showing reasonably high agreement (83.5% ± 4.3%), is based on only 75 stratified pairs with 3 annotators each; the authors themselves note that larger-scale studies of fine-grained subjective preference remain a promising direction. The paper also does not itself demonstrate downstream use of SDiaReward as a reward signal in an actual RL or DPO/GRPO alignment loop for spoken dialogue generation, leaving that application to future work.
Wiki Connections
- Spoken Language Model — SDiaReward repurposes a Qwen2.5-Omni spoken-language-model backbone as a multi-turn speech reward evaluator rather than a generator, extending the concept’s scope to preference scoring of full dialogue episodes.
- Speech-to-Speech — The paper targets evaluation and reward modeling for end-to-end spoken dialogue systems, the class of systems this concept covers, and explicitly motivates its work by the gap in reliable reward modeling for such systems.
- Evaluation Metrics — Introduces both a new pairwise-preference reward model and ESDR-Bench, a stratified episode-level benchmark specifically designed to resist domination by high-frequency data types during generalization assessment.
- Subjective Evaluation — Validates the dataset’s automatically derived preference labels against a blinded human study with three annotators per pair, reporting agreement rates broken down by prediction confidence.
- RLHF Speech — Frames the reward model as a prerequisite “compass” for downstream alignment of spoken dialogue models via DPO or GRPO, though it does not itself run an RL or preference-optimization loop on a generative policy.
- CosyVoice 2 — Used as one of three unseen TTS engines in the OOD generalization evaluation, where SDiaReward-7B retains high modality accuracy on its synthetic audio.
- WavBench — A closely related, contemporaneous spoken-dialogue benchmark also targeting colloquialism and paralinguistics, cited as recent work assessing reasoning, colloquialism, and non-verbal understanding in spoken dialogues.
- SoulX-Podcast — Used as the context-aware Dialogue-TTS system to synthesize the “hard negative” half of the modality-aware training pairs, chosen for its multi-turn speaker coherence.
- FireRedTTS-2 — Used as one of three unseen TTS engines in the OOD generalization evaluation; its dialogue-aware synthesis yields a notably higher rejected-score margin than single-turn engines, which the paper interprets as evidence of contextual prosody sensitivity.
- Qwen3-Omni — The Qwen 3 Omni 30B variant is evaluated as an open-source zero-shot audio-judge baseline, scoring well below SDiaReward on the modality benchmark.