arXiv · 2025 · Preprint
Ji et al. (Zhejiang University / Alibaba Group) · → Paper · Demo: ? · Code: ?
WavReward introduces the first reward model for evaluating end-to-end spoken dialogue systems directly from speech-to-speech input, jointly assessing textual content coherence and acoustic paralinguistic appropriateness via reinforcement learning on a new 30K-sample preference dataset.
Problem
Existing benchmarks for spoken dialogue models (VoiceBench, VoxDialogue, SD-Eval, AirBench) evaluate system outputs by transcribing audio to text and then applying text-based metrics such as BLEU or ChatGPT-based scoring. This approach is fundamentally ill-suited for speech-to-speech dialogue systems because it discards non-textual information entirely. When a user speaks with fear in their voice and the dialogue model responds with cheerful mockery, no text-based evaluator can detect the acoustic mismatch. Prior evaluation datasets also lack negative examples, implicit dialogue scenarios, and human-expert scoring across the generation dimension of spoken dialogue, leaving a gap in the ability to assess the “emotional quotient” of end-to-end systems.
Method
WavReward is built on Qwen2.5-Omni-7B-Think, a 7B-parameter speech-to-text audio language model, and adapted into a scoring evaluator through reinforcement learning post-training. The model receives a concatenated speech-to-speech dialogue (user query audio followed by system response audio) as input and outputs a score from 1 to 5 along with a chain-of-thought reasoning trace.
Three components distinguish WavReward’s training from straightforward supervised fine-tuning. First, a multi-sample feedback mechanism constructs three answer tiers for each query: an answer with incorrect content (score 1), an answer with correct content but mismatched acoustic features (score 3), and an answer with both correct content and appropriate acoustic delivery (score 5). Training on these ranked groups teaches the model to differentiate levels of dialogue quality. Second, a nonlinear reward function penalises large scoring errors exponentially rather than using a binary correct/incorrect signal, encouraging high accuracy on clear cases while tolerating small calibration errors. Third, chain-of-thought reasoning is integrated via a format reward that incentivises the model to produce an explicit analysis before emitting the final score.
Training follows the GRPO-style group advantage objective from DeepSeekMath, with KL divergence regularisation against a frozen reference policy. All model parameters are updated; LoRA is used only for a supervised fine-tuning ablation baseline.
The companion dataset, ChatReward-30K, contains 30,000 speech-to-speech dialogue samples covering four scenario types: text-based content (15%), explicit paralinguistic understanding (25%), explicit paralinguistic generation (35%), and implicit emotional dialogue (25%). Dialogue speech is synthesised using a mix of GPT-4o-mini-TTS, Step-Audio-TTS-3B, and CosyVoice2, with Whisper-Large-V3 filtering (WER < 5%) and Emotion2Vec verification for emotional samples. Human experts manually verified and scored all samples.
Key Results
On ChatReward-30K-test, WavReward achieves overall accuracy well above all baselines across content, explicit paralinguistic, and implicit dialogue categories (Table 2). Compared to the strongest open baseline Qwen2.5-Omni (direct inference), WavReward raises accuracy from approximately 53% to 91.5% on average, and improves implicit dialogue scoring from 48.5% to 74.3%. Against GPT-4o-audio, WavReward surpasses it on all categories, with improvements of 21.4 points on content scoring and 39 points on emotion-instruction scoring.
On the out-of-domain RealDialogue dataset (120 real human-machine dialogues with LLaMA-Omni and Kimi-Audio), WavReward achieves 80.8% accuracy. In subjective A/B testing against Qwen2.5-Omni direct inference, WavReward’s scoring was preferred by five human experts in 83% of comparisons; against GPT-4o-audio, the preference rate was 77% (Table 3).
Ablation results confirm that all three components contribute: removing chain-of-thought reasoning costs roughly 10 percentage points across categories and up to 21.7 points on out-of-domain data; removing the nonlinear reward reduces performance particularly on implicit and generation tasks; removing multi-sample feedback most notably degrades accent and implicit dialogue accuracy.
RL fine-tuning outperforms full-parameter supervised fine-tuning, which itself outperforms LoRA-based fine-tuning, consistent with findings from text-domain reward model training.
Novelty Assessment
The core contribution is straightforward: adapt an existing audio language model (Qwen2.5-Omni) into a speech-to-speech dialogue evaluator using group-based RL, with no new architecture. The RL training recipe (GRPO, nonlinear rewards, CoT integration) transfers well-established techniques from the text-domain reward modelling literature (DeepSeekMath, DeepSeek-R1) to the audio domain. The novelty is primarily in the dataset (ChatReward-30K) and the application framing: no prior work had constructed scored preference data for speech-to-speech dialogue covering implicit and acoustic dimensions.
The claim of being “the first” reward model for end-to-end spoken dialogue evaluation appears valid given the publication date and the benchmark landscape. However, the methodology is a direct adaptation of text-domain RLHF rather than a principled new approach for the audio modality. The biggest open question is whether the dataset’s synthesised speech (from GPT-4o-mini-TTS, Step-Audio, CosyVoice2) adequately represents real dialogue acoustics, which the RealDialogue experiments partially address but do not fully settle.
Field Significance
Moderate — WavReward fills a real gap in evaluation infrastructure for spoken dialogue systems, which have matured faster than the tools used to assess them. The dataset and evaluator together provide the field with a scoring mechanism that can handle acoustic dimensions of dialogue quality that text-based proxies miss. The methodological contribution is incremental — adapting text-domain RLHF to the audio scoring task — but the evaluation infrastructure contribution is substantive given the absence of alternatives.
Claims
- Text-only evaluation of speech-to-speech dialogue systems systematically misses acoustic-level mismatches that human judges readily detect. (§2.2, §4.3)
- Reinforcement learning with group-based advantage estimates substantially outperforms supervised fine-tuning for training audio-domain reward models, consistent with patterns observed in text-domain RLHF. (§4.2, Table 2)
- Chain-of-thought reasoning improves scoring accuracy of audio language model evaluators, with the benefit amplified in out-of-domain settings. (§4.4)
- Nonlinear reward shaping that penalises large prediction errors exponentially improves calibration across acoustic attributes compared to binary correct/incorrect rewards. (§3.1, §4.4)
- Implicit dialogue scenarios (where the appropriate response depends on inferring the user’s emotional state from speech) remain significantly harder to evaluate automatically than explicit instruction-following scenarios. (§4.2, Table 2)
Limitations and Open Questions
Warning
The entire ChatReward-30K training corpus is synthesised using TTS systems (GPT-4o-mini-TTS, Step-Audio-TTS-3B, CosyVoice2) rather than real human speech. WavReward’s ability to generalise from clean TTS training data to real-world dialogue audio with background noise, cross-talk, and spontaneous speech characteristics is tested only on a 120-sample RealDialogue set, which is too small to draw robust conclusions.
Accent accuracy in ChatReward-30K is noted as lower quality than other acoustic attributes, and WavReward scores correspondingly lower on accent sub-categories. The dataset lacks any music-related audio events by design, restricting scope to conversational acoustic phenomena. Implicit dialogue evaluation remains an open challenge: the paper achieves 74.3% accuracy but acknowledges there is no single ground truth for what constitutes an emotionally appropriate response. The model size is fixed at 7B parameters; the paper proposes scaling to 70B as future work but provides no scaling evidence. GPU training details are withheld due to company policy, reducing reproducibility.
Wiki Connections
- spoken-language-model — WavReward provides the first evaluation framework specifically designed to score the acoustic and content quality of end-to-end spoken dialogue systems.
- evaluation-metrics — introduces ChatReward-30K and an RL-trained scoring model as new evaluation tools covering dimensions absent from existing spoken dialogue benchmarks.
- rlhf-speech — applies group-based RL reward modelling (GRPO with nonlinear rewards and chain-of-thought) to audio language model fine-tuning, extending text-domain RLHF techniques to speech evaluation.
- 2411.13577 (WavChat survey) — the authors of WavReward overlap with WavChat; WavReward provides the evaluation tooling that the WavChat survey identifies as missing from the spoken dialogue ecosystem.
- 2410.00037 (Moshi) — Moshi is one of the end-to-end spoken dialogue systems whose evaluation gap WavReward is designed to address.
- 2409.06666 (LLaMA-Omni) — LLaMA-Omni responses are used in the RealDialogue test set alongside Kimi-Audio to evaluate WavReward in out-of-domain real-world conditions.
- 2410.17196 (VoiceBench) — VoiceBench represents the text-proxy evaluation approach that WavReward argues is insufficient for speech-to-speech dialogue assessment.
- 2503.20215 (Qwen2.5-Omni) — WavReward uses Qwen2.5-Omni-7B-Think as its backbone and Qwen2.5-Omni as its primary direct-inference baseline.
- 2407.10759 (Qwen2-Audio) — used as a baseline evaluator; shows substantially lower scoring accuracy than WavReward across all categories.
- 2412.15649 (SLAM-Omni) — one of the end-to-end spoken dialogue systems in the landscape that WavReward is positioned to evaluate.
- 2501.06282 (MinMo) — another end-to-end spoken dialogue model representative of the systems WavReward targets for evaluation.